Computer Says Maybe

Purity: [00:00:00] They would enter all these characteristics and boom, a certain figure would come. And whatever figure you're given initially, you're supposed to pay it.

Alix: And if you don't pay that, you're not gonna get access to public hospitals.

Purity: An algorithm is blind to realities, and artificial intelligence is insensitive to real life.

Alix: A mere handful of tech companies have enormous power over the systems we rely on, but the conversation about how these companies impact people are really vague and superficial. We need to hear from experts and advocates focused on understanding what's actually happening, knitting together a global understanding that empowers resistance.

Welcome to Real Resistance, a series produced in collaboration with Real ML, the network doing just that. We're talking with members of that community who have the receipts, the stories, the research, the insights to build a future that isn't governed by tech billionaires. And in this conversation, we're discussing how the digital welfare state is missing the one key thing it's supposed to [00:01:00] provide, benefits.

Maria Pilar: I am Maria Pliar Llorens. I am an associate researcher for the Center of Technology and Society Studies at the Universidad San Andres in Buenos Aires, where we are conducting research on the use of AI and the judiciary.

Divij: My name is Divij Joshi. I am a research fellow at ODI Global where I lead the program of work on digital societies.

Gabriel: I'm Gabriel Geiger. I'm an investigative journalist at Lighthouse Reports where I focus on technology investigations.

Purity: I am Purity Mukami. I'm a data and investigative journalist currently working as a freelancer and working with Lighthouse and African Censored on this project.

Alix: We're gonna start with a piece of investigative journalism that I feel like has been in progress for a really long time.

Um, Purity and Gabriel, how long have you been working on this, and can you sort of take us through what you've discovered, um, in the course of your investigation?

Purity: So how I [00:02:00] came across this project was through coming across a previous project that Gabriel had worked on with his colleagues, and they had been shortlisted to these awards, well-tuned data stories.

The project was the Inside the Suspicious Machine. I was reading the shortlist of these awards, and I came across this project, and Gabriel and his colleagues had found out about this computer code that was used to target Rotterdam residents, and they obtained the underlying code, and they found out that this code was targeting people based on their native language, gender, sometimes how they're dressed, and I found that really fascinating as a statistician.

But I thought that such an advanced economies kind of project or kind of problem, and this is nowhere near to happen in Kenya. Exactly two weeks down the line, I would put on the [00:03:00] live discussions of the Kenyan Parliament, and the members of Parliament would be shouting at each other or discussing issues.

I am not listening to them really, but I am having this background beautiful kind of distraction. And th- on this particular day, a certain MP went like, "Okay, thank you, honorable speaker." And going by the explanation given the cabinet secretary, she's talking about mean testing system being an algorithm that will be applied to assessments on households.

And he went like, "An algorithm is blind to realities, and artificial intelligence is insensitive to real life." I was like, "Did they just mention the word algorithm?" As a statistician, these are the kind of words that would pass me even when I'm not actively listening. And I was like, "Are we using algorithms like the [00:04:00] Rotterdam, like in Europe?"

I didn't expect this to be here this soon and in this mass because they were discussing this healthcare funding system That was going to be so mainstream, like that was going to cover 80% of the population that is in the informal sector. So to me, it sounded like such an important area, but I did not expect us to be using algorithms.

It had not occurred to me. So it really piqued my interest. In fact, I stopped working on what I was working on and started now researching about this mint testing thing. And I met Gabriel like three weeks later, and I told him, "Oh, I saw you did a story about algorithms in Rotterdam. Can you imagine we have algorithms in Kenya as well?

Can you help us uncover what's going on with this algorithm in Kenya?"

Alix: And so Gabriel, you get this email from a Kenyan journalist who's interested in this question and like [00:05:00] drawing a connection to your work. Were you like immediately pulled in and interested? Or sort of how did, how did that play out?

Gabriel: We were actually at a conference together.

We'd never met before, and Purity pulls me aside and starts telling me about this algorithm. You know, it's using what your roof is made out of. It's using, like, whether you share a toilet to predict your income. And I was like, "That sounds so out there." I was like, "Okay, there's obviously someone in front of me who's really, really smart, but this sound...

I've never heard of a system that works that way." But it piques my interest, and Purity starts sending me some links and some documents, and I realize that, like, this is actually happening. And so we sort of come across the backdrop to the system, which is that Kenya has this public healthcare system, and traditionally it's been mainly supported by salaried employees whom the government understands how much money they make.

Well, the problem is that in Kenya, 80% of the country works in the informal economy, so the government has no records regarding how much money they make, and [00:06:00] the healthcare system, the financing situation for the healthcare system is unsustainable. So they implement all these reforms, and the sort of centerpiece is this AI system that will predict how much money you make and then determine how much money you have to pay for public healthcare based off of that predicted income.

And one can imagine how-- the potential issues that could arise if you're trying to use a machine learning model to predict how much money people make in a country as complex as Kenya, in a country where 40% of people live below the poverty line. That's what they decided to do. So we embark on this investigation, Purity and I, to try to understand how the system is working.

Pretty early on during that process, we come across YouTube videos and TikTok videos where people are recording themselves as they fill out this form that's asking them things like, "Do you own a radio?" Like, "Are you married? How many kids do you have?" You know, "Where do you live?" And they sort of fill out this form, and then they click enter, and at the end [00:07:00] this number is spit out, and that's the number that they have to pay per month in order to have access to the public healthcare system.

And we see tons of people online complaining about this number. "It's too high. I can't afford it. I'm being turned away from the hospital because I can't afford my healthcare premium." And so we sort of embark on this, on this journey to actually understand how this system works. Purity comes over to Athens for a few weeks.

We're working out side by side at, we call it our boot camp, at the Athens office, and she makes this huge discovery, which is that the training data for the system, so the data that was actually used to build this model, is just lying on this government website on the internet. And, you know, we're both super excited, and we download it, and we start sort of going through the tedious process of reconstructing how this algorithm works.

I won't get into it, but we have this sort of long back and forth with the Ministry of Health that involves complaining to the ombudsman, who orders the Ministry of Health to respond to some of our questions. And using their [00:08:00] responses, this publicly available data, and these videos of people filling out the form, we're able to actually reconstruct this algorithm that's being used to determine how much Kenyans should pay for healthcare.

So proxy means testing is this decades-old idea that economists have been toying around with since the '90s, and basically what it says is like, okay, in contexts like Kenya where you don't know how much money people make and where self-reporting income is unreliable because most people don't keep records of how much they earn, what you do instead is you ask them questions about what they own and their demographics, proxies, in order to predict how much money they have, how much wealth they have, and that predicted wealth is used to determine typically access to different social programs.

So, you know, in a nutshell, it's a system that uses these proxies like where you live and how much, uh... you know, what, what you own to determine if you're poor enough, quote unquote, to access a [00:09:00] particular social service. Now, the Kenyan context is using this proxy means testing methodology but in a slightly different way.

So rather than a zero one, poor enough, not poor enough, which is how it's traditionally used, this proxy means testing system predicts your income, and then you are charged to access public healthcare 2.75% of the predicted income. So your predicted income is 20,000 shillings. That's how much the algorithm says you make, so you're gonna be charged 2.75% of 20,000 each month.

You have to pay that, and if you don't pay that, you're not gonna get access to public hospitals. What we found when auditing this, this system is that the system is really flawed, so it's over-predicting a lot of people's income, so they're being overcharged. It's under-predicting a lot of people's income, so they're being undercharged to access public healthcare.

Most importantly of all, it's overcharging the poorest Kenyans and undercharging the richest Kenyans. So the poorest Kenyans are [00:10:00] being charged more for public healthcare than they should be, and the richest Kenyans are being charged less for healthcare than they should be, and that's the sort of headline finding of our audit of this system.

Purity: While talking to some of the statisticians, we also ask them to try and explain to us what is means testing and why, while we discovered what they call means testing instrument or tool, it, uh, to me, a statistician, it's this linear regression formula. It was, like, explained as this AI system that is going to equalize everybody, and no one is going to be left behind, so we all need to register.

So they do not tell us we all need to provide information for them to calculate. It came as a way of registering. So people ... And this is to move into the effect or the [00:11:00] experience that Kenyans have had with this. So they would think it's like a registration form. They just think they're just filling the information as a way of registering to this system so that they're able to access healthcare.

And they would have all sorts of expectation based on the marketing and the campaign promises that was given about this system. So most of them, because there was a previous system where people would pay a flat rate of 500 Kenya shillings, it's like something less than $5, so they would think this system is going to charge them something less, something they can afford, and then they'll be happy to pay.

But no, as Gabriel explained earlier, they would enter all these characteristics, and boom, a certain figure would come. And whatever figure you're given initially, you're supposed to pay it. When we finished understanding the mathematics and the nitty-gritties of how this model is working, we went ahead to [00:12:00] talk to real Kenyans on the ground about their experience and their challenges and asking them, "Why are you not paying your contribution?"

My own experience with these people was quite humbling because I would go to the places they live in, and some of them, the house would be made of stone, but they are in, like, very low-class neighborhood kind of conditions. And you could literally see the amount they were given. They couldn't pay it because if they could afford such an amount, they wouldn't even be living in the kind of conditions they are living in.

And I would talk to them about, like, the answers you gave went into a formula that helped give you this amount of money, and it's quite not easy for most people to understand because of their level of education and the way this has been, um, communicated. They really avoided discussing[00:13:00]

How this model is working. In Kenya, a lot of people go into poverty base, like one sickness in the family sweeps away everything. So everyone wants to have affordable healthcare, so they, they really want to participate in the process, but the amount that is being spit out by the formula is not helping the same people that the government promised that this system is going to solve the problem for.

So this is why this project is, like, very close to my heart because I feel even though these people are not statisticians like me, they deserved better. They deserved to understand. They deserved to know what's going on so that they can question it, not so that they can put in the wrong answers. One of the questions one of the motorbike riders asked me, "I answered the same form with my fellow boda boda [00:14:00] rider."

Boda boda is like this motorcycle business that is very common to young Kenyans, and he's like, "We answered the same form. My friend got maybe 350 per month, and I got 750 per month, and we go to the same job, same location, go home with the same kind of income. But why did this model not know that? Why did it give me more than my friend?"

And I would be like, "Maybe it's because of your level of education." So there will be Kenyans who are university graduates, but because of the level of unemployment, they are doing these jobs like, uh, riding motorcycles. But because this formula is putting into account the level of education, it does not take into account that the job you're doing is not commensurate with this.

And some of the health program workers we talk to, they [00:15:00] would raise the same, like, level of education or the kind of house you live in does not necessarily depict what you earn. So people have been quite disappointed, and, uh, they've tried even to game the system, give wrong answers because they're trying to wonder, "Why is my neighbor who we live in the same place getting different formula than the other?"

So it's been very defeating for me to see them having this confusion- Mm-hmm ... of not understanding the why. So they've really been wondering why. I,

Alix: I mean, it's so- Um, similar it seems to the way that surveillance pricing works in the private sector, where you can go on to buy a product and all kinds of information has been gathered about you, and then you're being presented with a price that is different maybe than your neighbor.

Um, but I think this is-- these are just really interesting questions, uh, to ask when governments decide to use [00:16:00] proxy. What do you think motivated the Kenyan government to use these technologies? It sounds a little bit like we need more income from the people that live here to fund services. But I feel like this wasn't a member of parliament in Kenya got really excited about technology and was like, "Wouldn't it be cool if we rolled out this thing?"

I feel like there are other factors and maybe actors at play here. So any final words on how we got here with the Kenyan government being this excited about these technologies?

Gabriel: Sure. I mean, um International donors like the World Bank and the IMF and, uh, USAID, when it was still a thing, have really been pushing these types of systems in Kenya, and we know that this comes from the top.

So basically, there was this one advisor, close confidant of the president, who really pushed this idea of using proxy means testing to expand the revenue base. And there are, of course, many other ways you can design a healthcare system and funding for it. I mean, you could [00:17:00] place more taxes on the wealthy.

You could think about a sales tax, fuel tax, whatever it might be. Like, it's not like this is the only way you can fund a, a public healthcare system. But they went with this particular system, I think after pressure or, you know, advocacy from large multilateral development institutions, and many other countries have sort of been under similar pressure to implement these types of systems.

A lot of times they can be preconditions for receiving things, like World Bank loans, so there's a sort of incentive structure that I think pushes people to these types of tools. But I think on a deeper level besides that, and this is something that we heard from sources and stuff at the time who have been close to this process, is it's also a way to just sort of deflect accountability and responsibility, right?

Like, this tool is presented as, you know, this scientific, objective means testing tool. If you don't agree with its prediction or its decision, it's because you just don't understand it. This has been designed by a bunch of really smart economists and scientists in the room like you could never understand.

You wouldn't even be able to wrap your head around it, but trust us, [00:18:00] bro, it's, it's scientific. It's kind of the deeper current that pushes this, not just in this Kenyan context, but more broadly, I think a lot of times these automated systems can become a way to deflect accountability or the sort of like intermediary between you and the government, especially when the government sort of casts this as, as an objective tool.

And, and that's something we heard from sources who are really closely involved with this process, and it's worth saying that, you know, people inside the government, people outside the government who are consulted, were ringing the alarm about this system, telling them, "This is a really bad idea." We have one document where this group of consultants is saying, "This is gonna be a disaster, and this is gonna overcharge a bunch of poor people," and the government just like steamrolled right ahead.

So they knew this was gonna happen. Like, it's not like, oh, you know, they just didn't get it and, oh, this just, it turned out bad, but it was just poorly planned. No, like they were warned well in advance that this was gonna be the outcome and went ahead anyways because they felt like there was this sort of mad rush to get this new [00:19:00] system up and running, and they felt like this would expand the revenue base, and that's what needed to happen.

Alix: I feel like trust us, bro, this is going to work out is a posture that isn't just happening in Kenya. Uh, Maria, do you want to talk a little bit about the work you've done to uncover the applications of technology, AI, probabilistic systems in judicial contexts? Which feels ironic that some of the places that are meant to, um, be determiners of fact are turning to systems that are designed to be non-deterministic.

They would be using these technologies. But tell us, like, what is happening in Latin American countries in judicial contexts where governments and judicial bodies and instruments are saying, "Hey, we can be Cool too, and use technology to either help with the backlog or other, other ways that everyone will understand, even if we make some mistakes.

What have you found and what have you been working on?

Maria Pilar: Okay, thank you, Alex. It, it's a [00:20:00] very interesting process because, uh, until 2023, when ChatGPT came online, judicial systems were not using so much tools for automatizing process. But since the operation of the GPT models, all the judicial systems wants to use these tools because they are seen as very helpful for the efficient part of the, of the process.

I think the problem is that in Latin America, and maybe it can be replicated in other parts of the world, there are a big backlog of cases. The judicial systems needs to address this topic. For example, in Argentina, the judicial system or the judiciary has a very negative image in the population. They need to address this backlog of cases.

So they say, "Okay, we're using these [00:21:00] tools because they help us to be more efficient." But I think that there is a hype to use these tools and not a realistic need to use the tool. Because the problem is that nowadays when you speak to the judicial ecosystems, when you speak about AI, they associate AI with generative AI tools, and they think that these are the tools that will help them with everything.

And I think that this discussion is important because there are some tools for automatization that are useful, and they need to be developed. But generative AI is complicated because if people are using these tools without, uh, really control or without having a very good oversight from other parts or superior courts or the humans involved in the decisions.

For example, in Colombia, the [00:22:00] constitutional court developed a judgment where it explained how AI must be used. In Argentina, we have a case where a judge decided a penalty for a crime using AI. And this went up to the superior court of this province, and the superior court said, "Okay, it's not bad that the judge had used AI, but the judge must be responsible for this use."

And for me, it's the very bit tricky because how do you know that the judge exercise control over the things that they add, the generative models are producing? Uh, I don't know. And for me, it's very difficult to, to assess how we use these tools in the judiciary because they are useful But at, at certain point they, they affect how human rights must be protected.

I think that [00:23:00] the tools are going to be used more and more, but we have to develop a system to control better how the, the people are using the tools inside the judiciary.

Alix: And similar to the Kenyan case, do you get the sense that this is coming from people in the courts, or do you get the sense that there's this expectation from external actors or some type of pressure for courts to be perceived to be using these technologies in particular ways?

Maria Pilar: No, I think it comes from the people from the judicial system, and the people from the judicial systems wants to be perceived as tech-savvy. And I think someone said to me once that there is a rush to be the first judge to use AI and to be techy, uh, and to understand these tools. I think that's something that resonate with me is something that Gabriel said, that there is a sense that if you don't use these tools, you don't understand how they [00:24:00] work, so you use it because you are very cool using these tools.

So I think there is a push from the inside or coming inside the judiciary to use these tools. I think that we have to have in mind that not all judges want to use these tools, and some are very concerned on how these tools are used, and they push for capacity building exercise and to have a normative framework.

So they are not all judges, but there is a push from some judges and some influential people within the judicial community to use these tools. So I don't see a push from the companies to use these tools, but I think when the tools are integrated with some per, for example, processors, there will be a push to use these tools.

I don't know. I'm thinking like Copilot [00:25:00] on Microsoft.

Alix: It also feels like an interesting time for the human rights courts in general engaging with technology, so thinking about International Criminal Court and some of the ways that the US tech reliance has led to political vulnerability. So like the fact that Microsoft can just, like, turn off the email account of, uh, someone working at the ICC because there's a request from the Trump administration.

I don't know. I wonder if you have thoughts on the medium is kind of the message here, like sort of using these technologies in a human rights context when the technologies themselves have human rights implications. Any reflections on your feelings about that? And like should these courts be using it?

Maria Pilar: There is a need for the courts to answer to the question, what do we need these tools for? And it's the question that I think no court is asking because I think it, there is this need to use the tools to say if everyone is using th- this technology, [00:26:00] why we are not doing the same? So the courts didn't consider why and what they need the tools for.

For example, I'm thinking if they need tools for automatization, they not necessarily need to rely on commercial tools, and they could build a inside process to help this happen, and I know that some courts in the national level have done that. At the generative models, that's the, the, the ones that they are using, I think that the courts need to have more insights on which companies are they are relying on for these tools.

The other thing that I think that we have to have in mind is that courts usually don't have the power to discuss large contracts with these companies because we can think that the courts could negotiate some clauses to have protections on how they [00:27:00] use the information and when the service is going to be provide.

But I think the courts lack this standing at the international level to have the power to really negotiate clauses that they are built to protect the human rights or, or the normal functioning aspect of the court.

Alix: So now I feel like we've had two examples in my head. One, a government trying to use technology to kind of administer something with its people, and then a government using technology internally in its own processes to make things, you know, quote-unquote, more efficient, um, or try and deal with a lack of resources appropriately allocated, um, to a core function of, of, of its government.

Divij, you work a lot on thinking about the political economy of all these questions that governments ask, um, and the underlying infrastructure that's kind of slowly being built up, um, in the form of all kinds of systems that are meant to interact [00:28:00] with each other to kind of help lift all boats of governments all around the world, um, to use technology to greater effect, and has a very obvious ideological tint to it, which I think we're already seeing in all these examples.

Um, but do you wanna talk a little bit about your work? I don't know, what project you might wanna talk a little bit about in terms of digital public infrastructure that I feel like in some ways ties a lot of this all together.

Divij: I can speak to two main, maybe big projects I've done in this sphere. So one is the kind of standardization of, as you mentioned, both the technical but also the institutional and the political infrastructure behind some of these systems.

We've spoken before as well about this kind of new language and narrative and systems that are being built as digital public infrastructure, which basically take it for granted that there's these three kinds of systems that every society needs, or at least every state in the Global South in particular needs, which is if you want to be an effective welfare state, you need a digital ID, you need a payment system, also a digital public finance [00:29:00] system, and you need a working data exchange infrastructure.

And this idea of kind of what a state's digital infrastructure should be kind of emerged to a large extent from India somewhat- You know, pitching itself, pitching things that it had already done as the way forward for a lot of other developing or states in the Global South, which wanted to scale up their digital infrastructure for public services, but also to kind of, to also shape markets in different ways.

And so if you read, for example, some of the justifications provided by particularly large donors and funders, IMF or the World Bank, for example, on why this system should be created, it's not only for creating more efficient government services. It also says that, "Hey, at the same time, you know what?

You're now given access to a huge market through this new database that you have created, and now private organizations can happily serve this, uh, large population that you've now integrated into your digital networks." Um, and so a little bit you [00:30:00] can see some of the maybe ideological motivations behind why these systems are being created.

I think what, what's really curious and what's really interesting about some of what we've heard today as, as well, is that these systems are now again being linked in different ways, right? There's, like, this long history behind social registries. It's not necessarily like a super new thing. And there's a long history behind digital ID projects or public-- digital public finance or, and, and payment systems.

But now they're being integrated with this kind of new boostery language of DPI, which is kind of standardizing everything. It's saying that this is, like, the only way forward for you, and everyone must develop these systems so that- They can be kind of interoperable and work with each other and with artificial intelligence, right?

So there's a big, big push around the world, has been for years now, I think, to start experimenting with data analysis techniques. And when you start exploring these systems, you start seeing that there's a lot of strange experimentation [00:31:00] going on in the ways in which these systems are being standardized and the ways in which they're being piloted and rolled out, right?

So as an example of the kinds of projects this kind of language of boosterism enables is a project that I brought to RealML, I think 2022, which was very similar to the work Gabriel and Puriti are doing right now, in that it considered a social registry that was being developed in India called Samagra Vedika, where we had noticed that there were a lot of claims being made about the use of AI for social protection.

What got me interested in that was, like, this quote by a senior bureaucrat who said that, who when confronted by questions of bias or inaccuracy said, "The more data we feed into the system, the better it's gonna get, so you don't have to worry about bias anymore." And that got us really concerned about what was really going on.

We did a deep dive investigation and published in Al Jazeera on the kind of actual impact of the system and the kind of institutional opacity that we faced in actually trying to understand the system. We noted there that, okay, there's a lot of language [00:32:00] of AI and machine learning being used, but what's actually happening?

And we saw that, as Puriti mentioned, machine learning is already foundational to proxy means testing, that they use particular kinds of algorithms to do that PMT thing. But that's becoming far more complex given the kind of rapid integration of different kinds of databases. We saw that in Samagra Vedika, there were more than 30 databases of various levels being integrated.

This kind of was a systematic change from maybe what happened earlier, right? We saw that instead of what used to be kind of household declared information with, from household to household or things that were kind of observed by people going on survey This information has now gradually become centralized and basically takes whatever they can capture in any existing database.

So it's not that you have a specific idea about what the proxy is, it's what you have on hand. So if you have electricity information on hand, let's capture that information. If you have information about who drives a [00:33:00] four-wheeler on hand, let's capture that information. And the more kind of government information is getting integrated into systems that are called golden records or single registries, or i- in a somewhat Orwellian manner in India, they're often called 360 degree people's information hubs or 360 degree surveillance hubs.

And then a, a, a new, a, a different form of structural exclusion that these digital systems are, are brought into place is also around the unavailability of, for example, people to be able to reach or be logged into some of the systems that are being put in place like Aadhaar, um, you know, the biometric requirements that people often need to fulfill, or the fact that there are d- like far more additional administrative hurdles that people need to jump through to be able to be enrolled in these systems.

Where at one point you may have been able to simply go to your local government officer at a village level or at a block level, now in order to correct any kind of data authentication [00:34:00] error, you're now being told to go to the central data authority that's located in the state capital that's, that's like 500 kilometers away.

And so there's all of these multiple levels of kind of failures of infrastructure, even as this kind of boosterism is going on at, at kind of the policy level. I, I think what's happening is like with the integration of these systems and with standardization of these systems, we're not really looking at what's going on on the ground.

We're simply kind of, you know, taking claims about the efficacy of technology or the efficiency and the scale at which it works at face value. These technocratic models of what digital infrastructure can achieve are, are frankly failing. They assume that a welfare state can be built essentially without the input of the people who are actually affected by these systems, and only take into account this almost like kind of obsession about scale, about fiscal savings, about auditability, or about essentially innovation and efficiency and so on, without kind of, [00:35:00] you know, having real political legitimacy behind these systems.

Alix: I think this sequence was really interesting, having, you know, us starting in Kenya and looking specifically at an application of a system that's hugely consequential for people in terms of their financial health, their access to health systems, their general understanding of how government operates, and then moving to this more kind of judicially specific expert space where there's this drive towards efficiency, even if it doesn't feel aligned in terms of the actual subject matter of, of the courts.

And then sort of taking this more structural view about how this is an approach that lots of countries are taking in this era of if you're not tech bro-ing the nation state, then there's like you're missing a trick and you shouldn't be in charge of a country. Um, and it just feels like all of these-- to me, all of this connects in such an interesting way.

But I wanna give you all a chance to, like, ask each other questions about your work. So as you were hearing these other projects, um, you know, what does it make you think, or what [00:36:00] dots are you connecting between your work and, and what you've just heard?

Purity: I have a question for Malia. In terms of when these systems are being Used, is it the same as in Kenya where before they started using it, they had an entire law made that allows or legitimizes the use of this model?

Like, they made the law first and said mean testing with this particular formula is going to be used on these specific people, and that the amount that is given out, you must pay it. It's in the law. I was wondering for your case, when they're using these, uh, systems for judges to make such important decisions, are there laws that they go and legitimize that field before they implement?

Maria Pilar: That's an excellent question. Thank you, Verity. In fact, there are, in Argentinian case, we don't have [00:37:00] a general AI law for the nation. Each province has its own procedural laws, so they can decide if they are going to use AI or not. And the thing that has been happening during the last couple of years is that most courts are not saying anything.

They don't say you can use the tool, and they don't say you cannot use the tool. So they are just hoping that people work around the use of these tools, and if something happen, we will see. But there are other courts that are trying to embrace the AI, and they are producing some documents that are binding for the court or for the judiciary in that province, and they are trying to explain or trying to ascertain some rules.

And they say, "Okay, if you are going to use generative AI [00:38:00] tools, you can use it, but you have to abide by the principle of human center, the human on the loop, uh, protection of human rights." But they are very, very wide, and you don't actually know What are the principles or what are the limits for the use of these tools within the court?

And something that for me is very problematic is that you don't have to disclose the use of these tools to the people that are being judged by the decision of the court or, or the judge. The other thing is that we have this case where a judge used AI, and they have these binding rules that they need to use the tools within some principles that we don't have an oversight for the actual use of the tools in an specific case.

And we have [00:39:00] this case where the judge decided to use ChatGPT, and we know that he used one of the tools because there was a particular phrase that say something like, "Here you have your clean version for the fourth argument ready to copy and paste," and it was in the actual judgment. And the s- second tribunal said, "Okay, you used the tool, so we are not sure if you decided or not."

So you have to judge again. And the Superior Tribunal of Chubut, this is the province where the case occurred, said, "Okay, no, you cannot say that the judge is fault because the judge signed the, the judgment, and therefore the judge is responsible for the conduct." But I think here we have to think what is the control that we are requiring from the judges means, and are they having an actual control?

Because, for [00:40:00] example, you asked a model to write an argument, and the argument, it's okay. You ask a second time, "Write an argument." The second time the argument is okay. But what happens with the 50th time with this argument? Are you checking this, really checking, or you are just trusting the thing that the AI is saying?

I think that we have this problem. I think that having a law is not going to help us to navigate how we actually use the tools within the judiciary.

Divij: I think with the India experience, and actually this is maybe happening elsewhere as well, is that governments are finding new ways to evade the question of political legitimacy entirely.

So with Aadhaar, it was this idea of developing government as a startup, right? So startups need to be agile. They don't need to follow kind of the regular procedures for procurement. They don't need to follow the regular procedures for how, uh, you know, who gets in and who gets out, which meant that there was a big revolving [00:41:00] door between pri- private tech companies and the, and the public institution that actually developed Aadhaar.

None of them, I would say, are really politically legitimate, and this also has to do with the way in which kind of legislation is passed in India today, which means that you can ef- effectively legitimize anything because there's not really a lot of parliamentary scrutiny going on. A lot of times the rules of these systems are what the code is, the kind of policies really embedded within the technical infrastructure or within the kind of algorithmic systems that are built out.

And then that raises the question of who's making these systems. Even if they're pushed by the state, the code itself, the system itself is being built usually by some kind of private contractor. If I may turn that into a question, I'm curious about how you see the role of the state being changed in all of their work in these systems by their need, effectively in most situations, to onboard some kind of private technical capacity.

Gabriel: I mean, speaking more broadly [00:42:00] in the different systems that I and Lighthouse have investigated. I mean, there's a huge role for the private sector in building these systems, and oftentimes a lot of them are being built by, like, the five consultancies, so like Accenture, Deloitte, McKinsey, can't remember the other ones.

And I think there's a really interesting dynamic where these consultancies will kind of set up a problem in a report that a government didn't know it had, quote-unquote "problem," and then offer the tech to solve that problem. So the classic case in Europe is welfare fraud. So these, uh, consultancies put out these big reports like, "You're losing this much money to welfare fraud," da da da da.

"Okay, here's the solution, this AI system that's going to, like, grab all this money that's supposedly being lost from the public purse." I think in the Kenyan context we see this really clearly because This entire digital in- uh, public infrastructure that's running this healthcare system is being built by this contractor, Safaricom, which is the biggest telecommunications [00:43:00] company in the country.

They're being paid over 100 billion shillings for this really crude system, and they're basically locked into a contract. The sort of underlying infrastructure for this public health project isn't actually owned by the government. It's a software as a service, so it's actually owned by the contractor.

So if they don't pay or they want to move away, the, the actual intellectual property of this digital infrastructure is held by a, a private company. So it's basically like a hostage, a hostage situation. That's something we've seen as well with, with digital ID in, uh, in Kenya as well. IDEMIA, a French company, uh, owns a lot of that, that infrastructure.

So it's this pattern that we sort of have seen in a number of different contexts when we've investigated these tools. You build a welfare fraud prediction system, a means testing system, and basically you throw a rock and you'll hit one of the big consultancies and the World Bank and somebody who's close to, to someone in power.

Alix: We're not advocating for throwing a rock at McKinsey. Um, but just, just to [00:44:00] be, just to make it clear. Wouldn't be

Gabriel: the worst thing.

Alix: No. But I think it's a really good point that there's these, like, structural actors at a large enough size and kind of bureaucratic affinity with states that end up getting called upon, and they're not necessarily the direct tech companies themselves, but they obviously are very predisposed to thinking of those tech companies as the solution to a lot of these problems.

Maria, do you wanna jump in? I don't know if there were any specific companies that stepped in when courts said, "Please help," or, "We're gonna do this thing."

Maria Pilar: I think there is more bottom-up approach because the courts are trying to design their own tools because there are some concerns about data protection and data handling, what the laws on data protections expects from the courts.

So again, I'm going to reference Argentina. There are some developments that has been made by the courts or by the courts and some private companies, but they are companies from Argentina, [00:45:00] very little ones. So maybe these tools are being developed from the bottom up because the judges maybe detected one problem and they say, "I want to address this question" and go searching for the people to help them to build something.

For example, there is a court in Buenos Aires that use a tool, uh, to anonymize. To use this tool for the sentence, it was developed by an NGO because they extract some data to contract violence- statistics, so it's very helpful. But I think that at some point, the big companies are going to get involved in how the judiciary are going to use these tools.

For example, last week I saw that you can integrate Claude with Word. So these tools are going to be used by the courts because most [00:46:00] of the courts use these tools in their daily work, so we have to be able to address these topics. So there is a more bottom-up approach

Alix: A lot of the challenges we're describing, so trying to scale a solution immediately before you understand a problem, or a human problem can't be solved by a technical solution, or a nation state has a unique responsibility to be cautious, um, and even if tech bro energy is coursing through your veins, you should, like, not necessarily act on it.

Um, like what would you, what would you wanna see? Like, what do you think would be a good sign that nation states were moving in the right direction when thinking responsibly about these technologies and the role they should or shouldn't play in governance?

Divij: One thing I'd really like to see is just reevaluating how we assess the impacts of these systems, right?

Moving away from just questions of efficacy, of savings, of scale, [00:47:00] towards really trying to kind of qualitatively consider the impacts on people's lives, taking into account this kind of wealth of qualitative experience and research that already exists, showing that in practice, systems come with substantial hurdles, um, and that simply putting a technical system in place is not going to overcome some of those particular difficulties that this causes problems.

And, and you know, I think that, like, a lot of these systems are put in place in good faith in many cases, in that we have severely overburdened public systems, severely overburdened judicial systems, crazy backlogs. They're being sold again and again by very, very wealthy companies as well as funders with particular ideological commitments.

This is the way in which you go forward. This is the way in which you can solve your problems. Um, and maybe, you know, one way is just to be able to consolidate some of the, the findings from your own context and, and present it to say that this is not really what's working for us, right? If you want to help this context, [00:48:00] this country, like take into account what's happening in the local context.

Change your metrics for what success looks like for a digital intervention towards things that actually start improving people's lives rather than things that are only based on scale and, and, and some notion of efficacy

Purity: As I was thinking about that question of like without thinking like this will never be perfect or possible, I would want, for example, if it's me or users to be explained, this is the amount you're contributing because of one, two, three, four.

That's what other insurance companies d- do. Like they ask you all these things you have. Do you have this preexisting condition? How many members of your family? And you understand why you're being charged this amount, which is different from another person. I would want to see like an explanation or transparency, but on the FOIA responses, they confirmed [00:49:00] that they're using like other government databases to integrate in this system.

And instead of raising all this suspicion and making all the other technical projects government will, will take on bringing distrust to them, why don't you be transparent and say you're slowly trying to create these surveillance hubs so that you can have such models, like try to have them more, more refined.

If the model being used is inherently incorrect, it doesn't matter how correct, how much data you bring in from other databases, it wouldn't still work. Like let's take time To develop a model that is sensitive to the realities of where we are

Gabriel: I mean, I, yeah, wanted to highlight what Divij said. I mean, I think a lot of these systems are often deployed in good faith, and these are really difficult, [00:50:00] intractable social problems that they're trying to solve, and there are no easy answers to these questions.

But because there's not an easy answer doesn't mean that the answer is to, you know, throw an AI model at it or throw, like, an algorithm at it. I mean, with this Kenya case, like, yes, the previous healthcare system was in shambles. It was like a dumpster fire, basically. But then what they're doing is basically, like, tossing the AI equivalent of a Molotov cocktail in there while screaming that it will be scientific and efficient and helpful.

And I think if I was talking to this person, what I would say is, like, oftentimes, you know, if I want to give them something really concrete that they can do, like, tomorrow, oftentimes the people in the room who are making these decisions are economists and computer scientists. I would tell them, like, bring in the sociologists, bring in the anthropologists, bring in the human rights lawyers, bring in trusted people on the ground in the community, and, like, have them at least be part of [00:51:00] these conversations in shaping your perspective and, like, the approach you take to these really difficult problems.

And I think oftentimes these types of people are excluded from these conversations because they make these conversations much more complicated. But I would say, like, okay, you, you have to embrace that if you want to sort of solve some of these issues. So that would be my like, you can do this tomorrow, like bring in these people tomorrow type, type suggestion.

Maria Pilar: I think that it's important to have or to build interdisciplinary communities to address these topics because, for example, in the topic of the judiciary, all lawyers are having opinions on the topic, and we are not hearing from the IT community, for example. So we think that we have all the solutions, and we need to listen to the people, uh, with the actual knowledge on technology to know if our problems could be solved in other ways.

And also, something that resonates with me is something that has been said during our conversations with the [00:52:00] judicial ecosystems is that we have to take into account the things that the people needs because normally or usually we don't listen from the people that is actually affected by the technologies.

For example, in the judicial case, if we don't know what the lawyer thinks, we don't know what the actual people that is being judged thinks about the using or the use of AI in the judiciary. So we need to integrate the common people in these discussions. So I think it's important to build communities to discuss and, and have conversations on AI and technology.

Alix: Thanks again to Divij, Maria, Purity, and Gabriel for joining us. This series was produced in collaboration with Real ML. For the past six years, Real ML has brought together people around the world working to challenge the power and inequities built into AI systems, not just through critique, but also through practice.

And many of the people you hear from in this series met [00:53:00] or developed their work directly through Real ML workshops where ideas are tested and collaborations are formed. And actually, last year, someone said soulmates were found. I've also served on Real ML's board, and it's one of my favorite communities, and I mean that sincerely.

I really love any time this group of people gets together because magic always ensues. And to learn more about Real ML and future workshops, you can check out the link in our show notes. A special thanks to Anna Bacciarelli, Isha Keegan, Nushabadi, and Shazeda Ahmed from Real ML. And thank you to our production team, Sarah Myles, Georgia Iacovou, Kushal Dev, Marion Wellington, Van Newman, and Zoe Trout