Actors’ views: Gemma Galdón Clavell

The Eticas Foundation recently conducted and published an external audit for the “VioGén” algorithm, a risk-assessment system used by the Spanish Ministry of Interior to prevent gender violence recidivism. Could you tell us more about the Foundation and the motivation behind this project? 

At Eticas, our mission is to protect people involved in technology processes, and we have been trying to do anything that we can in that direction. We have several programs that target civil society, policymakers, and market actors¹.

We launched an external audit project last year, where we engage with civil society organisations whose members may be impacted by automated decision-making (ADM) systems. When we don’t manage to get inside access to the algorithms in question, we work with those organisations to reverse engineer the ADM systems. We are externally auditing several systems in different domains, working with people with disabilities, with banking clients, on recidivism, ADM systems implemented in the public sector, health systems… We are trying to make this project as broad as possible.

We started with VioGén because it’s an audit we had been trying to work on internally for a long time but which we didn’t manage to carry out. We thought it was urgent to have more transparency on this system. We also kept seeing reports where women who were assigned as “low-risk” ended up being victims  of domestic violence again. We wanted to see whether that was a bug or a feature, whether the system wasn’t working well or whether the cases that we were seeing in the press were just exceptions… or whether this phenomenon was impossible to predict, because that’s also an issue. We wanted to have more data and more information, and urge the Ministry of the Interior to be more transparent about the system.

Is the impact on the individuals affected by these automated decisions sufficiently taken into account?

One of the issues that we have seen around artificial intelligence (AI) in ADMs is the complete disregard for end users and those affected by those systems. I think that as a society we have normalised that technology issues are very complex and obscure, that we shouldn’t get engaged because we don’t have the knowledge. As a result, technical issues are only discussed in technical rooms. We have normalised the fact that end users, but also oftentimes policymakers, don’t have access to the decisions that are being made, which is really striking since, at the end of the day, those systems work with data that comes from society. You can only build good algorithms if you understand society. We keep asking engineers to code a world that they don’t understand, making those systems worse, but also completely disregarding the interests and the input of end users.

That’s one of the things that we wanted to put on the table. That’s why we put the women we interviewed at the centre of this audit: to show that, actually, a lot of the data we gathered comes from talking to them. To show that if the proponents of these systems had taken the time to talk to these women, their system would have been better.

It is difficult to understand that a supermarket hires marketing experts to assess how clients may feel about their brand, about the shopping experience… but that we wouldn’t do something similar with sensitive systems such as this one, making sure we engage the people affected, talk to them, assess how the system is working, make them part of the design and the improvement process. The absence of this collaboration is remarkable in VioGén, but I would say that it’s remarkable for anything AI-related. That is really concerning when those systems take more and more decisions that are sensitive and have a huge impact on people’s lives and possibilities.

In its role as an independent auditor, how does the Foundation position itself vis-à-vis authorities responsible for ensuring that such tools do not infringe upon the fundamental rights and freedoms of individuals?  

I’m afraid we don’t really have oversight structures at the moment. We have institutions that are supposed to be overseeing how these systems proliferate, but they often don’t have the tools or the resources to do so. We have a huge implementation issue. We may have theoretical rights, but because no one is implementing them in practice, we are completely unprotected in real life.

At Eticas, we believe that auditing is one of the steps needed not just to improve the legal standing, but also the quality and transparency of ADM systems. I often say that for us, the audits are like seatbelts in cars. Many people say seatbelts have saved the most lives in the world only after penicillin. They are fixes to innovations. Fixes that make them better, more usable. Current AI systems are like cars that would run not only without seatbelts, but also without brakes. We are letting AI innovations hit the market and impact people without any kind of oversight or standards indicating what it means to do things well, what safeguards are needed to protect people. That is a huge problem.

Audits are a part of the ecosystem of regulation we need to develop, but they represent only one piece of the puzzle. If we manage to make audits into a standard, that is to say if any AI system that wants to hit the market and be used has to go through an audit whose results are made public I think we will have made an amazing step towards improving the transparency, accountability and responsibility of those systems, and towards increasing trust in them.

I’m a firm believer that AI has a lot of positive things to bring. I am just really frustrated when I see how badly we are doing it. It is just a matter of making sure that we combine innovation with safeguards. Right now, we only have innovation. Innovation without safeguards is profoundly dangerous and non-desirable from a societal perspective.

As shown in your audit, police officers tend to blindly trust VioGén’s risk-level decisions. What place should be given to human responsibility in the use of such devices?

Something we often find is that while we see really good intentions from what policymakers see as solutions – e.g. the “human-in-the-loop” concept in EU ADM systems – what we see in practice differs. Concerning human-in-the-loop”, this solution either re-introduces bias into the system, or allows for someone to neglect their own responsibility and deny they are ultimately responsible for those decisions.

More than a technological fix to societal problems, AI systems and ADM systems are becoming a way for humans to claim lack of responsibility: “the algorithm decided, so it’s not my problem if things go wrong”. This is problematic. When these women – or anyone – goes to the police, they are hoping that a trained person will make a decision, and not an automated system which they have no idea exists. The issue of responsibility around ADM systems is really crucial. Policy requirements like human-in-the-loop need to be better defined, so as to make sure that someone does not just say “yes” to an algorithm, but also makes a qualitative contribution to the system.

An easy fix we suggest is that whenever a police officer approves or disapproves over a VioGén assessment, he or she must justify this decision with a report. In that sense, it is them, with the support of the ADM system, that decide and take responsibility for the final assessment. We believe that making it as hard to approve a VioGén risk assessment than to reject it would also contribute to fairness. Right now, it is easier for a police officer to just accept the outcome. When they accept it, they don’t have to justify their choice. That is terrible, because an overworked police officer will have a tendency to just approve the algorithmic result, even when they don’t agree with it. They just don’t have the time to justify their decision.

The system needs to incorporate this step and also incorporate the time for police officers to make that report at the end of the process. The end goal is for the system to be purely a contribution to decision-making, not a decision-making system.

The future European regulation on artificial intelligence (AI Act) introduces a classification of such technologies according to their level of risk, especially with regard to their impact on the fundamental rights and freedoms of individuals. Do you think this can contribute to a better consideration of these aspects?

The distinction between high-risk, medium-risk and low-risk being presented by the European Commission is problematic, as already proven. The only good thing about this categorisation is that it includes “high-risk”, which means that some things that are currently being passed as low-risk (as if they didn’t have a societal impact) will have to go through increased controls. The problem is that a lot of things that are labelled as “low-risk” are actually higher risk. And vice versa. The lack of a complex understanding and definition of those terms will just lead to further problems.

At the same time, the legislation calls for strict oversight, whereas we already have strict oversight. It’s just not being implemented. Unless we address the implementation gap, we will continue to have the same problems. The General Data Protection Regulation (GDPR) already mentions that automated systems need to be justified, that we need to conduct a proportionality assessment, an impact assessment… and those things are not being done. I don’t understand how the regulator continues to trust words instead of planning for actions and facts.

So yes, indeed, if the incorporation of machine learning into VioGén makes it a high-risk system, and strict oversight is implemented, that would be a positive thing. Still, “machine learning” doesn’t mean much. We need to know whether the learning is supervised or not. The fact that machine learning may be incorporated, per se, could actually improve the system, if it’s supervised machine learning.

The regulation doesn’t have enough granularity for us technical people to implement it in ways that lead to actual oversight of those systems. We still have a long way to go to reconcile actual technical practices with what policymakers expect from ADM systems.

¹ Learn more about Eticas: https://eticasfoundation.org/about/