An AI toolkit for uncovering hidden biases in police algorithms

An AI system can analyse a million clues in the blink of an eye. But what if the most important clue is actually a hidden bias? PhD candidate Marcel Robeer has developed a software tool that helps police officers check algorithms for unfairness.

Robeer conducted his research at the National Police Lab AI, where he was employed both by the police and the university during his PhD. The computer scientist will defend his thesis, Beyond Trust: A Causal Approach to Explainable AI in Law Enforcement, on 14 November.

Artificial intelligence is increasingly used in decision-making in critical contexts such as policing and the judiciary. While AI can analyse vast amounts of data, it is often unclear why a model reaches certain conclusions. If hidden biases are present, the model can produce unfair or incorrect decisions by detecting patterns that aren’t really there.

To reduce AI’s ‘black-box’ nature, there is growing attention for explainable AI (XAI): techniques that clarify how algorithms arrive at their decisions. But according to Robeer, many explanations lack depth. “Explainable AI often shows which correlations exist, but not why they exist. In contexts where evidence and accountability matter - such as in policing - you cannot blindly trust AI recommendations. If AI-derived evidence is used in court, it needs to hold up under close examination.”

In policing and justice, human lives are at stake, so standards must be high

Robeer focuses on natural language processing (NLP), a field in which AI interprets human language. Text is often ambiguous and context-dependent, making it difficult for AI systems to interpret meaning accurately. “Especially in high-risk environments like policing, it’s crucial to judge how much you can trust an AI system.”

The researcher developed several techniques to reveal which features an opaque algorithm weighs when making decisions. “This allows you to identify which factors are valid, which carry potential bias, and how sensitive the model is to small changes.”

One of these techniques is the ‘what-if machine’, which generates realistic alternative examples to test whether the AI system would reach a different conclusion. Robeer explains: “Imagine a message says: ‘I’ll DESTROY you!’ and the system flags it as a threat. Does it react the same if the message says ‘I’ll DESTROY you tomorrow!’ or if it’s written without capitals or an exclamation mark? If not, there’s an issue in the model.”

Digging into your AI model

Robeer also developed a software tool for police staff to assess AI systems for fairness and transparency. “Tools like this already existed, but they were highly technical and difficult to use. Explabox is a kind of toolkit that combines the methods I’ve developed. It allows police data scientists to dig into their AI models to see whether they work in a fair, reliable, and understandable way.”

During his PhD, Robeer observed that the police are still cautious about using AI. Rightly so, he says. “In policing and justice, human lives are at stake, and standards must be high. AI models can certainly add value as supporting evidence, but no case is built on them alone. I believe this is an interim phase, and the use of AI will increase. When that happens, it’s vital to use robust models and include checks and balances.”

Now working as an AI oversight officer at the Ministry of Defence, Robeer hopes his research will help police professionals assess whether they can trust their AI systems - and, perhaps even more importantly, whether we as a society can trust the police and the way they use AI.”