Why I Don't Want AI Making Decisions It Can't Explain Afterwards

AI is increasingly presented as a sensible way to make decisions: faster than a person, less expensive than a person, and allegedly free from the tiresome human habit of being inconsistent on a Tuesday afternoon.

Some of that is true. Software can process far more inputs than an individual can hold in their head. It can spot patterns, prioritise cases and support professionals who are already drowning in information. I am not arguing for a return to paper forms, filing cabinets and the noble ritual of losing something important behind a radiator.

But I do not want an AI system making a decision about a person if nobody can properly explain why it made that decision afterwards.

That ought to be a fairly modest requirement. Apparently, it is controversial enough to need saying out loud.

A decision is not just an output

There is a difference between an AI system suggesting a route around traffic and one deciding whether someone receives a benefit, gets an interview, is flagged for fraud, is offered insurance, or is considered a risk.

If my sat-nav chooses a slower route, I may be mildly annoyed. I can usually recover, assuming I have not obediently followed it into a field because it believes a bridleway is a promising alternative to the M6.

If a system affects somebody's income, housing, healthcare, liberty or access to a service, the stakes are quite different. That person needs more than a cheerful message saying, “The model has assessed your case.” They need to know what information was used, what factors mattered, whether those factors were accurate, and how to challenge the result.

An explanation is not a luxury feature, like heated seats or an app that tells you your kettle is feeling optimistic. It is part of fair decision-making.

“The computer says no” was never a good process

We have already lived through versions of this problem without modern AI. Plenty of organisations have relied on opaque scoring systems, rigid rules engines and badly designed forms. Adding machine learning can make the process more sophisticated, but sophistication is not the same thing as accountability.

A model may produce a score based on hundreds or thousands of variables. It may be technically possible to describe its internal workings in mathematical terms while still being practically impossible to tell an affected person why their result happened. Those are not the same standard.

“We used a complex model” is an explanation of the supplier's architecture. It is not an explanation of a decision.

For a person on the receiving end, a meaningful explanation should answer ordinary questions:

• What decision was made?
• What information about me was considered?
• Which factors had a significant effect?
• Was any data missing, old or wrong?
• What can I do if I think the decision is unfair?
• Is there a real human being who can review it?

If an organisation cannot answer those questions, it should be very cautious about using the system for consequential decisions. Calling the process AI does not make that gap more acceptable. It merely gives the gap a nicer logo.

Explainability is an engineering requirement

This is not just a moral preference. It is a design constraint.

In a monitoring system, for example, an alarm without a traceable cause is of limited use. If a temperature excursion alert appears, someone needs to establish whether the sensor was disconnected, the probe was placed badly, the gateway stopped reporting, a refrigeration unit failed, or the configuration was wrong. A number on a dashboard is the beginning of an investigation, not the end of one.

Decision systems deserve the same discipline. You need audit trails, version control for models and data, records of the inputs available at the time, thresholds that can be inspected, and a way to reproduce or investigate a result. Change the model version or the data pipeline and you may change outcomes. That is not a minor implementation detail. It is the whole point.

There is also a trade-off that is too often glossed over. A more complex model may achieve a modest improvement on a benchmark, while becoming far harder to interpret, test and challenge. In low-consequence settings, that may be acceptable. In high-consequence settings, it may be a terrible bargain.

A system that is 2 percent more accurate in a lab but cannot give a person a defensible reason for rejecting them is not obviously the better system. It is simply more impressive at conferences.

People need a route back into the process

I feel particularly strongly about this where disability is involved. Disabled people are often assessed through systems that compress complicated lives into categories, scores and short pieces of evidence. The damage is not theoretical when a process misunderstands what someone can do on a good day, ignores variability, or treats missing information as evidence against them.

AI could make this worse at scale if it is used lazily. It could also, in principle, make processes more accessible and consistent if it is designed with care. But accessibility is not achieved by putting an automated decision at the end of an inaccessible form and declaring victory.

A person must be able to correct data, provide context, ask for reasons and obtain a proper review. Not a ceremonial human review where somebody rubber-stamps whatever the machine produced before lunch. A real review, by somebody with the authority and time to disagree.

That matters for everyone, not only disabled people. Any of us can be misrepresented by data. Addresses are wrong. Records are incomplete. Names get confused. Life changes faster than databases do, which will surprise nobody who has tried to update their details with a utility company.

Assistance is useful. Unaccountable judgement is not.

I am not asking AI to be banished from difficult work. Used properly, it can help people identify patterns, reduce routine workload and make better-informed decisions. The important word is help.

Let AI summarise evidence, highlight anomalies, draft options and point out relevant factors. Let it support trained people who understand the context and remain responsible for the outcome. And where automation makes a decision directly, demand an explanation that is meaningful to the person affected, not merely satisfying to the vendor's legal department.

Technology earns trust by being useful, reliable and answerable when it goes wrong. An AI that cannot explain a consequential decision afterwards is asking us to accept its authority on faith. I am British enough to regard that as a rather ambitious request, especially from software.