Who's Accountable When AI Gets It Wrong? The Human Oversight Paradox in Financial Services AI

CTRL AI DEL is Dailoqa's weekly, no-nonsense look at what's actually happening with AI in financial services, minus the hype and the jargon. This issue looks at AI accountability in financial services, and why human oversight is not automatically the safe option it sounds like.

AI accountability in financial services is not solved by simply keeping a human in the loop. Human oversight fails too, through distraction, fatigue and automation bias, so the real answer is not defaulting all control to people, it is designed symbiosis between human judgement and machine consistency, with adjustable control built in from the start. 

Who's Watching the Watchers, and the Watched

AI guardrails have become the topic of the moment in every boardroom, and nowhere more so than in the tightly regulated world of financial services. Everyone is working out how to keep intelligent systems from making costly mistakes, and the stakes could not be higher. As a colleague once put it, "No AI will be sent to jail for getting it wrong, but a person could be for trusting it." That single line captures the real tension at the heart of this debate.

This is not an abstract worry either. Earlier this month, India's Supreme Court set aside a tribunal order that had relied on case judgments which turned out to be non-existent, apparently generated by AI. Nobody had designed a rogue judgment machine on purpose. Someone simply trusted an output without checking it, and that single unchecked step is exactly the gap every financial services board is now racing to close.

The Infallibility Myth, Human vs Machine Edition

The case for human guardrails is a reasonable one. Humans bring judgement and common sense, qualities that remain out of reach for even the most advanced algorithms. But the argument only holds up so far. Machines are prone to amplifying bias or generating convincing but false information. Humans are prone to stress, fatigue and simple inattention. Neither side of this equation is infallible, which raises an uncomfortable question, who is actually best placed to monitor whom? It is closer to a chicken-and-egg problem than most governance frameworks like to admit, and the financial consequences of getting it wrong are real.

Academics studying this from the boardroom down have landed on a similarly uncomfortable answer. Traditional accountability flows top down, from executives to managers, but AI's black box nature breaks that chain, and pinning the blame solely on developers or solely on users both turn out to have real limits. Shared accountability sounds sensible until you remember that shared accountability, spread thin enough, quietly becomes no accountability at all.

Dialling Up or Down the Trust Factor

It is entirely natural to approach a new technology cautiously, particularly when people feel they have limited visibility into how a decision was reached. This is often described as the "black box" problem, where outcomes are produced by processes that are difficult to see or explain. It is precisely why we favour the idea of a dial in agentic design. Teams that want tighter control can dial automation down. Teams that are comfortable extending more autonomy to the system can dial it up. The point is control that can be calibrated to the risk of the task, not a single fixed setting applied everywhere.

Multi-agentic design also promises more transparency and explainability. Deterministic agents are built not to hallucinate, gathering information from selected sources rather than inventing an answer when the honest response would be "I don't know." That is a meaningful shift from systems that are difficult to audit after the fact.

The Perils of Human Safety Measures 

It is worth pausing on the human element too. Who monitors the humans? History shows that human safety measures are not the impenetrable safeguard they are often assumed to be. In one widely reported self-driving car accident, the designated safety driver was found not to have been paying attention at the critical moment. In aviation, despite decades of technology built to support pilots, human error remains a leading contributing factor in incidents. People get distracted, fatigued or simply complacent, and no amount of training fully removes that risk. 

One of the most persistent challenges is the human tendency to accept an answer at face value, whether it comes from automation, convenience or cognitive bias; the tendency to trust information that is easiest to access or most familiar. Why interrogate an answer when a plausible one is already in front of you? It is a comfortable habit, and a risky one

There is a name for this: automation bias, and the research is not flattering. People consistently defer to automated systems even when those systems are visibly producing flawed results, and that tendency gets more dangerous, not less, the more sophisticated the system appears. A confident-sounding machine is, if anything, easier to trust blindly than an uncertain one.

The Symbiotic Solution

So where does this leave us? In our view, guardrails are essential. Effective oversight comes from humans and machines working in tandem, each covering the other's blind spots. Machines are exceptionally good at pattern recognition and can flag anomalies at a speed and scale no human team could match on its own. Humans bring judgement, context and empathy that no algorithm can replicate. An algorithm can optimise for efficiency. It cannot weigh the human cost of a bad outcome.

Ask an organisation who actually owns the outcome and the honest answer, according to researchers tracking this across enterprise AI deployments, is that accountability rests with whoever deployed the system, not the system itself. IBM's own framing puts it plainly too: when it comes to AI-assisted decisions, there is rarely a hard and fast line, only a moving target shaped by risk, reward and who is willing to own the call.

The goal is not to replace one infallible system with another, since neither exists. It is to build a genuine socio-technical system where the strengths of AI and human judgement are combined, and where each side's weaknesses are mitigated through deliberate design, rigorous training and continuous review. That means blending machine precision with human judgement and ethical reasoning to produce a safer, more accountable outcome overall.

So, who's monitoring whom? The answer, it seems, is everyone, and everything, all at once. And perhaps, just perhaps, that's precisely how it should be.

FAQ on Who's Accountable When AI Gets It Wrong

Who is liable when AI makes a wrong financial decision?

There is no single settled answer yet. Liability tends to sit with whoever deployed the system and acted on its output, rather than with the AI itself, but courts are still working this out case by case.

Does meaningful human control actually make AI safer?

Only if the human doing the controlling is genuinely paying attention. Human oversight fails through distraction, fatigue and automation bias just as often as machines fail through hallucination or bias.

What are AI guardrails in financial services?

Guardrails are the combined set of controls, both human review and machine checks, that keep an AI system's actions within agreed limits and keep its decisions explainable.

Is human review of AI outputs actually reliable?

Not on its own. People tend to defer to confident-sounding automated answers even when those answers are wrong, which is exactly the failure mode oversight is supposed to prevent.

How do you design AI systems with adjustable autonomy?

By building in a dial rather than a switch, letting a team turn automation up for low-risk, well-defined tasks and down for anything that needs tighter human control.

What is human-in-the-loop, and where does it fall short?

It means a person reviews or approves an AI action before it takes effect. It falls short when that review becomes a rubber stamp, because the human trusts the machine more than they question it.

Why does AI hallucination matter more in banking than elsewhere?

Because a fabricated figure or fictitious reference in a regulated decision can trigger real financial, legal and reputational consequences, not just an awkward correction.

Glossary

• AI Accountability, the question of who is responsible when an AI-assisted decision causes harm or turns out to be wrong.

• Human Oversight, a person reviewing, approving or monitoring an AI system's actions or outputs.

• Meaningful Human Control, oversight that is substantive enough to actually catch and correct errors, not just a formality.

• AI Guardrails, the combined human and machine controls that keep an AI system operating within agreed limits.

• Automation Bias, the tendency to trust an automated system's output even when it is visibly wrong.

• Deterministic Agent, an AI agent built to produce consistent, rule-bound outputs rather than open-ended generation, reducing the risk of hallucination.

• The Dial, Dailoqa term for adjustable autonomy in agentic design, letting a team turn automation up or down depending on the risk of the task.

References

1. Business Standard, AI is shaping decisions, But who is liable when it gets it wrong?, July 2026. https://www.business-standard.com/industry/news/artificial-intelligence-misinformation-decision-making-ai-accountability-126070600416_1.html

2. California Management Review, Critical Issues About A.I. Accountability Answered, November 2023. https://cmr.berkeley.edu/2023/11/critical-issues-about-a-i-accountability-answered/

3. Forbes, Who's Accountable When AI Makes Decisions and the Law Says No One Has to Answer?, February 2026. https://www.forbes.com/sites/matthewerskine/2026/02/19/whos-accountable-when-ai-makes-decisions-and-the-law-says-no-one-has-to-answer/

4. VKTR, Who Is Accountable When AI Gets It Wrong?, July 2026. https://www.vktr.com/ai-ethics-law-risk/when-ai-gets-it-wrong-who-pays/

5. IBM, AI decision-making, where do businesses draw the line?, 2026. https://www.ibm.com/think/insights/ai-decision-making-where-do-businesses-draw-the-line