There is a dial in every operation that adopts artificial intelligence. On the left, a human makes the decision and the machine assists. On the right, the machine makes the decision and the human assists — or isn't in the loop at all. Almost nobody sets that dial on purpose. It drifts.
It drifts because the people paid to install your AI are paid to turn the dial up. More automation is faster, cheaper, and easier to sell as progress. Every vendor demo ends with the same implicit promise: let the system handle it. And in most cases the system genuinely can handle it — most of the time.
The problem is the rest of the time. When an AI-driven decision goes wrong in a high-stakes setting — a missed hazard, a wrongly denied claim, a defect that shipped, a navigation call a human used to make — someone has to answer for it. To a regulator. To an insurer. To a plaintiff's attorney. And when that moment comes, the question is never "was the AI good?" The question is "who was accountable, and can they show it?"
This is not a technology problem. It's a controls problem. And that changes who is qualified to solve it.
The optimization–accountability trade-off
Two forces pull in opposite directions the moment AI enters a serious operation.
Optimization wants the human out of the way. Human review is the slowest, most expensive, least scalable part of any workflow. The entire economic case for AI is built on removing it. This is a legitimate goal — the efficiency is real and the savings are real.
Accountability wants the human kept in. Not for sentiment, but because accountability is a legal and practical construct that attaches to people, not code. A regulator sanctions a person or a company. A jury assigns fault to a defendant. An insurer prices the behavior of the insured. None of these mechanisms know how to hold a model responsible — they reach through the model to the humans who deployed it and relied on it.
The company that builds or optimizes your AI is incentivized to turn the autonomy dial up. Nobody in that room is incentivized to ask how far up you can defend. That question needs to be asked by someone whose job is not to ship the system — and whose credibility rests on being right about risk.
This is why "we bought a good tool" and "we are protected" are not the same statement, and why the gap between them is where the liability lives.
The autonomy dial, honestly read
It's tempting to say "AI is making the decisions now." In most operations that's not yet literally true — and overclaiming it will get you caught by exactly the sophisticated buyer you want. The more accurate and more unsettling picture is subtler.
- AI assists — the human decides, the AI surfaces information. Low autonomy, clear accountability.
- AI recommends — the AI proposes, a human is supposed to approve. This is where most "safety" and "quality" systems actually live.
- AI recommends, human rubber-stamps — the human is technically in the loop but defers to the machine every time. This is the dangerous middle, and it's the most common real state.
- AI decides — the system acts and a human reviews exceptions, or nobody reviews at all. High autonomy, accountability in question.
The quiet risk is not the far right of the dial. It's that dangerous middle — the setting where a human is nominally accountable but has, in practice, stopped exercising judgment. On paper there's oversight. In reality there's deference. That drift is invisible until an incident forces someone to reconstruct who actually decided — and the honest answer is "the model, and nobody checked."
Even where AI only recommends, teams defer to it as if it decided. That quiet drift — not full autonomy — is the ungoverned risk in most operations today.
Where accountability goes — and how to get it back
When you let the dial drift, accountability doesn't disappear. It scatters. And in a dispute, a scattered accountability trail is indistinguishable from negligence.
| When AI makes the call… | The accountability question becomes… |
|---|---|
| A hazard is flagged and not addressed | Who was supposed to act, was there a protocol, and does the record show a known risk was ignored? |
| The model's thresholds were set by the vendor's defaults | Can anyone credibly testify to why the settings are what they are, as a reasonable decision? |
| The system quietly degrades (model drift) | Who is monitoring reliability, and how would you even know it stopped working well? |
| A human "approved" the AI's output | Was that review real and documented, or a rubber stamp with no independent judgment? |
Getting accountability back is not about turning the dial down or distrusting the technology. It's about making the setting defensible: a documented decision, a real human in a real loop, a record that helps you rather than convicts you, and an independent party who can attest that the governance is genuine. That's not friction added to the system. It's the thing that lets you rely on the system safely.
The party that builds or optimizes your AI cannot also be the party that validates it — for the same reason a company doesn't audit its own books. Independent governance oversight isn't a tax on the project. It's the assurance your insurer, your regulator, and your eventual defense counsel all want to exist. Someone has to hold the chair whose only job is to ask: how far up is the dial, and can you defend it?
Why a forensic controls lens
The discipline of asking "who is accountable, is there a control, and can you prove it operated?" is not new. It is the core of forensic accounting and fraud examination — professions built entirely around evidence trails, control failures, and the gap between what an organization believed was happening and what the record actually shows.
AI in a safety-critical or liability-heavy operation is simply a new place for that same gap to open. The tools are novel. The governance question is ancient: can you demonstrate that a responsible human was in control of a consequential decision? If you can't, it doesn't matter how good the model is.
The series ahead
This piece frames the problem. The ones that follow put the autonomy dial into specific, high-stakes settings where getting it wrong is expensive:
- Texas non-subscriber employers — where AI safety systems can quietly manufacture the evidence a plaintiff's attorney needs.
- Manufacturing quality and safety — where a vision system that "catches defects" degrades silently until someone catches the vision system.
- Maritime navigation and training — where decision-support AI makes calls a bridge officer used to own.
- Independent oversight — the case for separating the party that builds the AI from the party that governs it.
Find out where your dial is — before someone else does
The AI Governance Readiness Assessment is a focused, independent review: no downtime, no code access, no committee. I look at the decisions your AI is making, the human oversight around them, and the record it's leaving behind — then hand you a short findings report that rates your exposure and names the smallest set of fixes that make the system defensible.
Independent by design. I don't build or sell these systems, so I have no stake in telling you yours is fine. That's the point — it's the same reason you don't let a company audit its own books.
Request an assessment