In 2025, a government efficiency effort deployed an AI tool to review federal agency contracts — vendor and service agreements — and flag them for cancellation. Independent experts who later reviewed the tool's code found it was error-prone: it hallucinated contract values and made glaring mistakes. The episode drew a Senate investigation.
Set aside the politics of it entirely. What's left is a near-perfect governance teaching case, because every failure mode you worry about with high-stakes AI showed up in one episode: an autonomous-ish tool making consequential decisions, on messy real-world data, with no visible validation layer, no documented human-in-the-loop, and no one clearly accountable for the errors until after they'd already landed.
An AI system was pointed at a large body of complex documents and asked to make or recommend decisions with real financial and operational consequences. It got some of them wrong in ways a competent human reviewer would have caught immediately — wrong dollar figures, misread terms. The errors weren't discovered by the system or its operators. They were discovered by outside experts, after the fact, reading the code. That is the exact sequence a governance layer exists to prevent.
What a controls lens would have asked first
A forensic-controls review doesn't start with "is the AI good?" It starts with the questions that determine whether you can defend the output when it matters. Applied to this case, four of them are damning:
| Governance question | What the episode revealed |
|---|---|
| Was there a validation step before the AI's output was acted on? | Contract values were hallucinated and reached decision-making without an evident check against the actual source documents. |
| Was a competent human in the loop — really, not nominally? | Errors a human reviewer would catch on sight weren't caught, suggesting the human review was absent or perfunctory. |
| Who was accountable for an error before it happened? | Accountability surfaced only after outside experts and a Senate inquiry — i.e., after the damage, not before. |
| Was the tool tested and its reliability documented? | The error-prone behavior was found by others reading the code later, not established by the operators up front. |
None of these are AI-engineering questions. They're controls questions — the same ones a forensic accountant asks of any process that touches money or consequential decisions. The presence of an algorithm doesn't change the discipline. It raises the stakes, because the system can produce wrong answers faster and at greater scale than any human, and it does so with a veneer of confidence that discourages the very scrutiny it most needs.
The failure here wasn't that the AI made mistakes. All systems make mistakes. The failure was that there was no layer designed to catch them before they became decisions.
The autonomy dial, turned too far with nothing underneath
Every AI deployment sits somewhere on a dial from "human decides, AI assists" to "AI decides, human reviews exceptions — or nobody does." The efficiency case for AI always pushes that dial toward autonomy: faster, cheaper, less human bottleneck. That pressure is legitimate. What makes it dangerous is turning the dial up without building the controls that make higher autonomy defensible.
In this episode the dial was turned high — the tool was making or driving cancellation recommendations at scale — while the governance underneath it (validation, human review, reliability testing, clear accountability) appears to have been thin or absent. High autonomy plus thin governance is the single most expensive configuration in AI deployment, because it fails silently and at scale until something external forces the reckoning.
You don't have to be a federal agency for this to apply. Any organization letting AI make or drive consequential decisions — which contracts to cut, which claims to deny, which transactions to flag, which vehicles to service — is running the same risk if the same four controls are missing. The scale differs. The failure mode is identical: wrong outputs, acted on, with no one accountable until after the fact.
Why the builder can't be the one who validates
There's a structural lesson buried in how these errors surfaced: they were found by independent experts reviewing the code, not by the people who built or ran the tool. That's not a coincidence — it's the general rule. The party that builds or deploys an AI system is the least well-positioned to catch its failures, for the same reason a company doesn't audit its own books. They're invested in it working, they share its blind spots, and they have every incentive to report that it's fine.
Independent validation isn't bureaucratic friction. It's the mechanism by which errors get caught before a Senate investigation, a regulator, or a plaintiff's attorney catches them for you. The cost of building that layer is almost always trivial next to the cost of the reckoning that arrives without it.
Is your AI turned up further than your controls can defend?
The AI Governance Readiness Assessment is a focused, independent review built to answer exactly the four questions above for your deployment: is there a validation step, is a human really in the loop, who's accountable before an error, and is the system's reliability documented. No downtime, no committee — a short findings report that rates your exposure and names the smallest fixes that make the system defensible.
Independent by design. I don't build or sell these systems, so I have no stake in telling you yours is fine — which, as this case shows, is exactly the seat that catches what the builder can't.
Request an assessment
This article is an independent governance analysis based on public reporting and Congressional inquiry as of June 2026. It is informational, not legal advice, and takes no political position — the episode is discussed solely as a governance case study. Monte Fisher's CPA license is retired and inactive.