From Insight to Action: Why AI Agents Still Need a Governance Layer
AI Agent Governance: The 5 Essential Lessons From a Legal AI Disaster
A lawyer, working on a case in US federal court, Mata v. Avianca, turned to a general purpose AI chatbot for legal research after their usual research tool came up short. They treated it as if it were a reliable legal research tool. It wasn’t.
The chatbot generated entirely convincing case names, judicial opinions, quotations and citations, several of which simply didn’t exist. The material was submitted to court without being independently checked, and the lawyers continued to stand behind it even after opposing counsel and the judge couldn’t locate the cases being cited. The court found they had submitted fabricated legal opinions, failed in their professional responsibilities, and made misleading statements once the problem came to light. Sanctions and a financial penalty followed.
It’s a striking story on its own, but it’s also something more useful: a preview of exactly what goes wrong when AI output isn’t checked. Strictly speaking, this was a verification failure, a person copying and trusting fabricated output, rather than an autonomous agent acting on its own. But it shows exactly how much greater the risk becomes once AI can act rather than just answer, since the same unchecked confidence that slipped past two lawyers could just as easily slip past a human reviewing an agent’s output.
Most businesses will never end up in front of a judge over an AI mistake, but the underlying failure, confident output mistaken for verified output, plays out quietly in ordinary companies every week. As AI moves from simply answering questions to taking action through “agents”, AI agent governance is quickly becoming the question that matters most. The legal world’s brush with disaster offers five lessons that apply well beyond any courtroom.
What makes this moment different is scale. A single overworked professional cutting corners is a familiar, if regrettable, story. An AI agent making the same kind of mistake isn’t limited to one case or one afternoon, it can repeat the same flawed judgement across hundreds of customers, contracts or transactions before anyone notices, and it can do so with the same unwavering confidence every single time. That’s the real reason regulators, boards and customers are all starting to ask harder questions about how AI agents are actually supervised, not just what they’re capable of.
Why AI agent governance is suddenly everyone’s problem
Over the last couple of years, AI assistants have become a familiar part of everyday business software. They summarise documents, draft emails, build slides and answer questions about company data. Increasingly, they can also be extended with “agents”: configurable helpers that go and query specific data sources, such as a data platform or a business intelligence model, and bring the answer back into a chat window or a report.
A recent conversation with a specialist in this space, someone who has spent around a decade working with cloud data platforms, business intelligence tools and AI assistants, surfaced a set of insights worth sharing more widely. None of them are really about the technology itself. They’re about the gap between having access to powerful AI tools and actually getting reliable, safe, useful outcomes from them, which is exactly what good AI agent governance is meant to close.
Very few organisations consistently turn data into controlled action. Many still use BI primarily to understand and present information. The real gap is turning those insights into governed action through the right workflows, business rules and controls.
The more interesting opportunity isn’t producing another dashboard. It’s closing the loop: finding a relevant document, extracting the terms that matter, comparing them against an invoice or a transaction, spotting a discrepancy, explaining it in plain language, and then either taking an approved action or flagging it for a person to decide. That whole chain needs to leave behind an auditable trail of what was checked and why a decision was made, which is precisely where the courtroom story above went wrong: nothing was checked, and nothing was left behind to show that it should have been.
Picture the same dynamic playing out in an ordinary finance team. An agent is asked to reconcile a supplier invoice against a contract, flag anything unusual, and draft a query email if something doesn’t add up. Done well, that saves hours of manual checking every week. Done without governance, the agent might miss a clause, misread a currency, or simply invent a total that looks plausible, and nobody notices until the payment has already gone out. The technology capability is often already there. What’s usually missing is the layer that checks the agent’s own working before anything happens in the real world.
Lesson 1: Confident output is not the same as verified output
The lesson isn’t just that AI can occasionally get things wrong. It’s that generative AI can produce answers that look completely authoritative and well sourced, even when they have no factual basis at all. The chatbot in the courtroom story didn’t hedge, didn’t flag uncertainty, and didn’t warn that its case citations might be invented. It presented fiction with the same fluent confidence it would have used for fact.
That’s the trap. An answer that sounds confident and well referenced isn’t automatically an answer that can be trusted, and that distinction matters enormously once AI generated output starts feeding into real business decisions. A finance team asking an agent to summarise a supplier contract, a support agent drafting a refund policy answer, or a sales assistant pulling numbers for a board pack, all carry the same risk as the lawyer’s research tool: fluent, plausible, and wrong.
Put simply: information tools show you what happened. The next generation of tools needs to help a business understand why it happened, decide what should happen next, and safely make it happen, and every one of those steps needs a way to check the AI’s working before anyone acts on it. That checking step is the first and most basic requirement of AI agent governance, and it’s the one the legal disaster skipped entirely.
It helps to understand why this kind of mistake happens in the first place. Generative AI models are built to produce fluent, plausible sounding language, not to check facts against a verified database. When a model doesn’t know the answer, it doesn’t necessarily say so, it produces the most statistically likely sounding response instead, complete with confident phrasing, specific sounding details and, in the legal case, entirely invented citations formatted exactly like real ones. Understanding that distinction is the difference between treating AI output as a helpful first draft and treating it as an established fact.
Lesson 2: Access controls are not the same as governance
It’s tempting to think the hard part of deploying agents is technical: building or configuring an agent. In reality, Copilot spans Microsoft’s products and can be extended with specialised agents, but access to agents is not the same as having a governed operational process.
The real vulnerability isn’t a lack of security features. It’s that organisations have to configure their data, their permissions, their policies and their agents correctly, and that’s a much harder and more ongoing job than it sounds. An agent that has access to overshared information, or that can be nudged by a cleverly worded prompt into revealing something it shouldn’t, is a genuine and demonstrated risk. There are even public “capture the flag” style games built specifically to show how easily this can happen, where players compete to trick an agent into leaking data it was never meant to share.
Letting agents loose and hoping for the best simply isn’t a viable strategy once real company data, real customers or real financial decisions are involved. This is exactly the kind of risk that structured frameworks such as the NIST AI Risk Management Framework are designed to address, and it’s a big part of why AI agent governance has become a board level topic rather than a purely technical one. Ticking the box on permissions is necessary, but it was never meant to be the whole answer, and treating it that way is how organisations end up with agents that are secure on paper and risky in practice.
The practical fix is to test an agent the way a security team would test any new system before it goes live: try to break it on purpose. That means checking what happens if someone asks it a question it shouldn’t answer, feeds it a document containing hidden instructions, or asks it to combine two pieces of information that individually seem harmless but together reveal something sensitive. An agent that passes that kind of deliberate stress test before launch is a very different proposition to one that’s simply switched on and monitored after the fact.
Lesson 3: Most organisations do not yet have the skills to build this well
There’s also a more human, less technical gap. Plenty of people understand roughly what an AI agent could do. Far fewer know how to:
- Define a genuinely valuable use case for one
- Structure the underlying information so an agent can use it properly
- Build and configure the agent itself
- Assign appropriate permissions
- Decide where a human needs to approve or intervene
- Test what it produces and monitor it once it’s live
That’s a meaningful competence gap, and it’s not solved just by giving people access to agent building tools. It’s the same gap that let a fabricated legal citation slip past two experienced lawyers: the tool was available, but the discipline to test and verify its output before relying on it wasn’t there.
There’s a useful test for whether an agent is worth building at all: if one especially knowledgeable person in the business fields forty specialist questions a month while everyone else asks two a year, an agent that just imitates that one person may be solving too narrow a problem. A stronger use case is usually an agent that helps that expert make their knowledge safely and consistently available to everyone else, with the same checks that expert would apply themselves built into the process.
Closing this gap doesn’t require hiring a team of specialists overnight. It usually starts smaller: picking one well understood, lower risk process to automate first, pairing the person who knows that process inside out with whoever is building the agent, and treating the first version as a pilot to learn from rather than a finished product. Businesses that build this muscle gradually, on contained problems, tend to make far better decisions later when the stakes, and the potential for an AI generated mistake, are higher.
Lesson 4: Full autonomy does not exist yet, and pretending otherwise is dangerous
There’s a broader cultural expectation at play too, a hope for something like a self-driving car or a self-playing piano, where you set it going and it just works, indefinitely, without supervision. Dependable autonomy at that level doesn’t really exist yet, at least not in the way people imagine, and the courtroom disaster is a direct result of someone acting as though it did.
Agents still need people to define the outcome they’re aiming for, provide context and organisational knowledge, decide what the agent is and isn’t allowed to do, supervise the higher risk actions, and correct mistakes as they happen. Access to a capable AI platform isn’t the same as having a structured approach to deciding where agents genuinely add value, and where they don’t. Every business currently experimenting with agents is, in effect, deciding how much supervision is enough, and getting that judgement wrong is exactly how a confident, well written, entirely false answer ends up in a court filing, a customer email, or a financial report.
A practical way to think about this is in tiers. Low risk actions, like drafting an internal summary for a colleague to review, can reasonably run with light oversight. Medium risk actions, like sending a routine customer update, might warrant regular spot checks. High risk actions, like anything touching money, contracts, legal filings or customer commitments, should always have a human review step before anything is final. The legal disaster sat firmly in that last category, and was treated as though it belonged in the first.
Lesson 5: AI agent governance is a structured layer, not another dashboard
Put these threads together and a pattern emerges. The opportunity isn’t simply “more agents” or “another dashboard.” Good AI agent governance is a structured layer that sits between raw enterprise information, general purpose AI capability, and the actions a business actually needs to take, one that can:
- Understand documents and unstructured information, not just tidy databases
- Extract the specific facts that matter, such as contract terms
- Ground any AI-driven output in defined business rules
- Check outputs and proposed actions before they go anywhere
- Decide when a human absolutely needs to be in the loop
- Keep a clear record of what was decided and why
- Actually carry out approved actions in existing systems
None of this replaces the analytics or productivity tools already in place. It complements them, by adding the governance, verification and controlled-action layer that turns insight into something a business can safely rely on. With that layer in place, and source verification made mandatory, the fictional citations would have been stopped before they reached the court.
In practice, that layer tends to look less like a single piece of software and more like a set of habits backed by the right tooling: every agent action logged in a way a person can review later, a clear rulebook for what an agent is and isn’t allowed to do without sign off, a named person accountable for each agent’s outcomes, and a routine check on whether the agent is still doing what it was originally built to do. None of that is exotic. It’s the same discipline businesses already apply to financial controls, just extended to cover a new kind of colleague.
A handful of phrases from that original conversation capture the underlying theme rather well:
Very few organisations take action on the data they already have.
The agentic world needs a new layer of trust, control and assurance.
Agent deployment can’t be treated as uncontrolled experimentation when company data, customers and financial decisions are involved.
As agents start supervising processes, and other agents, organisations need assurance over the agents themselves, not just the humans.
Correct outcomes come from structure: governed information, clear rules, controlled actions and accountable human oversight.
AI assistants and agents are only going to become more embedded in everyday business tools. The organisations that get real value from them won’t be the ones with the most agents, they’ll be the ones that have invested in proper AI agent governance, learned the five lessons above, and made sure their own version of that courtroom disaster never gets the chance to happen. The technology will keep improving, but the discipline of checking, supervising and taking responsibility for what an agent does is a business decision, not a software feature, and it’s one every organisation using AI agents needs to make deliberately rather than by accident.
Comments are closed