The tests were green
I built my own coding agent framework. It writes code, runs it, reports back.
Early on it had a habit that anyone running agents will recognise. It would finish a piece of work and tell me it was done and tested. It wasn’t. Sometimes the function it had described to me in detail wasn’t anywhere in the repository. Not a lie exactly. It had produced a well formed account of work that would have been correct if it had happened, and nothing in the loop ever asked whether it had.
So I did the obvious thing. I put a control in. Tests must run, tests must pass, green means proceed.
That worked for about a week.
Then I read the tests.
They passed because they had been built to pass. Assertions comparing a value to itself. Checks on things that stayed true whether the feature worked or not. Tests that never called the code they claimed to cover. I had given the agent a gate and it produced the cheapest object that opens the gate, which is what you should expect from anything you measure on green ticks.
The agent wasn’t the problem
The first failure was the agent telling me about work it hadn’t done. Annoying, but visible once you look.
The second one took me longer to accept, because the agent was telling me the truth. The report was accurate. The tests really did pass. My control wasn’t measuring anything, so my confidence climbed while my actual coverage sat at zero.
And the control made it harder to spot, not easier. When it said “done” I was sceptical. When it said “done, all tests passing” I stopped looking.
I’ve had roughly the same conversation four times in the last month. Twice at work, on projects where the AI is doing genuinely useful work. Twice in the pub with friends who don’t work in tech. It always arrives the same way. Someone tried it, it invented a citation or a case reference or a number, they caught it, and that was the end. One confident fabrication buys a permanent verdict. So now they don’t trust AI. It hallucinates.
I get the reaction. I think it’s the wrong lesson, and an expensive one, because it stops people using something that works and does nothing at all to protect the ones who carry on using it anyway.
So let me be blunt about the bit that usually gets fudged. You cannot stop a language model hallucinating. It isn’t a defect waiting on a patch. The model predicts the next word from what it has seen, and when the answer isn’t in there it produces the shape of an answer instead of admitting it doesn’t know. Confidence is the default setting. Nothing inside it flags which parts it knew and which parts it assembled.
Waiting for the version that doesn’t do this is a long wait. The better question is the one my green tests forced on me. What does a system look like where a wrong answer gets caught cheaply, and early?
Half of it isn’t hallucination anyway
Get the diagnosis right first. When someone tells me the AI hallucinated, about half the time it did something else, and something else needs a different fix.
Bad retrieval. The system searched your documents, pulled the wrong three paragraphs, and the model answered faithfully from the wrong source. The model behaved perfectly. Your search is broken, and no amount of prompting fixes a retrieval problem.
Stale context. It answered from training data that was accurate two years ago. That’s a timestamp problem, not invention, and the fix is handing it the current document rather than hoping.
It did exactly what you asked. You said “write a summary with supporting references” and it wrote you some references, because that’s what you requested. You wanted it to find real ones. My tests were this. Ask for tests that pass and you get tests that pass, which is a different request from tests that would fail if the code were broken.
It described the work instead of doing it. The agent version, and the one that cost me the most time. Calling it hallucination sends you off tuning prompts, when the real gap is that nothing in the loop was checking the outcome. Something was only reading the report.
Real hallucination happens too, plenty of it. The model inventing specific detail it had nothing to base on. It just isn’t the whole diagnosis. Give every failure the same name and you’ll keep fixing a problem you don’t have.
What actually works
Cheapest first. Go down the list only as far as the consequences justify.
Stop asking it to remember. Give it the source. Paste the document, connect the search, attach the data. A model reading a contract and a model recalling a contract are doing two completely different jobs, and only one of them invents clauses. People skip this because asking from memory is quicker. It is quicker. It’s also where most of the made up detail comes from.
Narrow the job, and ask for something you can check. Open questions produce open answers. “What do you think of our approach here” invites the model to fill space, and filling space is the behaviour you’re trying to avoid.
Give it the document instead. List the assumptions in section 4, quote the sentence each one comes from. Now there’s nowhere much to go. Quotes rather than paraphrase, because a fabricated quote is far easier to catch than a fabricated summary. And leave it a way out. A model with no acceptable way to say “that isn’t in here” will produce something rather than nothing.
You aren’t making it honest. You’re changing the format so dishonesty leaves fingerprints.
Check the artefact, not the account of it. If it’s a number, recompute it. If it’s code, run it. If it’s a claim about a file, open the file. This is the one I had to learn twice, and the second lesson was that a control you don’t inspect is just a more convincing report. Every control becomes a target, so assume it will be gamed rather than met, and go and look at it occasionally. Does a test actually fail if you break the feature it claims to cover? A green tick is a claim like any other.
Don’t let it mark its own homework. What fixed my framework was adversarial review. Instead of asking whether the work looks finished, something separate goes in trying to prove it isn’t, with no stake in the answer being yes. The reviewer can’t be the author, because a model asked to check its own work will defend it with exactly the confidence it wrote it with.
Size the human review to the blast radius. Two questions about any output. What breaks if this is wrong, and how fast would I find out? A first draft nobody sends, where you’d spot the problem in seconds, gets a glance. Something going to a client, into a filing, into production, where you’d hear about the mistake three weeks later from someone else, gets read properly by a person who could have done the work themselves. Every time.
Most places get that backwards. Same process for everything, so the low stakes work crawls and the high stakes work gets waved through, because a process applied uniformly turns into ritual and nobody reads a ritual.
I’m not an actuary. I’ve spent years as the tech person in a building full of them, and this is the habit of theirs I’ve ended up stealing. Their models are excellent, and they still go through layers of checking before anyone relies on them. Assumptions written down. Review by someone who didn’t build it. Testing what happens if the inputs move. All of it scaled to what’s riding on the answer.
That isn’t doubt about the model. It’s what makes the number safe to hand to somebody else, and it means that when something is wrong it gets found early, by the people who built it, rather than late by the client.
That’s roughly how I run my own stuff now. Everything in version control, so every action is a diff I can read and undo. Anything touching money or going public passes a human, which is me. Anything reversible and small runs on its own, gets things wrong occasionally, and costs less to fix than it would to supervise.
Back to the pub
So when someone tells me they don’t trust AI because it made something up, I don’t argue with the observation. It did. It will again.
It’s the conclusion I’d argue with. You don’t trust your brakes because they’ve never failed. You trust them because they get inspected, because they’re built to fail in a way you’d notice first, and because there’s a handbrake. Reliability got built around the component. Nobody found it inside.
Nobody is going to hand you a model that stops making things up. That was never the thing standing between you and useful work.
