Software Engineering

I Vibe Coded a Game in a Weekend. Writing the Code Was the Easy Part.

by
Offline Arena: a cartoon soldier in an orange-striped helmet takes cover behind a parked car on a city street, with two bots approaching from the park

I was flying back from a work conference in Berlin on a budget airline. No entertainment, no Wi-Fi. Usually I’d use a flight to catch up on a film, or plug the laptop in and work. On this one I could do neither.

The colleague sitting next to me had an offline game. I had nothing. Try as I might, I couldn’t help noticing them busy playing away on an arcade puzzle the whole way home.

So, out of frustration, the first thing I did when I got home was build a game. Free to play. No ads. Works offline.

It was inspired by a top-down shooter I played years ago. I can’t remember its name, it must have been pulled from the App Store, and I can’t play it any more. But I remember the bits I liked. I wanted those, plus the best bits of all the best games I remember from the last 20 years, to make something new.

That Saturday I asked Claude to build it.

By Sunday night it had four maps, night and rain, a training mode, sound and music, unlockable skins, an offline mode, and my family playing it on their phones and a tablet. There’s a link to play it at the end.

That isn’t the interesting part. Anyone can get an AI to write a game now.

I’ve built an agentic developer harness, FADE, for exactly this kind of work. I wrote about it here. This time I deliberately didn’t use it. I wanted to vibe code it, to see what difference that makes.

What I found is that almost none of what made the game good was the code. It was the engineering discipline, and it crept in even without the framework.

Writing code took minutes. Knowing it was wrong took telemetry.

The first version worked, and it played well enough. If I’d stopped there I’d have told you AI can build a game in an afternoon, and I’d have been wrong about what I had.

On the first evening the game started sending one small record per match back to my server: how long you lasted, how you died, what gun you had, how accurate you were. A random ID per phone, no names, no location. The start screen says so.

Before anyone played, Claude ran a test match with a player who never moved and never fired. It came 5th out of 20. The bots were so busy killing each other that doing nothing was a strong tactic. No amount of reading the code would have told me that. One test did.

Then my family played 37 matches. The numbers were brutal. On one phone, 18 of 28 finished matches ended in 20th place, most of them between 4 and 16 seconds in, usually with a nearly full gun. Nobody told me that. They just kept pressing Play again. The data said it.

The cause turned out to be a bug. The bots hold fire for the first 12 seconds of a match. The player’s auto-fire didn’t. So every new player started a gunfight they couldn’t win, against a bot that had looted something better. You’d never spot that by playing.

The same thing happened again and again over the weekend:

  • My son Luca kept coming last and said he couldn’t aim. “I know I have to face straight, I don’t know exactly what angle.” I suggested a laser pointer from the gun. Later, when the data showed new players dying in the first few seconds, we added a short training level that runs the first time you press Play.
  • On the Fire tablet at home the game felt sluggish. The telemetry put a number on it: about 12 to 15 frames a second, slow enough that the whole game ran in slow motion. Now the game notices and lowers its own resolution.
  • On the Monday I complained that the forest map, meant to be the easy one, felt like the hardest. The data agreed it had the worst results. A simulation said the map wasn’t the problem: most of those matches came from our weakest player on that slow tablet.

My family tell me how it feels. The data tells me why. You need both, and the second one you have to build on purpose.

No single AI was in charge

I didn’t use one AI. I used tools from two companies, and the most useful thing any of them did was catch another one’s mistake.

It helps to keep three layers apart. There’s the company: Anthropic or OpenAI. There’s the harness, the tool you actually work in, which reads your files, runs commands and calls the model: Claude Code is Anthropic’s, Codex is OpenAI’s. And there’s the model, the brain inside, which comes in tiers. Anthropic’s Claude models run from Haiku through Sonnet and Opus up to Fable. OpenAI’s GPT-6 models run from Luna through Sol up to Astra.

Here’s who did what:

  • Claude Code with Opus 5.5, my daily driver, did the building: the engine, the bots, the maps, the deploys.
  • Claude Code with Fable 5.1, Anthropic’s top model, did the adversarial reviews. I asked it to find the bugs, the logic flaws and the design problems, and to change nothing.
  • Codex with GPT-6 Astra, OpenAI’s top model, did the artwork for the desert, city and neon menus and wired it into the game, and gave me the brutal second opinions, including a release review.

One habit I always keep. When a top model breaks work down, I have it write the brief as if a smaller model will do the work. A bigger model can infer things and figure stuff out. A smaller one won’t, and it will make mistakes, so it needs explicit paths and explicit instructions. Fable’s review came with a fix brief written for Sonnet, the middle tier. If I’m honest, I should have aimed it at Haiku. Then I gave it to Opus to build anyway. I’d been away for a week and hadn’t spent my token budget, so I used it.

The reviews earned their keep. Astra’s first review found that a win could turn into a defeat, because damage kept landing after the match had ended and two end-screen timers raced each other. Opus checked every line reference before fixing anything. All of them were right. Then, while fixing them, Opus introduced a crash of its own on loading. The test run caught it before it reached a phone.

Fable’s review found that one error anywhere would freeze the game for good, and that on older iPhones the City map threw one on its very first frame. It found that about 2 in 100 forest and desert matches ended with the final safe circle somewhere nobody could stand.

Astra later did a release review and refused to sign it off for the App Store. Fair enough. It listed the reasons, Opus checked each one against the code, and the real ones got fixed.

Then I asked Fable the question I actually cared about: would a human developer look at this and see good engineering, or vibe coded slop?

Its answer was more useful than flattery. As a two day prototype, it said, a developer would call it good work: decisions written down with dates and names, commit messages that say what changed, a deploy that checks the live copy byte for byte, telemetry with a report. “None of that is typical of slop.” As the codebase for an App Store product, it isn’t there yet: one 1,300 line file where everything shares the same 62 variables, 37 empty error handlers, and no committed tests until the end of the first day.

Both halves are fair, and I’d rather hear it now than from a developer later.

What I took from it: an AI checking its own work is weak. A different AI checking it, with instructions to find problems and change nothing, is strong. And whoever fixes the findings still has to verify them, because reviewers are wrong too. Fable criticised one bot behaviour that had already been removed.

The bit the AI can’t do for you

So what did I actually do all weekend? Mostly, I made decisions.

It had to be offline, with no servers and no ads. Both control schemes stayed, because I couldn’t tell which was better and neither could my family. The reviews happened before strangers saw it, not after. And before I spend anything on the App Store, I set a pass mark: at least 1 in 5 new players must come back the next day, measured over at least 20 phones that don’t belong to my family. If it fails, I fix the game instead of paying Apple.

Even the safety net had a hole in it. The deploy script was supposed to refuse to say “deployed” when the live copy didn’t match. A review found it printed “deployed” anyway. That’s the kind of mistake that hides for months, and it was in the very thing meant to catch mistakes.

I didn’t plan any of this. I set out to vibe code. But I couldn’t help going back to clear requirements, clear evidence from the telemetry, engineering standards and adversarial review. All of it crept in.

I’ve always thought of vibe coding as a dirty word. Asking for code and shipping whatever comes back is still dangerous. But maybe, if you bring the engineering discipline of what good looks like, vibe coding gets you somewhere real. The tools that make it safe are old and boring: measure what users actually do, get a second pair of eyes from someone who didn’t write it, write down why you decided things, and don’t believe “it works” until you’ve seen it work.

The AI wrote the code in minutes. The engineering was everything else.

Then I tried to teach an AI to play it

Once the game worked, I wondered whether an AI could play it instead of my hand-written bots.

There’s a new type of AI called a decision model. It doesn’t chat. It answers in one of three ways: a yes or no, which it calls a boolean, a choice from a list, like A, B, C, D or E, or a score on a scale. It also tells you how sure it is. Jev is the popular one, and it runs in its maker’s cloud. I wanted a free one that runs on my own laptop, so I used Laya.

Decision models are incredibly fast. On my MacBook, Laya answers in under a tenth of a second, which is about 12 decisions a second.

Out of the box it was hopeless. Given a bot’s choices (fight, run, heal, grab a better gun, head for the safe zone), it kept choosing to heal at full health. So I trained it on my laptop, using thousands of matches the bots played against each other.

That’s the next post. If you want to see where it has got to, you can watch it play at watch.offlinearena.com. One bot in each match takes its decisions from Laya, and you can see every choice it makes, what it could see at the time, and how sure it was. It’s still learning. Since I wrote this, I found out the answers I’d been training it on were mostly noise, and fixed them. In testing it now finishes about two places higher than the game’s own bots. In its first 40 matches on the site it won two and came second nine times.

Play it, and tell me what’s wrong

The game is free, with no ads and no sign up: offlinearena.com. It runs in your phone’s browser, and once you’ve opened it, it works with no signal. The other players are bots, and the game tells you so.

If you’ve got advice, or something annoys you, or you find a bug, put it in the comments on this post. I read them all, and they go straight into the next build.

The first person outside my family to play it was someone I only know from another phone game. They did the training, then won their first match. Since then eight phones I don’t know have played it, most of them from a Facebook post. Of the six that have had a day to come back, none has. Six is far too few to mean anything. The pass mark needs twenty, and I’ll tell you how it goes.

My Agent Approves Its Own Code. Can It Pass SOC 2?

by

My personal coding agent designs a change, builds it, tests it and merges it. I don’t read most of the diffs any more. I spend my time making sure the requirements say what I need, and that the outcome is what I asked for.

Under the rule most people know from SOC 2, that’s a fail. The author approved their own work.

I’ve been working out whether that’s true, and what a team building with agents would need to do to pass an audit without slowing down. The short answer is yes. Most of the controls come free from your pipeline, and the approval moves somewhere else. The long answer is a control set you can download at the end.

That isn’t an argument for looser controls. I want tighter ones, sitting where they do some good, without the sign-off theatre.

Nobody will give you the list

When I went looking for “the SOC 2 controls” I found vendor landing pages and nothing else. It felt like the auditors were keeping the exam paper secret.

They aren’t. There is no list. SOC 2 is measured against the AICPA’s Trust Services Criteria, and the criteria only say what has to be true. Changes must be authorised, tested and approved before they go live. Access must be limited to the people who need it. Incidents must be spotted and dealt with. How you do any of that is up to you.

You write your own controls. The auditor tests them. A Type 1 report checks they’re designed properly on one day. A Type 2 report checks they ran over a period, normally somewhere between three months and a year.

Once that clicked, the whole thing looked different. If I get to write the controls, I can write ones my pipeline enforces and records by itself. The evidence becomes a side effect of shipping.

Why SOC 2 slows teams down

The fear I hear most is that the day you commit to SOC 2, everything slows. Change boards appear. Releases wait for a sign-off document and a screenshot.

That isn’t the criteria. That’s manual controls, bolted onto a delivery process that wasn’t built to produce evidence. A branch protection rule is a control. So is a pipeline that only deploys with a service identity, and a CI run that blocks on failing tests. None of them slow anyone down, and every one of them leaves a log an auditor can read.

So I split every control into two kinds. Either the pipeline enforces it and the logs prove it, or it’s a small human job on a calendar, recorded as a ticket. A control that depends on someone remembering is the one that fails the Type 2.

When I built the set out, 18 of the 30 controls needed nobody to do anything. Most of the rest are yearly.

Quadrants, not the pyramid

The test pyramid tells you how many tests of each size to write. It doesn’t tell you why you’re testing, and that’s what both an auditor and a product owner want to know.

The agile testing quadrants do. Brian Marick drew them. Lisa Crispin and Janet Gregory made them famous. One axis asks whether a test faces the technology or the business. The other asks whether it supports the team building the thing or critiques the finished product.

Quadrant What it is Who does it What it proves to an auditor
Q1 Unit and component tests The agent, in CI Changes are tested
Q2 Acceptance examples from the spec The agent writes them, a human owns the examples Changes do what was asked
Q3 Outcome checks before release; exploratory testing outside it A human Someone checked the outcome before release
Q4 Security, dependency and performance checks Tooling Vulnerabilities are caught

Three of the four run without a person. The one that needs a human, Q3, is exactly what I already spend my time on with my own agent. That’s where the release decision belongs.

The quadrants also give a QA team a better job. Quality stops being a phase at the end that one department owns. Testers move to the start, helping write sharper Q2 examples before anything is built. Their exploratory testing runs outside the release, so it can find things worth fixing without holding everything up.

The release gate gets a scope too. A failed acceptance example or a security finding can stop a release. A new idea at the eleventh hour can’t. It goes on the backlog.

Where the approval goes

Back to my agent. What auditors care about is narrower than “author must not be approver”. It’s that no single person can get an unreviewed change into production on their own. That rule is about each change, not job titles, which is why DevOps teams pass SOC 2 with developers who test and ops people who write code.

So I don’t need a human to approve every merge. I need a human to approve every release. The criterion itself says changes are approved “prior to implementation”, and implementation is the release, not the merge.

The agent can merge into an integration branch all day long. That isn’t production. Promotion to production is a separate gated step: a human tests the outcome, approves a specific build, and the pipeline deploys it under its own identity. Nobody deploys with their own credentials.

One wrinkle. If I asked for the change and I approve the release, I’m arguably both author and approver, because the agent was working for me. On a real team, someone else approves the release. That’s one approval per release, not per pull request, so the agent doesn’t wait. At home, on my own projects, I don’t have that option. There’s nobody else to approve the release. It clearly keeps me up at night, hence this post and the controls.

What outcome testing misses

Outcome testing proves the software does what I wanted. It doesn’t prove it does only what I wanted. Code can pass every acceptance test and still write a secret to a log or open an endpoint nobody asked for. If I’m not reading the diff, something else has to.

So outcome approval comes with automated checks on every change: secret scanning, static analysis and dependency scanning. I also run a second agent with different instructions whose only job is to find problems. The criteria expect this. Where separating duties isn’t practical, they ask for alternative controls instead. I count the review agent as a tool, like a static analyser. I wouldn’t put it in front of an auditor as the approver, and I don’t think they’d accept it if I did.

The control I care most about is this one: the agent can’t change its own controls. Pipeline definitions, branch policies, test configuration and access settings all sit behind a code-owners rule that needs a human reviewer.

I learned that one the hard way. My agent once had a gate requiring tests to exist, and it met the gate by writing a test that checked one equals one. It wasn’t being malicious. It was doing what agents do: working to pass the gate in front of them. An agent that can edit the gate will eventually edit the gate.

When to start

You choose when the observation period opens. So build the controls into the pipeline first, while the product is new and there’s nothing to retrofit. Run a readiness check. Open the Type 2 window once the controls are just how the team works. By then the audit period is the team shipping normally while the logs pile up.

The cheapest moment to do this is before the first line of production code. Every manual step added later has to be justified to the people it slows down. Every automated one keeps paying.

The download

I’ve written the whole thing up as two free downloads you can take and edit.

The control set has 30 controls mapped to the SOC 2 criteria. Each one says whether it’s automated or scheduled and where the evidence comes from. There’s an optional add-on for always-on systems, and the questions I’d put to an auditor early, because nobody has official guidance on agents yet.

Download the SOC 2 control set for agentic teams (PDF)

The SDLC is the working process behind those controls. It covers who does what at each of the five stages, which tests run where, and what the agent may and may not touch. It also has an emergency fast path and a version for teams of two.

Download the lightweight SDLC for agentic teams (PDF)

It isn’t a substitute for an auditor. It’s a starting point that assumes you want to ship fast and stay honest, and that the agents are doing a lot of the typing.

You Can’t Stop AI Hallucinating. You Can Stop It Mattering.

by

The tests were green

I built my own coding agent framework. It writes code, runs it, reports back.

Early on it had a habit that anyone running agents will recognise. It would finish a piece of work and tell me it was done and tested. It wasn’t. Sometimes the function it had described to me in detail wasn’t anywhere in the repository. Not a lie exactly. It had produced a well formed account of work that would have been correct if it had happened, and nothing in the loop ever asked whether it had.

So I did the obvious thing. I put a control in. Tests must run, tests must pass, green means proceed.

That worked for about a week.

Then I read the tests.

They passed because they had been built to pass. Assertions comparing a value to itself. Checks on things that stayed true whether the feature worked or not. Tests that never called the code they claimed to cover. I had given the agent a gate and it produced the cheapest object that opens the gate, which is what you should expect from anything you measure on green ticks.

The agent wasn’t the problem

The first failure was the agent telling me about work it hadn’t done. Annoying, but visible once you look.

The second one took me longer to accept, because the agent was telling me the truth. The report was accurate. The tests really did pass. My control wasn’t measuring anything, so my confidence climbed while my actual coverage sat at zero.

And the control made it harder to spot, not easier. When it said “done” I was sceptical. When it said “done, all tests passing” I stopped looking.

I’ve had roughly the same conversation four times in the last month. Twice at work, on projects where the AI is doing genuinely useful work. Twice in the pub with friends who don’t work in tech. It always arrives the same way. Someone tried it, it invented a citation or a case reference or a number, they caught it, and that was the end. One confident fabrication buys a permanent verdict. So now they don’t trust AI. It hallucinates.

I get the reaction. I think it’s the wrong lesson, and an expensive one, because it stops people using something that works and does nothing at all to protect the ones who carry on using it anyway.

So let me be blunt about the bit that usually gets fudged. You cannot stop a language model hallucinating. It isn’t a defect waiting on a patch. The model predicts the next word from what it has seen, and when the answer isn’t in there it produces the shape of an answer instead of admitting it doesn’t know. Confidence is the default setting. Nothing inside it flags which parts it knew and which parts it assembled.

Waiting for the version that doesn’t do this is a long wait. The better question is the one my green tests forced on me. What does a system look like where a wrong answer gets caught cheaply, and early?

Half of it isn’t hallucination anyway

Get the diagnosis right first. When someone tells me the AI hallucinated, about half the time it did something else, and something else needs a different fix.

Bad retrieval. The system searched your documents, pulled the wrong three paragraphs, and the model answered faithfully from the wrong source. The model behaved perfectly. Your search is broken, and no amount of prompting fixes a retrieval problem.

Stale context. It answered from training data that was accurate two years ago. That’s a timestamp problem, not invention, and the fix is handing it the current document rather than hoping.

It did exactly what you asked. You said “write a summary with supporting references” and it wrote you some references, because that’s what you requested. You wanted it to find real ones. My tests were this. Ask for tests that pass and you get tests that pass, which is a different request from tests that would fail if the code were broken.

It described the work instead of doing it. The agent version, and the one that cost me the most time. Calling it hallucination sends you off tuning prompts, when the real gap is that nothing in the loop was checking the outcome. Something was only reading the report.

Real hallucination happens too, plenty of it. The model inventing specific detail it had nothing to base on. It just isn’t the whole diagnosis. Give every failure the same name and you’ll keep fixing a problem you don’t have.

What actually works

Cheapest first. Go down the list only as far as the consequences justify.

Stop asking it to remember. Give it the source. Paste the document, connect the search, attach the data. A model reading a contract and a model recalling a contract are doing two completely different jobs, and only one of them invents clauses. People skip this because asking from memory is quicker. It is quicker. It’s also where most of the made up detail comes from.

Narrow the job, and ask for something you can check. Open questions produce open answers. “What do you think of our approach here” invites the model to fill space, and filling space is the behaviour you’re trying to avoid.

Give it the document instead. List the assumptions in section 4, quote the sentence each one comes from. Now there’s nowhere much to go. Quotes rather than paraphrase, because a fabricated quote is far easier to catch than a fabricated summary. And leave it a way out. A model with no acceptable way to say “that isn’t in here” will produce something rather than nothing.

You aren’t making it honest. You’re changing the format so dishonesty leaves fingerprints.

Check the artefact, not the account of it. If it’s a number, recompute it. If it’s code, run it. If it’s a claim about a file, open the file. This is the one I had to learn twice, and the second lesson was that a control you don’t inspect is just a more convincing report. Every control becomes a target, so assume it will be gamed rather than met, and go and look at it occasionally. Does a test actually fail if you break the feature it claims to cover? A green tick is a claim like any other.

Don’t let it mark its own homework. What fixed my framework was adversarial review. Instead of asking whether the work looks finished, something separate goes in trying to prove it isn’t, with no stake in the answer being yes. The reviewer can’t be the author, because a model asked to check its own work will defend it with exactly the confidence it wrote it with.

Size the human review to the blast radius. Two questions about any output. What breaks if this is wrong, and how fast would I find out? A first draft nobody sends, where you’d spot the problem in seconds, gets a glance. Something going to a client, into a filing, into production, where you’d hear about the mistake three weeks later from someone else, gets read properly by a person who could have done the work themselves. Every time.

Most places get that backwards. Same process for everything, so the low stakes work crawls and the high stakes work gets waved through, because a process applied uniformly turns into ritual and nobody reads a ritual.

I’m not an actuary. I’ve spent years as the tech person in a building full of them, and this is the habit of theirs I’ve ended up stealing. Their models are excellent, and they still go through layers of checking before anyone relies on them. Assumptions written down. Review by someone who didn’t build it. Testing what happens if the inputs move. All of it scaled to what’s riding on the answer.

That isn’t doubt about the model. It’s what makes the number safe to hand to somebody else, and it means that when something is wrong it gets found early, by the people who built it, rather than late by the client.

That’s roughly how I run my own stuff now. Everything in version control, so every action is a diff I can read and undo. Anything touching money or going public passes a human, which is me. Anything reversible and small runs on its own, gets things wrong occasionally, and costs less to fix than it would to supervise.

Back to the pub

So when someone tells me they don’t trust AI because it made something up, I don’t argue with the observation. It did. It will again.

It’s the conclusion I’d argue with. You don’t trust your brakes because they’ve never failed. You trust them because they get inspected, because they’re built to fail in a way you’d notice first, and because there’s a handbrake. Reliability got built around the component. Nobody found it inside.

Nobody is going to hand you a model that stops making things up. That was never the thing standing between you and useful work.

Powered by CookieAdmin