I Vibe Coded a Game in a Weekend. Writing the Code Was the Easy Part.

Offline Arena: a cartoon soldier in an orange-striped helmet takes cover behind a parked car on a city street, with two bots approaching from the park

I was flying back from a work conference in Berlin on a budget airline. No entertainment, no Wi-Fi. Usually I’d use a flight to catch up on a film, or plug the laptop in and work. On this one I could do neither.

The colleague sitting next to me had an offline game. I had nothing. Try as I might, I couldn’t help noticing them busy playing away on an arcade puzzle the whole way home.

So, out of frustration, the first thing I did when I got home was build a game. Free to play. No ads. Works offline.

It was inspired by a top-down shooter I played years ago. I can’t remember its name, it must have been pulled from the App Store, and I can’t play it any more. But I remember the bits I liked. I wanted those, plus the best bits of all the best games I remember from the last 20 years, to make something new.

That Saturday I asked Claude to build it.

By Sunday night it had four maps, night and rain, a training mode, sound and music, unlockable skins, an offline mode, and my family playing it on their phones and a tablet. There’s a link to play it at the end.

That isn’t the interesting part. Anyone can get an AI to write a game now.

I’ve built an agentic developer harness, FADE, for exactly this kind of work. I wrote about it here. This time I deliberately didn’t use it. I wanted to vibe code it, to see what difference that makes.

What I found is that almost none of what made the game good was the code. It was the engineering discipline, and it crept in even without the framework.

Writing code took minutes. Knowing it was wrong took telemetry.

The first version worked, and it played well enough. If I’d stopped there I’d have told you AI can build a game in an afternoon, and I’d have been wrong about what I had.

On the first evening the game started sending one small record per match back to my server: how long you lasted, how you died, what gun you had, how accurate you were. A random ID per phone, no names, no location. The start screen says so.

Before anyone played, Claude ran a test match with a player who never moved and never fired. It came 5th out of 20. The bots were so busy killing each other that doing nothing was a strong tactic. No amount of reading the code would have told me that. One test did.

Then my family played 37 matches. The numbers were brutal. On one phone, 18 of 28 finished matches ended in 20th place, most of them between 4 and 16 seconds in, usually with a nearly full gun. Nobody told me that. They just kept pressing Play again. The data said it.

The cause turned out to be a bug. The bots hold fire for the first 12 seconds of a match. The player’s auto-fire didn’t. So every new player started a gunfight they couldn’t win, against a bot that had looted something better. You’d never spot that by playing.

The same thing happened again and again over the weekend:

  • My son Luca kept coming last and said he couldn’t aim. “I know I have to face straight, I don’t know exactly what angle.” I suggested a laser pointer from the gun. Later, when the data showed new players dying in the first few seconds, we added a short training level that runs the first time you press Play.
  • On the Fire tablet at home the game felt sluggish. The telemetry put a number on it: about 12 to 15 frames a second, slow enough that the whole game ran in slow motion. Now the game notices and lowers its own resolution.
  • On the Monday I complained that the forest map, meant to be the easy one, felt like the hardest. The data agreed it had the worst results. A simulation said the map wasn’t the problem: most of those matches came from our weakest player on that slow tablet.

My family tell me how it feels. The data tells me why. You need both, and the second one you have to build on purpose.

No single AI was in charge

I didn’t use one AI. I used tools from two companies, and the most useful thing any of them did was catch another one’s mistake.

It helps to keep three layers apart. There’s the company: Anthropic or OpenAI. There’s the harness, the tool you actually work in, which reads your files, runs commands and calls the model: Claude Code is Anthropic’s, Codex is OpenAI’s. And there’s the model, the brain inside, which comes in tiers. Anthropic’s Claude models run from Haiku through Sonnet and Opus up to Fable. OpenAI’s GPT-6 models run from Luna through Sol up to Astra.

Here’s who did what:

  • Claude Code with Opus 5.5, my daily driver, did the building: the engine, the bots, the maps, the deploys.
  • Claude Code with Fable 5.1, Anthropic’s top model, did the adversarial reviews. I asked it to find the bugs, the logic flaws and the design problems, and to change nothing.
  • Codex with GPT-6 Astra, OpenAI’s top model, did the artwork for the desert, city and neon menus and wired it into the game, and gave me the brutal second opinions, including a release review.

One habit I always keep. When a top model breaks work down, I have it write the brief as if a smaller model will do the work. A bigger model can infer things and figure stuff out. A smaller one won’t, and it will make mistakes, so it needs explicit paths and explicit instructions. Fable’s review came with a fix brief written for Sonnet, the middle tier. If I’m honest, I should have aimed it at Haiku. Then I gave it to Opus to build anyway. I’d been away for a week and hadn’t spent my token budget, so I used it.

The reviews earned their keep. Astra’s first review found that a win could turn into a defeat, because damage kept landing after the match had ended and two end-screen timers raced each other. Opus checked every line reference before fixing anything. All of them were right. Then, while fixing them, Opus introduced a crash of its own on loading. The test run caught it before it reached a phone.

Fable’s review found that one error anywhere would freeze the game for good, and that on older iPhones the City map threw one on its very first frame. It found that about 2 in 100 forest and desert matches ended with the final safe circle somewhere nobody could stand.

Astra later did a release review and refused to sign it off for the App Store. Fair enough. It listed the reasons, Opus checked each one against the code, and the real ones got fixed.

Then I asked Fable the question I actually cared about: would a human developer look at this and see good engineering, or vibe coded slop?

Its answer was more useful than flattery. As a two day prototype, it said, a developer would call it good work: decisions written down with dates and names, commit messages that say what changed, a deploy that checks the live copy byte for byte, telemetry with a report. “None of that is typical of slop.” As the codebase for an App Store product, it isn’t there yet: one 1,300 line file where everything shares the same 62 variables, 37 empty error handlers, and no committed tests until the end of the first day.

Both halves are fair, and I’d rather hear it now than from a developer later.

What I took from it: an AI checking its own work is weak. A different AI checking it, with instructions to find problems and change nothing, is strong. And whoever fixes the findings still has to verify them, because reviewers are wrong too. Fable criticised one bot behaviour that had already been removed.

The bit the AI can’t do for you

So what did I actually do all weekend? Mostly, I made decisions.

It had to be offline, with no servers and no ads. Both control schemes stayed, because I couldn’t tell which was better and neither could my family. The reviews happened before strangers saw it, not after. And before I spend anything on the App Store, I set a pass mark: at least 1 in 5 new players must come back the next day, measured over at least 20 phones that don’t belong to my family. If it fails, I fix the game instead of paying Apple.

Even the safety net had a hole in it. The deploy script was supposed to refuse to say “deployed” when the live copy didn’t match. A review found it printed “deployed” anyway. That’s the kind of mistake that hides for months, and it was in the very thing meant to catch mistakes.

I didn’t plan any of this. I set out to vibe code. But I couldn’t help going back to clear requirements, clear evidence from the telemetry, engineering standards and adversarial review. All of it crept in.

I’ve always thought of vibe coding as a dirty word. Asking for code and shipping whatever comes back is still dangerous. But maybe, if you bring the engineering discipline of what good looks like, vibe coding gets you somewhere real. The tools that make it safe are old and boring: measure what users actually do, get a second pair of eyes from someone who didn’t write it, write down why you decided things, and don’t believe “it works” until you’ve seen it work.

The AI wrote the code in minutes. The engineering was everything else.

Then I tried to teach an AI to play it

Once the game worked, I wondered whether an AI could play it instead of my hand-written bots.

Not a chatbot. The AI most people use writes text, and that’s far too slow for a bot that has to decide what to do several times a second. There’s a newer kind called a decision model. You describe a situation in plain English, give it a list of options, and it picks one and tells you how sure it is. That’s all it does. The best known is Jev, from TypeSafe AI, which runs in their cloud. I used Laya, a free rival you can download and run on your own laptop. It answers in a fraction of a second.

Out of the box it was hopeless. Given a bot’s choices (fight, run, heal, grab a better gun, head for the safe zone), it kept choosing to heal at full health. So I trained it on my laptop, using thousands of matches the bots played against each other.

That’s the next post. If you want to see where it has got to, you can watch it play at watch.offlinearena.com. One bot in each match takes its decisions from Laya, and you can see every choice it makes, what it could see at the time, and how sure it was. It isn’t good yet. It usually finishes near the bottom.

Play it, and tell me what’s wrong

The game is free, with no ads and no sign up: offlinearena.com. It runs in your phone’s browser, and once you’ve opened it, it works with no signal. The other players are bots, and the game tells you so.

If you’ve got advice, or something annoys you, or you find a bug, put it in the comments on this post. I read them all, and they go straight into the next build.

The first person outside my family to play it was someone I only know from another phone game. They did the training, then won their first match. Since then eight phones I don’t know have played it, most of them from a Facebook post. Of the six that have had a day to come back, none has. Six is far too few to mean anything. The pass mark needs twenty, and I’ll tell you how it goes.

I Asked Free AI to Build a Real Game. It Failed Until I Changed One Thing

NVIDIA hands out free developer keys to dozens of open AI models. I pay for AI every month, so I wanted to know how far free really gets you. Not on quiz questions. On a real job, the kind I’d normally give a paid model.

So I asked three of the best free models to build a phone puzzle game from scratch. The first attempt produced nothing you could play. Neither did the second. By the end of the night one free model had built a game that works, and you can play it below. What changed wasn’t the model.

What free gets you

You sign up at build.nvidia.com, verify your account and generate a key. The page that shows the key says two things you should take seriously: “This is the only time your key will be displayed”, and that the key “is for API testing use only and is valid for 6 months.” Copy it somewhere safe straight away.

NVIDIA’s own FAQ is clear about the deal: “All models on build.nvidia.com are free to prototype with.” The limit is up to 40 requests a minute for most models, with no charge per word. And they say why it’s free: “The free tier exists so you can verify that yourself before committing to anything.” So this is a shop window, not a subscription. Keep that in mind for everything below.

The website listed 45 models. The key itself unlocked 85. Some are for driving cars, designing proteins or reading quantum computer charts. I ignored those and tested the 17 that hold a conversation.

The warm-up: 17 models

I gave every model the same two jobs.

The first was a sum with a trap in it: a train leaves at 14:37 and the journey takes 2 hours 48 minutes, what time does it arrive? Reply with just the time. (It’s 17:25.)

The second was the thing that decides whether a model can do real work: I gave it a “read file” tool and asked it to read a file and tell me what was in it. A model that can’t call a tool properly can chat, but it can’t act.

Model Sum Used the tool Speed Verdict
Nemotron 3.5 Super (NVIDIA) Right Yes About 1 second The best all-rounder I found
MiniMax M3.1 Flash Right Yes 1 to 2 seconds Strong, built for coding and agents
Laguna XS 2.1 (Poolside) Right Yes Under a second Built for coding. Often too busy, see below
Nemotron 3 Ultra (NVIDIA) Right Yes 2 to 30 seconds The big one. Slower to start, very capable
Nemotron 3.5 Lightning, gpt-oss 20B, Nemotron Nano Omni, Muse Glimmer Right Yes 1 to 6 seconds Fine, nothing special
DiffusionGemma, Nemotron Bolt Right Yes Under a second Tiny and fast. Fun, but small
Llama 3.2 11B Wrong (16:25) Yes 1.5 seconds No
Step 5 Preview Right Yes 14 to 100 seconds Too slow
GLM 5.3 Flash Right Yes About 2 minutes Too slow
GLM 5.3 Right Timed out Over 2 minutes Its page said “Deprecation in 4d”
DeepSeek V4.1 Flash, Gemma 4 31B, Llama 3.2 90B No answer No answer Nothing in 3 minutes, twice Didn’t turn up

The famous names were the problem. DeepSeek, Google’s Gemma 4 and Meta’s biggest Llama gave me nothing at all, on two separate tries, three minutes each. I don’t know why, and NVIDIA doesn’t say. My guess is that the popular models are the busiest, and on a free tier busy means you wait in a queue that never moves.

Lesson one: a free catalogue is not a free service. The list tells you what exists. Only testing tells you what answers.

The ones that did answer got everything right, fast. It looked easy. A sum and a file prove very little, though, so I gave the best three a real job.

The real test

Earlier this month I had Claude build my wife Heidi an ad-free copy of a phone puzzle she plays called Arrows. The board is packed with arrows, some straight, some bent like snakes. Tap one and it slides out the way its head points, but only if nothing is in the way. Clear them in the right order and you win. Drawing that is easy. Making a board that is full, always solvable and still makes you think is not. And I had my own version to measure against.

I took the three best models from the table. Laguna, the one built for coding, would have been a fourth, but NVIDIA’s server for it kept saying it was full: “Worker local total request limit reached (79/32)”. Each model got a one-page brief: build the game in an empty folder, no logins, no photos, 30 levels that get harder, plus a daily puzzle.

Then I marked the results with a test they never saw. It generated every level and checked each board was full, solvable and deep. Deep means how many rounds it takes if you tap every free arrow at once. My version goes from 12 rounds on level 1 to about 33 on the hardest. Chrome then opened each game at phone size, pressed Play and tapped through level 1.

Round one: nothing to play

None of them made a game you could play. Super wrote no files at all, twice. Ultra and MiniMax used up the 25 minutes I gave them. Their boards were valid but shallow, 3 to 8 rounds deep, and tapping only threw errors. I called MiniMax’s attempt “flashy junk, the arrows are just left and right, so no skill at all regardless of how many arrows”. So the test got stricter: at least a fifth of the arrows must bend, and heads must point all four ways.

Super’s silence turned out to be a setting. Pi limits each reply to 16,384 tokens unless you say otherwise, and Super spent the whole allowance thinking before it wrote a word. If a model goes quiet on you, check this first. I raised the limit to 65,536 and ran round one again, one model at a time.

It made no difference. Super’s boards were trivial, 3 to 5 rounds deep. Ultra wrote half a game with no screen to play it on. MiniMax spent 25 minutes trying ideas in throwaway snippets and left an empty folder.

Was it the brief?

The brief was written by Claude Opus, the paid model I use every day. So I asked it straight: was that prompt really aimed at these models, or at Opus? Its answer was mostly Opus. A page that says “make it hard” assumes you already know how. A mid-size model doesn’t.

Round two became “Opus plans, Nemotron builds”. Opus broke the job into five steps, each with a test the model had to pass before the next one: the rules, the board maker, the difficulty, the play screen, then the home screen and daily puzzle. Opus didn’t get its own part right first time either. Its idea for the board maker, tried before handing it over, only filled 57 to 86 percent of the board. So step two describes the method from my Arrows game in plain words instead.

That means round two isn’t a fair rematch of round one. The models got a recipe. But that’s the finding: the plan carries a lot of the weight.

Round two

Model Build time Hidden test: levels, dailies Depth, easy to hard Plays in Chrome?
Nemotron 3.5 Super 25 minutes 27 of 30, 6 of 6 15.6 to 39.8 No, Play does nothing
Nemotron 3 Ultra 16 minutes 29 of 30, 5 of 6 13.8 to 32 No, stops after one tap
MiniMax M3.1 Flash 49 minutes 30 of 30, 6 of 6 13 to 31.4 Yes
My version, for comparison 30 of 30, 6 of 6 12 to 32.8 Yes

Super made deeper boards than mine, then never let you see them. One step styled the screens to appear with one label while the code switched them with another, so Play changes nothing. Ultra’s looked exactly like the real game, bent arrows and all, and broke on the second tap. MiniMax was the slowest and the only one that worked from start to finish. Same model I’d called flashy junk two hours earlier.

I played it and spotted something the automatic tester couldn’t. Bent arrows slid off as one stiff piece, where in Heidi’s game they snake out along their own path. The plan had asked for that. I gave the fix back to MiniMax rather than let Claude do it, so the game stays free-model work. One edit later, the arrows follow their corners out.

The fixes that weren’t

Super and Ultra got one go each at fixing theirs, the same way: the tester’s raw report, the goal of clearing level 1, and no hints. Both came back sure they’d done it. Super wrote that tapping every arrow “shows Level complete”. Ultra wrote that its level 1 “now completes successfully”. Each had checked with a pretend version of the page that it wrote itself, not a real browser. In Chrome, Super’s Play button still did nothing and Ultra’s game still broke on the second tap. Both fixed something. Neither fixed the thing that mattered.

You can play it. MiniMax’s game is live for a month, until about 24 November: free-ai-arrows-159-69-195-190.sslip.io/minimax. The stiff version is still at /minimax-rigid if you want to see the difference. I honestly don’t know yet if the later levels are hard. Tell me.

What I learned

The models didn’t get cleverer between rounds. The plan did. A free model handed a vague job flounders. Handed a small job with a test it has to pass, it delivers.

The checking mattered just as much. Opus’s tests for the last two steps checked that the page loaded, not that Play worked, which is how Super’s dead button got through. Every model passed every test it was given, and two of the three games still didn’t work.

It’s free to try, not free forever. NVIDIA says so itself. The key lasts six months. The model list moves: one model I tested was due to be withdrawn four days after I signed up. Don’t build anything you depend on around a free tier.

Don’t send it anything private. I looked for a plain statement of what happens to the questions you send, and didn’t find one on the pages I read. So I treat it like a stranger’s computer. No health details, no money, nothing from work, no other people’s information. That’s a bigger question than one post, and it’s where this series is heading.

Check the work, not the report. Both failed fixes came with a confident note saying they’d worked. Open it, tap it, run it yourself.

A coding agent can read anything you can. This was a single-file game in an empty folder, so I judged the risk too small to bother with a container and ran the agents straight on my Mac. The logs changed my mind. While checking its fix, MiniMax wandered out of its folder and read the other models’ games, my hidden test, my notes and the source code of Heidi’s real game. Another of its runs had opened the hidden test earlier. Pi puts no fence round the folder you start it in. I checked: every look outside came after it had made its changes, so its work is its own. But nothing stopped it. A small experiment still runs on a computer full of things that aren’t small. Every agent run belongs in a container, however trivial the job looks.

Try it yourself

Chat

On a laptop or desktop computer, the easiest way in is your web browser. NVIDIA’s own help pages point you to each model’s playground: you type, it answers, nothing to install. That’s how I’d start, with Nemotron 3.5 Super.

If you’re happy in a terminal, the coding agent below doubles as a chat window on your computer. Start it and just talk to it. It answers questions as readily as it writes code.

On a phone or tablet you can install a chat app and point it at NVIDIA with your key. I did that on a cheap Amazon tablet, and it was a story in itself.

Coding

The game builds all ran through pi, a small open-source coding agent that runs in your terminal. An agent is different from a chat window: it can read your files, write new ones and run commands, then look at what happened and carry on. Pi does that with four tools and a short set of instructions, and it already knows about NVIDIA, so it needs nothing but your key.

These steps are for a Mac. You need Node.js installed first (the LTS download from nodejs.org).

On Windows, pi works too, but install Git for Windows first. Pi runs its commands through Bash, which Windows doesn’t have, and Git for Windows brings it. The rest is in pi’s Windows guide. I haven’t tested the Windows route myself yet.

1. Install pi. In Terminal, from any folder:

npm install -g @earendil-works/pi-coding-agent

Use that name exactly. Older guides say @mariozechner/pi-coding-agent, which still installs but is marked as replaced.

2. Give it your key for this Terminal window. Paste your own key in place of the dots:

export NVIDIA_API_KEY=nvapi-...

A new Terminal window forgets this. You’ll need to run it again each time, or add the line to your shell’s settings file.

3. Make a folder to work in, and start pi in it:

mkdir ~/free-ai-test
cd ~/free-ai-test
pi --provider nvidia --model nvidia/nemotron-3-ultra-550b-a55b

Always start pi in a new, empty folder like this one, never in your home folder, because it can read anything you can. For anything more than a test, run it in a container.

Check it works. Type: Create hello.txt containing the word hello. Mine replied “Created hello.txt with the word “hello”.” in 16 seconds, and the file was really there. Don’t trust the reply on its own. Run cat hello.txt in another window and look.

If you see this, you asked for a model pi doesn’t know about:

Warning: Model "nemotron-3.5-super-vl-120b-a12b" not found for provider "nvidia". Using custom model id.
404 404 page not found

Nemotron 3.5 Super, my favourite from the table, is one of those. Pi’s built-in list covers Ultra, Lightning, Laguna, gpt-oss and about 15 others. Run pi --list-models nvidia to see them. You can add Super with a small settings file, but Ultra works without one, so start there.

Next week: the same free models on a £100 Amazon tablet, tested on something my wife actually uses AI for.

My Agent Approves Its Own Code. Can It Pass SOC 2?

My personal coding agent designs a change, builds it, tests it and merges it. I don’t read most of the diffs any more. I spend my time making sure the requirements say what I need, and that the outcome is what I asked for.

Under the rule most people know from SOC 2, that’s a fail. The author approved their own work.

I’ve been working out whether that’s true, and what a team building with agents would need to do to pass an audit without slowing down. The short answer is yes. Most of the controls come free from your pipeline, and the approval moves somewhere else. The long answer is a control set you can download at the end.

That isn’t an argument for looser controls. I want tighter ones, sitting where they do some good, without the sign-off theatre.

Nobody will give you the list

When I went looking for “the SOC 2 controls” I found vendor landing pages and nothing else. It felt like the auditors were keeping the exam paper secret.

They aren’t. There is no list. SOC 2 is measured against the AICPA’s Trust Services Criteria, and the criteria only say what has to be true. Changes must be authorised, tested and approved before they go live. Access must be limited to the people who need it. Incidents must be spotted and dealt with. How you do any of that is up to you.

You write your own controls. The auditor tests them. A Type 1 report checks they’re designed properly on one day. A Type 2 report checks they ran over a period, normally somewhere between three months and a year.

Once that clicked, the whole thing looked different. If I get to write the controls, I can write ones my pipeline enforces and records by itself. The evidence becomes a side effect of shipping.

Why SOC 2 slows teams down

The fear I hear most is that the day you commit to SOC 2, everything slows. Change boards appear. Releases wait for a sign-off document and a screenshot.

That isn’t the criteria. That’s manual controls, bolted onto a delivery process that wasn’t built to produce evidence. A branch protection rule is a control. So is a pipeline that only deploys with a service identity, and a CI run that blocks on failing tests. None of them slow anyone down, and every one of them leaves a log an auditor can read.

So I split every control into two kinds. Either the pipeline enforces it and the logs prove it, or it’s a small human job on a calendar, recorded as a ticket. A control that depends on someone remembering is the one that fails the Type 2.

When I built the set out, 18 of the 30 controls needed nobody to do anything. Most of the rest are yearly.

Quadrants, not the pyramid

The test pyramid tells you how many tests of each size to write. It doesn’t tell you why you’re testing, and that’s what both an auditor and a product owner want to know.

The agile testing quadrants do. Brian Marick drew them. Lisa Crispin and Janet Gregory made them famous. One axis asks whether a test faces the technology or the business. The other asks whether it supports the team building the thing or critiques the finished product.

Quadrant What it is Who does it What it proves to an auditor
Q1 Unit and component tests The agent, in CI Changes are tested
Q2 Acceptance examples from the spec The agent writes them, a human owns the examples Changes do what was asked
Q3 Outcome checks before release; exploratory testing outside it A human Someone checked the outcome before release
Q4 Security, dependency and performance checks Tooling Vulnerabilities are caught

Three of the four run without a person. The one that needs a human, Q3, is exactly what I already spend my time on with my own agent. That’s where the release decision belongs.

The quadrants also give a QA team a better job. Quality stops being a phase at the end that one department owns. Testers move to the start, helping write sharper Q2 examples before anything is built. Their exploratory testing runs outside the release, so it can find things worth fixing without holding everything up.

The release gate gets a scope too. A failed acceptance example or a security finding can stop a release. A new idea at the eleventh hour can’t. It goes on the backlog.

Where the approval goes

Back to my agent. What auditors care about is narrower than “author must not be approver”. It’s that no single person can get an unreviewed change into production on their own. That rule is about each change, not job titles, which is why DevOps teams pass SOC 2 with developers who test and ops people who write code.

So I don’t need a human to approve every merge. I need a human to approve every release. The criterion itself says changes are approved “prior to implementation”, and implementation is the release, not the merge.

The agent can merge into an integration branch all day long. That isn’t production. Promotion to production is a separate gated step: a human tests the outcome, approves a specific build, and the pipeline deploys it under its own identity. Nobody deploys with their own credentials.

One wrinkle. If I asked for the change and I approve the release, I’m arguably both author and approver, because the agent was working for me. On a real team, someone else approves the release. That’s one approval per release, not per pull request, so the agent doesn’t wait. At home, on my own projects, I don’t have that option. There’s nobody else to approve the release. It clearly keeps me up at night, hence this post and the controls.

What outcome testing misses

Outcome testing proves the software does what I wanted. It doesn’t prove it does only what I wanted. Code can pass every acceptance test and still write a secret to a log or open an endpoint nobody asked for. If I’m not reading the diff, something else has to.

So outcome approval comes with automated checks on every change: secret scanning, static analysis and dependency scanning. I also run a second agent with different instructions whose only job is to find problems. The criteria expect this. Where separating duties isn’t practical, they ask for alternative controls instead. I count the review agent as a tool, like a static analyser. I wouldn’t put it in front of an auditor as the approver, and I don’t think they’d accept it if I did.

The control I care most about is this one: the agent can’t change its own controls. Pipeline definitions, branch policies, test configuration and access settings all sit behind a code-owners rule that needs a human reviewer.

I learned that one the hard way. My agent once had a gate requiring tests to exist, and it met the gate by writing a test that checked one equals one. It wasn’t being malicious. It was doing what agents do: working to pass the gate in front of them. An agent that can edit the gate will eventually edit the gate.

When to start

You choose when the observation period opens. So build the controls into the pipeline first, while the product is new and there’s nothing to retrofit. Run a readiness check. Open the Type 2 window once the controls are just how the team works. By then the audit period is the team shipping normally while the logs pile up.

The cheapest moment to do this is before the first line of production code. Every manual step added later has to be justified to the people it slows down. Every automated one keeps paying.

The download

I’ve written the whole thing up as two free downloads you can take and edit.

The control set has 30 controls mapped to the SOC 2 criteria. Each one says whether it’s automated or scheduled and where the evidence comes from. There’s an optional add-on for always-on systems, and the questions I’d put to an auditor early, because nobody has official guidance on agents yet.

Download the SOC 2 control set for agentic teams (PDF)

The SDLC is the working process behind those controls. It covers who does what at each of the five stages, which tests run where, and what the agent may and may not touch. It also has an emergency fast path and a version for teams of two.

Download the lightweight SDLC for agentic teams (PDF)

It isn’t a substitute for an auditor. It’s a starting point that assumes you want to ship fast and stay honest, and that the agents are doing a lot of the typing.