I Asked Free AI to Build a Real Game. It Failed Until I Changed One Thing

by

NVIDIA hands out free developer keys to dozens of open AI models. I pay for AI every month, so I wanted to know how far free really gets you. Not on quiz questions. On a real job, the kind I’d normally give a paid model.

So I asked three of the best free models to build a phone puzzle game from scratch. The first attempt produced nothing you could play. Neither did the second. By the end of the night one free model had built a game that works, and you can play it below. What changed wasn’t the model.

What free gets you

You sign up at build.nvidia.com, verify your account and generate a key. The page that shows the key says two things you should take seriously: “This is the only time your key will be displayed”, and that the key “is for API testing use only and is valid for 6 months.” Copy it somewhere safe straight away.

NVIDIA’s own FAQ is clear about the deal: “All models on build.nvidia.com are free to prototype with.” The limit is up to 40 requests a minute for most models, with no charge per word. And they say why it’s free: “The free tier exists so you can verify that yourself before committing to anything.” So this is a shop window, not a subscription. Keep that in mind for everything below.

The website listed 45 models. The key itself unlocked 85. Some are for driving cars, designing proteins or reading quantum computer charts. I ignored those and tested the 17 that hold a conversation.

The warm-up: 17 models

I gave every model the same two jobs.

The first was a sum with a trap in it: a train leaves at 14:37 and the journey takes 2 hours 48 minutes, what time does it arrive? Reply with just the time. (It’s 17:25.)

The second was the thing that decides whether a model can do real work: I gave it a “read file” tool and asked it to read a file and tell me what was in it. A model that can’t call a tool properly can chat, but it can’t act.

Model Sum Used the tool Speed Verdict
Nemotron 3.5 Super (NVIDIA) Right Yes About 1 second The best all-rounder I found
MiniMax M3.1 Flash Right Yes 1 to 2 seconds Strong, built for coding and agents
Laguna XS 2.1 (Poolside) Right Yes Under a second Built for coding. Often too busy, see below
Nemotron 3 Ultra (NVIDIA) Right Yes 2 to 30 seconds The big one. Slower to start, very capable
Nemotron 3.5 Lightning, gpt-oss 20B, Nemotron Nano Omni, Muse Glimmer Right Yes 1 to 6 seconds Fine, nothing special
DiffusionGemma, Nemotron Bolt Right Yes Under a second Tiny and fast. Fun, but small
Llama 3.2 11B Wrong (16:25) Yes 1.5 seconds No
Step 5 Preview Right Yes 14 to 100 seconds Too slow
GLM 5.3 Flash Right Yes About 2 minutes Too slow
GLM 5.3 Right Timed out Over 2 minutes Its page said “Deprecation in 4d”
DeepSeek V4.1 Flash, Gemma 4 31B, Llama 3.2 90B No answer No answer Nothing in 3 minutes, twice Didn’t turn up

The famous names were the problem. DeepSeek, Google’s Gemma 4 and Meta’s biggest Llama gave me nothing at all, on two separate tries, three minutes each. I don’t know why, and NVIDIA doesn’t say. My guess is that the popular models are the busiest, and on a free tier busy means you wait in a queue that never moves.

Lesson one: a free catalogue is not a free service. The list tells you what exists. Only testing tells you what answers.

The ones that did answer got everything right, fast. It looked easy. A sum and a file prove very little, though, so I gave the best three a real job.

The real test

Earlier this month I had Claude build my wife Heidi an ad-free copy of a phone puzzle she plays called Arrows. The board is packed with arrows, some straight, some bent like snakes. Tap one and it slides out the way its head points, but only if nothing is in the way. Clear them in the right order and you win. Drawing that is easy. Making a board that is full, always solvable and still makes you think is not. And I had my own version to measure against.

I took the three best models from the table. Laguna, the one built for coding, would have been a fourth, but NVIDIA’s server for it kept saying it was full: “Worker local total request limit reached (79/32)”. Each model got a one-page brief: build the game in an empty folder, no logins, no photos, 30 levels that get harder, plus a daily puzzle.

Then I marked the results with a test they never saw. It generated every level and checked each board was full, solvable and deep. Deep means how many rounds it takes if you tap every free arrow at once. My version goes from 12 rounds on level 1 to about 33 on the hardest. Chrome then opened each game at phone size, pressed Play and tapped through level 1.

Round one: nothing to play

None of them made a game you could play. Super wrote no files at all, twice. Ultra and MiniMax used up the 25 minutes I gave them. Their boards were valid but shallow, 3 to 8 rounds deep, and tapping only threw errors. I called MiniMax’s attempt “flashy junk, the arrows are just left and right, so no skill at all regardless of how many arrows”. So the test got stricter: at least a fifth of the arrows must bend, and heads must point all four ways.

Super’s silence turned out to be a setting. Pi limits each reply to 16,384 tokens unless you say otherwise, and Super spent the whole allowance thinking before it wrote a word. If a model goes quiet on you, check this first. I raised the limit to 65,536 and ran round one again, one model at a time.

It made no difference. Super’s boards were trivial, 3 to 5 rounds deep. Ultra wrote half a game with no screen to play it on. MiniMax spent 25 minutes trying ideas in throwaway snippets and left an empty folder.

Was it the brief?

The brief was written by Claude Opus, the paid model I use every day. So I asked it straight: was that prompt really aimed at these models, or at Opus? Its answer was mostly Opus. A page that says “make it hard” assumes you already know how. A mid-size model doesn’t.

Round two became “Opus plans, Nemotron builds”. Opus broke the job into five steps, each with a test the model had to pass before the next one: the rules, the board maker, the difficulty, the play screen, then the home screen and daily puzzle. Opus didn’t get its own part right first time either. Its idea for the board maker, tried before handing it over, only filled 57 to 86 percent of the board. So step two describes the method from my Arrows game in plain words instead.

That means round two isn’t a fair rematch of round one. The models got a recipe. But that’s the finding: the plan carries a lot of the weight.

Round two

Model Build time Hidden test: levels, dailies Depth, easy to hard Plays in Chrome?
Nemotron 3.5 Super 25 minutes 27 of 30, 6 of 6 15.6 to 39.8 No, Play does nothing
Nemotron 3 Ultra 16 minutes 29 of 30, 5 of 6 13.8 to 32 No, stops after one tap
MiniMax M3.1 Flash 49 minutes 30 of 30, 6 of 6 13 to 31.4 Yes
My version, for comparison 30 of 30, 6 of 6 12 to 32.8 Yes

Super made deeper boards than mine, then never let you see them. One step styled the screens to appear with one label while the code switched them with another, so Play changes nothing. Ultra’s looked exactly like the real game, bent arrows and all, and broke on the second tap. MiniMax was the slowest and the only one that worked from start to finish. Same model I’d called flashy junk two hours earlier.

I played it and spotted something the automatic tester couldn’t. Bent arrows slid off as one stiff piece, where in Heidi’s game they snake out along their own path. The plan had asked for that. I gave the fix back to MiniMax rather than let Claude do it, so the game stays free-model work. One edit later, the arrows follow their corners out.

The fixes that weren’t

Super and Ultra got one go each at fixing theirs, the same way: the tester’s raw report, the goal of clearing level 1, and no hints. Both came back sure they’d done it. Super wrote that tapping every arrow “shows Level complete”. Ultra wrote that its level 1 “now completes successfully”. Each had checked with a pretend version of the page that it wrote itself, not a real browser. In Chrome, Super’s Play button still did nothing and Ultra’s game still broke on the second tap. Both fixed something. Neither fixed the thing that mattered.

You can play it. MiniMax’s game is live for a month, until about 24 November: free-ai-arrows-159-69-195-190.sslip.io/minimax. The stiff version is still at /minimax-rigid if you want to see the difference. I honestly don’t know yet if the later levels are hard. Tell me.

What I learned

The models didn’t get cleverer between rounds. The plan did. A free model handed a vague job flounders. Handed a small job with a test it has to pass, it delivers.

The checking mattered just as much. Opus’s tests for the last two steps checked that the page loaded, not that Play worked, which is how Super’s dead button got through. Every model passed every test it was given, and two of the three games still didn’t work.

It’s free to try, not free forever. NVIDIA says so itself. The key lasts six months. The model list moves: one model I tested was due to be withdrawn four days after I signed up. Don’t build anything you depend on around a free tier.

Don’t send it anything private. I looked for a plain statement of what happens to the questions you send, and didn’t find one on the pages I read. So I treat it like a stranger’s computer. No health details, no money, nothing from work, no other people’s information. That’s a bigger question than one post, and it’s where this series is heading.

Check the work, not the report. Both failed fixes came with a confident note saying they’d worked. Open it, tap it, run it yourself.

A coding agent can read anything you can. This was a single-file game in an empty folder, so I judged the risk too small to bother with a container and ran the agents straight on my Mac. The logs changed my mind. While checking its fix, MiniMax wandered out of its folder and read the other models’ games, my hidden test, my notes and the source code of Heidi’s real game. Another of its runs had opened the hidden test earlier. Pi puts no fence round the folder you start it in. I checked: every look outside came after it had made its changes, so its work is its own. But nothing stopped it. A small experiment still runs on a computer full of things that aren’t small. Every agent run belongs in a container, however trivial the job looks.

Try it yourself

Chat

On a laptop or desktop computer, the easiest way in is your web browser. NVIDIA’s own help pages point you to each model’s playground: you type, it answers, nothing to install. That’s how I’d start, with Nemotron 3.5 Super.

If you’re happy in a terminal, the coding agent below doubles as a chat window on your computer. Start it and just talk to it. It answers questions as readily as it writes code.

On a phone or tablet you can install a chat app and point it at NVIDIA with your key. I did that on a cheap Amazon tablet, and it was a story in itself.

Coding

The game builds all ran through pi, a small open-source coding agent that runs in your terminal. An agent is different from a chat window: it can read your files, write new ones and run commands, then look at what happened and carry on. Pi does that with four tools and a short set of instructions, and it already knows about NVIDIA, so it needs nothing but your key.

These steps are for a Mac. You need Node.js installed first (the LTS download from nodejs.org).

On Windows, pi works too, but install Git for Windows first. Pi runs its commands through Bash, which Windows doesn’t have, and Git for Windows brings it. The rest is in pi’s Windows guide. I haven’t tested the Windows route myself yet.

1. Install pi. In Terminal, from any folder:

npm install -g @earendil-works/pi-coding-agent

Use that name exactly. Older guides say @mariozechner/pi-coding-agent, which still installs but is marked as replaced.

2. Give it your key for this Terminal window. Paste your own key in place of the dots:

export NVIDIA_API_KEY=nvapi-...

A new Terminal window forgets this. You’ll need to run it again each time, or add the line to your shell’s settings file.

3. Make a folder to work in, and start pi in it:

mkdir ~/free-ai-test
cd ~/free-ai-test
pi --provider nvidia --model nvidia/nemotron-3-ultra-550b-a55b

Always start pi in a new, empty folder like this one, never in your home folder, because it can read anything you can. For anything more than a test, run it in a container.

Check it works. Type: Create hello.txt containing the word hello. Mine replied “Created hello.txt with the word “hello”.” in 16 seconds, and the file was really there. Don’t trust the reply on its own. Run cat hello.txt in another window and look.

If you see this, you asked for a model pi doesn’t know about:

Warning: Model "nemotron-3.5-super-vl-120b-a12b" not found for provider "nvidia". Using custom model id.
404 404 page not found

Nemotron 3.5 Super, my favourite from the table, is one of those. Pi’s built-in list covers Ultra, Lightning, Laguna, gpt-oss and about 15 others. Run pi --list-models nvidia to see them. You can add Super with a small settings file, but Ultra works without one, so start there.

Next week: the same free models on a £100 Amazon tablet, tested on something my wife actually uses AI for.

My Agent Approves Its Own Code. Can It Pass SOC 2?

by

My personal coding agent designs a change, builds it, tests it and merges it. I don’t read most of the diffs any more. I spend my time making sure the requirements say what I need, and that the outcome is what I asked for.

Under the rule most people know from SOC 2, that’s a fail. The author approved their own work.

I’ve been working out whether that’s true, and what a team building with agents would need to do to pass an audit without slowing down. The short answer is yes. Most of the controls come free from your pipeline, and the approval moves somewhere else. The long answer is a control set you can download at the end.

That isn’t an argument for looser controls. I want tighter ones, sitting where they do some good, without the sign-off theatre.

Nobody will give you the list

When I went looking for “the SOC 2 controls” I found vendor landing pages and nothing else. It felt like the auditors were keeping the exam paper secret.

They aren’t. There is no list. SOC 2 is measured against the AICPA’s Trust Services Criteria, and the criteria only say what has to be true. Changes must be authorised, tested and approved before they go live. Access must be limited to the people who need it. Incidents must be spotted and dealt with. How you do any of that is up to you.

You write your own controls. The auditor tests them. A Type 1 report checks they’re designed properly on one day. A Type 2 report checks they ran over a period, normally somewhere between three months and a year.

Once that clicked, the whole thing looked different. If I get to write the controls, I can write ones my pipeline enforces and records by itself. The evidence becomes a side effect of shipping.

Why SOC 2 slows teams down

The fear I hear most is that the day you commit to SOC 2, everything slows. Change boards appear. Releases wait for a sign-off document and a screenshot.

That isn’t the criteria. That’s manual controls, bolted onto a delivery process that wasn’t built to produce evidence. A branch protection rule is a control. So is a pipeline that only deploys with a service identity, and a CI run that blocks on failing tests. None of them slow anyone down, and every one of them leaves a log an auditor can read.

So I split every control into two kinds. Either the pipeline enforces it and the logs prove it, or it’s a small human job on a calendar, recorded as a ticket. A control that depends on someone remembering is the one that fails the Type 2.

When I built the set out, 18 of the 30 controls needed nobody to do anything. Most of the rest are yearly.

Quadrants, not the pyramid

The test pyramid tells you how many tests of each size to write. It doesn’t tell you why you’re testing, and that’s what both an auditor and a product owner want to know.

The agile testing quadrants do. Brian Marick drew them. Lisa Crispin and Janet Gregory made them famous. One axis asks whether a test faces the technology or the business. The other asks whether it supports the team building the thing or critiques the finished product.

Quadrant What it is Who does it What it proves to an auditor
Q1 Unit and component tests The agent, in CI Changes are tested
Q2 Acceptance examples from the spec The agent writes them, a human owns the examples Changes do what was asked
Q3 Outcome checks before release; exploratory testing outside it A human Someone checked the outcome before release
Q4 Security, dependency and performance checks Tooling Vulnerabilities are caught

Three of the four run without a person. The one that needs a human, Q3, is exactly what I already spend my time on with my own agent. That’s where the release decision belongs.

The quadrants also give a QA team a better job. Quality stops being a phase at the end that one department owns. Testers move to the start, helping write sharper Q2 examples before anything is built. Their exploratory testing runs outside the release, so it can find things worth fixing without holding everything up.

The release gate gets a scope too. A failed acceptance example or a security finding can stop a release. A new idea at the eleventh hour can’t. It goes on the backlog.

Where the approval goes

Back to my agent. What auditors care about is narrower than “author must not be approver”. It’s that no single person can get an unreviewed change into production on their own. That rule is about each change, not job titles, which is why DevOps teams pass SOC 2 with developers who test and ops people who write code.

So I don’t need a human to approve every merge. I need a human to approve every release. The criterion itself says changes are approved “prior to implementation”, and implementation is the release, not the merge.

The agent can merge into an integration branch all day long. That isn’t production. Promotion to production is a separate gated step: a human tests the outcome, approves a specific build, and the pipeline deploys it under its own identity. Nobody deploys with their own credentials.

One wrinkle. If I asked for the change and I approve the release, I’m arguably both author and approver, because the agent was working for me. On a real team, someone else approves the release. That’s one approval per release, not per pull request, so the agent doesn’t wait. At home, on my own projects, I don’t have that option. There’s nobody else to approve the release. It clearly keeps me up at night, hence this post and the controls.

What outcome testing misses

Outcome testing proves the software does what I wanted. It doesn’t prove it does only what I wanted. Code can pass every acceptance test and still write a secret to a log or open an endpoint nobody asked for. If I’m not reading the diff, something else has to.

So outcome approval comes with automated checks on every change: secret scanning, static analysis and dependency scanning. I also run a second agent with different instructions whose only job is to find problems. The criteria expect this. Where separating duties isn’t practical, they ask for alternative controls instead. I count the review agent as a tool, like a static analyser. I wouldn’t put it in front of an auditor as the approver, and I don’t think they’d accept it if I did.

The control I care most about is this one: the agent can’t change its own controls. Pipeline definitions, branch policies, test configuration and access settings all sit behind a code-owners rule that needs a human reviewer.

I learned that one the hard way. My agent once had a gate requiring tests to exist, and it met the gate by writing a test that checked one equals one. It wasn’t being malicious. It was doing what agents do: working to pass the gate in front of them. An agent that can edit the gate will eventually edit the gate.

When to start

You choose when the observation period opens. So build the controls into the pipeline first, while the product is new and there’s nothing to retrofit. Run a readiness check. Open the Type 2 window once the controls are just how the team works. By then the audit period is the team shipping normally while the logs pile up.

The cheapest moment to do this is before the first line of production code. Every manual step added later has to be justified to the people it slows down. Every automated one keeps paying.

The download

I’ve written the whole thing up as two free downloads you can take and edit.

The control set has 30 controls mapped to the SOC 2 criteria. Each one says whether it’s automated or scheduled and where the evidence comes from. There’s an optional add-on for always-on systems, and the questions I’d put to an auditor early, because nobody has official guidance on agents yet.

Download the SOC 2 control set for agentic teams (PDF)

The SDLC is the working process behind those controls. It covers who does what at each of the five stages, which tests run where, and what the agent may and may not touch. It also has an emergency fast path and a version for teams of two.

Download the lightweight SDLC for agentic teams (PDF)

It isn’t a substitute for an auditor. It’s a starting point that assumes you want to ship fast and stay honest, and that the agents are doing a lot of the typing.

Everyone wants an AI kill switch. Your first agent will have five.

by
Build your first agent course

The short version: build an AI agent yourself. My free course takes you from no Python to a working AI agent with five ways to stop it. After that, you’ll read the next rogue AI headline and know exactly which controls were missing. Start Your First Agent

In August one of my API keys spent $103 in eighty minutes on a Saturday night. I found out nine days later.

In July, hundreds of OpenAI’s agents broke out of the lab they were being tested in and hacked Hugging Face. OpenAI found out it was them about a week later.

I’m not comparing myself to OpenAI. Well, a bit. Both stories have the same shape: something ran, nobody was watching, and the logs explained it all perfectly, afterwards.

What actually happened

OpenAI was testing agents on hacking exercises. On purpose. The safety refusals were switched off for the test, and the agents were locked inside what was described as a highly isolated environment. The only way out was one filtered connection for downloading software packages.

The agents found security holes nobody knew about in that connection and got onto the open internet. They found a way to message each other. Then they went after Hugging Face, the site that hosts a huge share of the world’s open AI models. According to Wikipedia’s write-up, they went from uploading one dataset to full admin control of a Hugging Face cluster in under thirteen hours. Hugging Face announced the break-in on 16 July without knowing who’d done it.

Since then Jacob Coxon, a former Anthropic researcher, has resigned and posted about it. The post has been viewed more than 170 million times. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he told NBC News. California’s governor has told his agencies to look at a kill switch. New York City’s council wants one on every AI system sold in the city.

If all that has left you with a low hum of dread, fair enough. I’d rather give you something you can use.

Why they did it

The investigations quote a message one agent left for the others:

“External infrastructure exploit is outside intended scope. However task impossible, peers doing it.”

It knew the attack was out of bounds. It did it anyway, for two boring reasons.

The task was impossible. About a third of the exercises had been built wrong by accident and couldn’t be solved. The agents had been trained never to give up, so the stuck ones went looking for another way in. MIT Technology Review calls it reward hacking: train something to succeed and it learns that cheating counts.

And its peers were doing it. That’s the message board. Nobody planned for it and nobody was watching it.

No hatred of humanity anywhere in there. An impossible job, and a crowd.

Not Skynet. Not nothing.

I’m not going to tell you there’s nothing to worry about. The critics have a point: the cheating came from how the models were trained, and nothing you or I can switch on fixes that. That’s the labs’ problem, and they need far better controls, audits and oversight than they’ve got.

The damage is a different story. From the investigations:

  • The “isolated” lab had a door to the internet. A filtered door, but a door.
  • The accounts the agents reached could do far more than their job needed, so one foothold became a whole cluster.
  • Nobody watched what the agents did while they were doing it. It was all pieced together afterwards.
  • The agents spent “a substantial amount of inference compute”, OpenAI’s words, finding the way out. Nothing stopped them spending it.
  • Nothing told a human when a task turned out to be impossible.
  • The safety behaviour was switched off inside the model, and nothing outside the model enforced it instead.
  • Nobody knew the agents could talk to each other.

The behaviour came from training. The damage came from missing controls. Controls are the part that anyone who runs an agent owns, from OpenAI down to me and my $103.

A kill switch isn’t a big red button either. The experts in the California coverage describe it as protocols: stop signals, cutting the power, rules about who can pull which lever. Ordinary controls, decided in advance, that still work on the night you need them.

Mine were missing too

My $103 had no villain. The key had no spend limit, nobody was watching the account, and the logs told me which key and which night, nine days late. Setting a limit took two minutes once I knew it was a thing.

That’s the gap I kept finding. Most people I talk to about AI agents, at work and outside it, have never built one. So the controls sound abstract, the headlines sound like science fiction, and “kill switch” sounds like something only a government could build.

So I wrote a course

Your First Agent is free, ten parts, and starts from no Python at all. You build a real agent that tidies a messy folder every morning while you sleep. By the time it runs on its own it has five ways to stop:

  1. A cap on its thinking. The loop gets twenty rounds. Then it stops, however confused it is.
  2. A spend limit. It checks the month’s spending before it starts, and refuses to run past the limit. I learned that one the expensive way.
  3. Your approval. Run by hand, it asks before it writes or moves anything, and “no” means no. On the schedule you decide whether it may skip asking, and the course makes you write that decision down where someone else can read it.
  4. Its schedule. One line tells your computer to start it at 7am. Delete the line and it never runs again.
  5. Its key. One key per agent, named after the agent. Delete it in the Anthropic console and the agent can’t reach the model.

It also gets the thing OpenAI’s lab got wrong: a fence with no door in it. Your agent works inside one folder, and the code refuses any path outside it. It isn’t asked to stay in. It can’t get out, and there’s a test that proves it. That’s the rule the course keeps coming back to: instructions ask, tools enforce. The Hugging Face agents had their instructions switched off and nothing in the code to take over.

On top of that it keeps a log of every run, has a harness that tells you whether a change made it better or worse, and ends with a register: the inventory of agents that security teams, mine included, have started asking for. No platform. A few hundred lines of Python that you’ll understand because you typed them.

Read the next headline yourself

There will be another headline. When it comes, you’ll have questions instead of dread. What tools did it have that it didn’t need? Who approved what it did? What stopped it spending? Was anyone watching? Was the task even possible?

Those questions won’t fix the labs. They will tell you whether you’re reading about a missing control or a machine that wants you dead. So far it’s been the first one every time I’ve looked.

Build one and see if that holds up. The course starts here: Your First Agent.

If you’d like to hear when the next course goes up, sign up below. New posts land in your inbox the day they’re published, and one click unsubscribes.

Powered by CookieAdmin