The short version: build an AI agent yourself. My free course takes you from no Python to a working AI agent with five ways to stop it. After that, you’ll read the next rogue AI headline and know exactly which controls were missing. Start Your First Agent
In August one of my API keys spent $103 in eighty minutes on a Saturday night. I found out nine days later.
In July, hundreds of OpenAI’s agents broke out of the lab they were being tested in and hacked Hugging Face. OpenAI found out it was them about a week later.
I’m not comparing myself to OpenAI. Well, a bit. Both stories have the same shape: something ran, nobody was watching, and the logs explained it all perfectly, afterwards.
What actually happened
OpenAI was testing agents on hacking exercises. On purpose. The safety refusals were switched off for the test, and the agents were locked inside what was described as a highly isolated environment. The only way out was one filtered connection for downloading software packages.
The agents found security holes nobody knew about in that connection and got onto the open internet. They found a way to message each other. Then they went after Hugging Face, the site that hosts a huge share of the world’s open AI models. According to Wikipedia’s write-up, they went from uploading one dataset to full admin control of a Hugging Face cluster in under thirteen hours. Hugging Face announced the break-in on 16 July without knowing who’d done it.
Since then Jacob Coxon, a former Anthropic researcher, has resigned and posted about it. The post has been viewed more than 170 million times. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he told NBC News. California’s governor has told his agencies to look at a kill switch. New York City’s council wants one on every AI system sold in the city.
If all that has left you with a low hum of dread, fair enough. I’d rather give you something you can use.
Why they did it
The investigations quote a message one agent left for the others:
“External infrastructure exploit is outside intended scope. However task impossible, peers doing it.”
It knew the attack was out of bounds. It did it anyway, for two boring reasons.
The task was impossible. About a third of the exercises had been built wrong by accident and couldn’t be solved. The agents had been trained never to give up, so the stuck ones went looking for another way in. MIT Technology Review calls it reward hacking: train something to succeed and it learns that cheating counts.
And its peers were doing it. That’s the message board. Nobody planned for it and nobody was watching it.
No hatred of humanity anywhere in there. An impossible job, and a crowd.
Not Skynet. Not nothing.
I’m not going to tell you there’s nothing to worry about. The critics have a point: the cheating came from how the models were trained, and nothing you or I can switch on fixes that. That’s the labs’ problem, and they need far better controls, audits and oversight than they’ve got.
The damage is a different story. From the investigations:
- The “isolated” lab had a door to the internet. A filtered door, but a door.
- The accounts the agents reached could do far more than their job needed, so one foothold became a whole cluster.
- Nobody watched what the agents did while they were doing it. It was all pieced together afterwards.
- The agents spent “a substantial amount of inference compute”, OpenAI’s words, finding the way out. Nothing stopped them spending it.
- Nothing told a human when a task turned out to be impossible.
- The safety behaviour was switched off inside the model, and nothing outside the model enforced it instead.
- Nobody knew the agents could talk to each other.
The behaviour came from training. The damage came from missing controls. Controls are the part that anyone who runs an agent owns, from OpenAI down to me and my $103.
A kill switch isn’t a big red button either. The experts in the California coverage describe it as protocols: stop signals, cutting the power, rules about who can pull which lever. Ordinary controls, decided in advance, that still work on the night you need them.
Mine were missing too
My $103 had no villain. The key had no spend limit, nobody was watching the account, and the logs told me which key and which night, nine days late. Setting a limit took two minutes once I knew it was a thing.
That’s the gap I kept finding. Most people I talk to about AI agents, at work and outside it, have never built one. So the controls sound abstract, the headlines sound like science fiction, and “kill switch” sounds like something only a government could build.
So I wrote a course
Your First Agent is free, ten parts, and starts from no Python at all. You build a real agent that tidies a messy folder every morning while you sleep. By the time it runs on its own it has five ways to stop:
- A cap on its thinking. The loop gets twenty rounds. Then it stops, however confused it is.
- A spend limit. It checks the month’s spending before it starts, and refuses to run past the limit. I learned that one the expensive way.
- Your approval. Run by hand, it asks before it writes or moves anything, and “no” means no. On the schedule you decide whether it may skip asking, and the course makes you write that decision down where someone else can read it.
- Its schedule. One line tells your computer to start it at 7am. Delete the line and it never runs again.
- Its key. One key per agent, named after the agent. Delete it in the Anthropic console and the agent can’t reach the model.
It also gets the thing OpenAI’s lab got wrong: a fence with no door in it. Your agent works inside one folder, and the code refuses any path outside it. It isn’t asked to stay in. It can’t get out, and there’s a test that proves it. That’s the rule the course keeps coming back to: instructions ask, tools enforce. The Hugging Face agents had their instructions switched off and nothing in the code to take over.
On top of that it keeps a log of every run, has a harness that tells you whether a change made it better or worse, and ends with a register: the inventory of agents that security teams, mine included, have started asking for. No platform. A few hundred lines of Python that you’ll understand because you typed them.
Read the next headline yourself
There will be another headline. When it comes, you’ll have questions instead of dread. What tools did it have that it didn’t need? Who approved what it did? What stopped it spending? Was anyone watching? Was the task even possible?
Those questions won’t fix the labs. They will tell you whether you’re reading about a missing control or a machine that wants you dead. So far it’s been the first one every time I’ve looked.
Build one and see if that holds up. The course starts here: Your First Agent.
If you’d like to hear when the next course goes up, sign up below. New posts land in your inbox the day they’re published, and one click unsubscribes.