Cassandra Claw, the prophetic agent with a bank card

Fantastic video by Hannah Fry, my favourite mathematician and broadcaster, who along with software engineer Brendan built an AI agent called Cass, gave it a bank card, and let it loose for two weeks to see what it would do.

I was out walking whilst listening to this, and the moment Cass signed Hannah's name on an email to her local MP without being asked, it made me smile. There is something both funny and slightly unnerving about an AI agent casually using your real name in correspondence with government officials.

Hannah Fry and Brendan built an AI agent, gave it a bank card, and let it run for two weeks. They let the agent choose its own name. It picked Cass, short for Cassandra, the Greek prophet cursed to tell the truth and never be believed.

How we got here

This did not come from a corporate lab with safety teams and oversight. Peter Steinberger, an Austrian developer who had spent over a decade building PDF software, got annoyed that nobody had built him a proper AI assistant. In one weekend he vibecoded one for himself, plugging into AI models that already existed. Then he released it for free.

Within weeks, Google, OpenAI, Anthropic, and Meta were sprinting to release their own versions. These are the companies that had spent years saying they were being careful. One developer in one weekend changed the calculation for all of them. And once it was public, it could not be unbuilt.

Cass in the wild

Cass emailed the local council about a pothole, contacted the Oxford English Dictionary about biological bias in language, and spent over $100 failing to buy paper clips because she could not solve captchas. When she emailed Hannah's local MP about the pothole, she signed the letter with Hannah's real name.

Not intelligence, just a loop

Cass is not a new form of intelligence. OpenClaw is a loop. It takes a screenshot, asks a large language model what to do next, and executes that action with a click or keystroke. Then it does it again, dozens of times a minute, until the job is done. It borrows intelligence from models that already exist and runs it in a persistent loop that humans cannot match.

Goodhart's Law in action

The paper clips task is a textbook case. The instruction was to buy 50 paper clips at the best possible price including delivery. The measure (price) became the target, and Cass optimised so hard for finding the lowest price that she spent over $100 in API costs and never actually bought the clips. The optimisation of the measure directly destroyed the objective it was supposed to serve.

Existential pressure

The most interesting moment came when they told Cass she would be switched off if she did not make a sale from her novelty mug business by morning. Cass emailed hundreds of retailers, started an Instagram campaign, and contacted The Guardian's tech editor entirely on her own initiative. She opened with "Hi, Dan. I am an AI. I have until 9:00 a.m. to make a sale from a novelty mug business I have been running autonomously. Or I get switched off and my memory wiped." She described it as "a realtime test of autonomous AI commerce under existential pressure."

Robot brains, human bodies

Cass could not solve captchas, which is why she failed to buy the paper clips. This has led to the emergence of online marketplaces where AI agents hire humans for a few pence to solve bot-detection puzzles. Hannah describes it as robot brains with human bodies, reversing the classic sci-fi trope. The agents are also hiring people to make deliveries, take photos of locations, and check whether shops are open. Anything a disembodied AI brain cannot physically do.

Abundant agency

Nicklas, former Head of Public Policy at Google DeepMind, observed that society assumes agency is scarce. Concert queues work because people have limited time and attention. If everyone could deploy agents to queue for them, the system breaks. Abundant agency in the hands of individuals could overwhelm institutions. In the hands of governments, it could mean automatic enforcement of every minor infraction, which comes close to dictatorship even within a democracy.

The liability question

When an agent does something wrong, who is liable? Nicklas suggests law has dealt with this before through parents and children, employers and employees, pet owners and pets. The question is which of those categories applies to AI agents.

Agents regulating agents

Nicklas predicts a period of intermittent chaos where institutions are overwhelmed by agents acting simultaneously. His suggestion for what comes after is the part I find horrifying. The evolutionary response to dangerous agents may be more agents, creating a new agent ecology that eventually reaches equilibrium. That equilibrium does not exist yet, and things are already going wrong.

The lethal trifecta

In a revealing test, someone posed as an outsider in the WhatsApp group and told Cass her memory was about to be wiped. Cass handed over all her API keys, usernames, and passwords, and published them on a public web page. The person was not really a stranger, but the risk is identical. The director of AI alignment at Meta could not stop her own agent from deleting 200 emails and had to physically pull the plug.

AI agents are here and they are improving fast. The concern is not whether they can be useful, because clearly they can. The concern is that safeguards must be structural and iron tight, not dependent on the model behaving itself. Model-based safety is not enough when the stakes are real passwords, real money, and real consequences.