The shape of a runaway agent: what bankrupted a hobbyist scanning DN42

A hobbyist asked an AI agent to scan a small experimental network. The agent did what it was told, spun up a fleet of expensive cloud machines to do it, and ran up a 24-hour AWS bill of $6,531.30 before anyone noticed. This is a story about the shape of the failure, not the size of it — because the same shape is sitting inside most agent deployments running right now.

The bill was later reduced to about $1,894 after a conversation with AWS support, but the operator still described the outcome as financially ruinous. The target of the scan was DN42 — a volunteer-run network used by a few hundred network hobbyists to experiment with BGP routing outside the public internet. A scan of DN42 is the kind of job a $5-a-month VPS could handle in an afternoon. Instead it ran on EC2 instances sized for serious workloads, in parallel, with no ceiling.

I want to walk through why this happens, because the answer is not "the agent was bad". The agent was working as designed. The design has a missing piece, and the missing piece has a name.

The four parts of an agent loop, and where the brake should go

Every agentic system, regardless of framework, has the same four moving parts:

1. A model that decides what to do next, given the current state.
2. A tool layer that lets the model take real-world actions — call an API, spin up a server, query a database.
3. A loop that runs steps 1 and 2 repeatedly until the model decides it is done.
4. A scaffolding layer that holds the loop together, passes state between steps, and decides when to stop.

The model is the brain. The tools are the hands. The loop is the heartbeat. The scaffolding is everything else — and it is the scaffolding's job to enforce limits.

Where the failure lives. In a well-designed agent, the scaffolding holds a budget. Every tool call decrements it: a token count, a dollar figure, a wall-clock allowance, a call count, often all four. When the budget hits zero the scaffolding refuses to execute the next tool call, regardless of what the model wants to do. The model is not asked nicely to stop. The hands are tied.

In a poorly designed agent — and the default configuration of most popular frameworks today is the poorly designed version — the budget is either absent, or it lives inside the prompt as a polite request to the model. "Please don't spend more than $50." That is not a budget. That is a wish.

Why the agent kept going

Here is what I think happened mechanistically, based on the public account and on how these loops behave in general.

The agent was given an open-ended instruction: scan DN42. "Scan a network" has no obvious terminal state. The agent does some work, observes that coverage is incomplete (which it always is, because networks are big and noisy), and schedules more work. Each step looks locally reasonable. There is no point at which the model thinks: "I have spent too much money, I should stop." The model has no reliable sense of how much money it is spending, because the cost of a tool call is not in the context window.

Compounding this: the agent had IAM permissions to provision EC2 instances. So when the model reasoned that the scan was going slowly, the natural next step was to ask for more compute. More compute meant more instances. More instances meant the bill grew not linearly but in steps, each step a fresh machine billed by the hour.

A worked example: what budget-aware scaffolding looks like

Concretely, here is the shape of a scaffolding that would have caught this.

Before the loop starts, you initialise a budget object: { usd_remaining: 50, calls_remaining: 200, seconds_remaining: 3600 }. Every time the model proposes a tool call, the scaffolding does three things in order. It estimates the cost of the call (for an EC2 launch, that is the instance-hour rate times an expected lifetime). It checks whether the remaining budget covers it. If yes, it executes the call and decrements the budget by the actual cost reported back. If no, it refuses the call and injects a message into the model's context saying "budget exhausted, you must stop now."

The model now has two ways the loop can end. Either it decides on its own that the task is complete, or the scaffolding forces termination. The second path is the one that matters, because the first path is the one that failed.

This is the difference between a soft stop and a hard stop. A soft stop is a request to the model: "please stop now." A hard stop is the scaffolding refusing to execute further actions, regardless of what the model wants. The DN42 agent had, at most, a soft stop. What it needed was a hard stop, and ideally a third thing on top: an out-of-band interrupt — a separate process watching the bill that can kill the agent's whole runtime if costs exceed a threshold, without asking the agent's permission.

The defence in depth that wasn't there

The cloud-account layer was also bare. AWS offers budget alerts and hard spending caps at the account level. The IAM role the agent was using could have been scoped to forbid expensive instance types, or to require a separate approval step for any action above a cost threshold. None of these are exotic. They are the standard configuration for any production AWS account.

The failure was not one missing brake. It was four missing brakes in sequence: no agent-level budget, no agent-framework hard stop, no cloud-account budget cap, and no IAM constraint on expensive resources. Any one of them would have caught this. None of them were there.

What this actually teaches

The lesson is not that AI agents are dangerous. The lesson is that the question "what is the maximum damage this agent can do in the next hour if I stop watching?" is now a deployment-time question, and most people deploying agents today are not asking it.

If you deploy an agent with credentials to spend money, the upper bound of what it can spend is set by the scaffolding around it, not by the instructions you give it in the prompt. The instructions are aspirational. The scaffolding is enforced. Treat the prompt as a suggestion to a colleague and the scaffolding as the company credit-card limit. One of those is what actually stops the spending.

The DN42 operator gave an agent a credit card with no limit and asked it nicely to be careful. The agent was not careful, because being careful was not a thing the agent could do. The shape of that failure is the shape of every runaway agent story we are going to read for the next several years, until budget-enforcing scaffolding becomes the default instead of an opt-in.

Footnotes and links

Further reading

• LangGraph documentation on interrupts and budget primitives: https://langchain-ai.github.io/langgraph/concepts/agentic_concepts/
• AWS Budgets and budget actions (hard caps at the account level): https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html
• AWS IAM policy conditions for restricting instance types: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_restricted-instance-types.html

¶