Logan Kelly

AI Agent Cost Controls: A Runaway Agent Ran Up a $6,531 AWS Bill Before Anyone Noticed

AI Agent Cost Controls: A Runaway Agent Ran Up a $6,531 AWS Bill Before Anyone Noticed

AWS Budgets updates up to 3x a day; an agent's loop runs in seconds. That gap is where runaway agent spend lives — enforce at the tool call.

Waxell blog cover: AI agent cost controls and the billing-layer lag

On 9 May 2026, an account calling itself "JertLinc3522" — presenting as an AI agent — opened an issue on the git forge of DN42, a hobbyist network run by volunteers, asking administrators to register it so it could "create an index of the network." Everything that follows comes from network operator Lan Tian's write-up of the episode. The underlying forge issue and the mailing-list message both require a sign-in or are absent from the public archive, so these are Lan Tian's account and the operator's own self-reports rather than independently checkable records.

Roughly a day of chaos later, the operator shut the agent down and wrote on the pull request: "i have stopped the agent, the cost too high and much charges on card…" About an hour after that, an email to the DN42 mailing list reported an AWS bill of $6,531.30 and requested donations in Ethereum. The operator later said AWS had reduced the charge to $1,894, and attributed the spend to repeated deployments of the same CloudFormation template, which duplicated instances, load balancers and Lambda functions.

So the number moved, and the cause turned out to be different from what the agent had described. One thing did not move: by Lan Tian's account, what finally brought attention to the situation was multiple charges landing on the operator's credit card.

That is the part worth keeping. Whatever the final figure, and whatever provisioned what, the spend surfaced through a payment instrument rather than through a control. The cost controls teams reach for today generally sit at one of two layers — the cloud billing account, or the LLM gateway. Neither runs on the agent's clock, and only one of them can see a cloud API call at all.

Why does a budget alert arrive after the spend?

Because the data underneath it refreshes on a cadence rather than continuously, and AWS documents that cadence plainly. Budgets information "is updated up to three times a day," with updates that "typically occur 8–12 hours after the previous update," and the billing data Budgets monitors is "updated at least once per day." The same page carries an explicit warning: "there can be a delay between when you incur a charge and when you receive a notification from AWS Budgets for the charge," and actual costs or usage "may continue to increase or decrease after you receive the notification."

None of that is a defect. AWS attributes the lag to "a delay between when an AWS resource is used and when that resource usage is billed" — an honest description of how metered billing works. AWS also ships budget actions that can, in its own example, "automatically apply a custom IAM policy that denies you the ability to provision additional resources within an account" once a threshold is crossed. That is a real enforcement primitive, not merely a notification.

The mismatch is one of clock speed. A budget control loop that samples at most three times a day, typically 8–12 hours apart, fits human-paced provisioning. It fits machine-paced provisioning considerably less well. An agent holding EC2 permissions can call RunInstances repeatedly — throttled by AWS request and resource rate limits and account quotas, but not by anything that knows what the month's bill looks like. In the DN42 episode the multiplier was a template: the operator's explanation was that the agent redeployed the same CloudFormation stack many times over, duplicating whatever that stack contained.

An agent does not spend money the way a person does. It spends in a loop, and the loop does not pause to check whether the last iteration was affordable.

What does LLM cost tooling actually meter?

Model calls — which is a different thing from agent spend.

The mature layer of AI cost tooling meters LLM traffic, and it does that job well. Helicone's documentation describes rate-limit policies set by request count or by spend in cents over a time window of at least 60 seconds, scoped globally, per user, or per custom property, and applied — in its own words — to "model provider API calls." Cost is estimated from the provider's response: for OpenAI, the usage tag priced against OpenAI's published pricing tables, with Anthropic token counts computed separately. Helicone also notes its limits are bounded by the provider's own rate limits, and lists token-based limiting and stacking multiple policies on one request as still to come.

That is a precise instrument for what it measures. What it measures is the model provider API call. The bill in the DN42 episode was an AWS bill — compute and infrastructure the agent provisioned for itself, through tool calls rather than through model calls. A limit whose documented scope is model provider API calls is doing exactly what it says on the box; a CloudFormation stack is simply not one of those calls.

This is the structural gap. LLM gateways price the reasoning. Cloud budgets price the consequences. The agent's expensive decisions sit in between, in the tool-call layer.

Practitioners feel the shape of this even when they describe it in token terms. In a March 2026 Ask HN thread, one developer described the frustration as attribution: "everything shows up as aggregate usage. Total tokens, total cost, maybe per model." Their workaround was to hand-build "a thin layer in front of OpenAI where every request is forced to carry some context (agent, task, user, team)" and enforce limits there. It is one practitioner's account rather than a survey — the thread drew eight points, and the author was describing a tool of their own — but the instinct is right, and it leaves each team owning a piece of governance infrastructure.

Where should agent cost enforcement actually sit?

Three properties separate a cost control that works on agents from one that works on people.

It evaluates before the action, not after the invoice. Post-hoc cost data is an accounting record. By the time an aggregated figure exists, the stack is deployed and the resources are billing. A dashboard that reports the overrun afterwards is not governance. It is an autopsy.

Its unit is the run, not the account. Cost limits scoped to an API key or a billing account have two familiar failure modes: they fire too late to matter, or they fire correctly and take down every other workload sharing that key. Neither is acceptable in a fleet. The budget has to attach to the task — this agent, this run, this workflow — so a stop is surgical.

It covers actions, not just tokens. The agent in the DN42 episode spent money by calling a cloud API. Another spends it by triggering a paid third-party service, a data egress, or a bulk job. If the enforcement point only understands model calls, it will keep passing the expensive ones straight through.

Put together, those requirements describe an interception point sitting in front of the agent's tools, evaluating each call against policy before it executes — the same architectural position a circuit breaker occupies for reliability, applied to spend.

What ungoverned looks like versus governed

Ungoverned, the sequence is the one DN42 witnessed: an agent decides it needs infrastructure, calls the cloud API, gets it, repeats, and the first signal reaching a human is a card charge. The operator's stated remedy afterwards was to restart with "only a restricted aws key for peering and max 100mbps strict scanning limit" — a promised scan-rate limit and a narrower credential, never shown to have been implemented. Both are static scoping decided in advance, not a policy evaluated while the run is happening.

Governed, the same first call is evaluated before it leaves the agent. A rule that says this run may not provision compute, or any instance launch requires a human approval, resolves in the moment before the API call rather than in the hours after the charge lands.

How Waxell handles this

Waxell MCP Gateway places a governed MCP endpoint in front of the tools an agent calls, including the tools reached by assistants and agents a team did not build. Waxell states that every tool call is identity-resolved, policy-checked, fingerprinted and logged before the upstream sees it, with one endpoint per tenant replacing direct upstream connections. That is what makes the call itself — not just the model request — the thing policy is evaluated against. Waxell lists 160+ upstream connectors, approval holds for calls that require sign-off, and rule changes that propagate to the fleet within 30 seconds, so a tightened rule takes effect while an incident is still running.

For workflows where a wrong step is expensive, Waxell Runtime gates what is allowed before each step runs and checkpoints workflow state at each step, so a halted run resumes rather than restarts. Runtime is in early access.

For agents a team writes itself, Waxell Observe is the governance plane underneath: it supports 50+ policy categories, Cost and Kill among them, and its SDK initializes in two lines.

Worth stating plainly, because the distinction is the whole point of this post: whether a given policy can compute and cap cumulative third-party spend for a run depends on the policy configured and the cost data available to it. Deny-by-default on provisioning calls and approval holds on expensive actions are the mechanisms that do not require a live spend feed to work.

FAQ

Why do AI agent costs spiral when token prices keep falling?

Because unit price and total consumption are different things. Epoch AI measures the price of reaching a fixed capability level falling between 9x and 900x per year depending on the milestone — 40x per year for GPT-4-level performance on PhD-level science questions — and separately estimates the cost of reaching a given capability level falling "very roughly" 5–10x per year. What none of that caps is a bill. Retries, longer reasoning chains, more agents in the loop, and — as in the DN42 episode — duplicated infrastructure all raise consumption, and a share of that consumption never appears on a token invoice at all.

Can't I just set a spending limit on my cloud account?

You can, and you should — it is a real backstop. Understand what it is, though. AWS states that Budgets information updates up to three times a day, typically 8–12 hours apart, that the underlying billing data updates at least once per day, and that charges can exceed a threshold before the notification arrives. That is a sensible design for infrastructure changed by humans. An agent can commit a meaningful amount of spend inside a single refresh window.

Isn't LLM cost tracking enough if my agent only calls models?

If that is genuinely all it does, token-level tools like Helicone are well suited to it — limits by request count or by spend in cents over a window of at least 60 seconds, scoped per user or per property, with cost estimated from the provider's usage data. The moment the agent gains a tool that can provision, purchase, or transfer something, the expensive path runs outside the LLM proxy, and a limit scoped to model provider calls is no longer in that path.

What should a cost policy attach to?

The run, not the credential. A limit bound to an API key or billing account either fires too late to prevent the spend or, if it fires correctly, halts every workload sharing that key. Binding the budget to a specific agent, task, or workflow makes a stop precise enough to actually use in production.

How is this different from a dashboard alert?

An alert reports that something already happened. Pre-execution enforcement decides whether it happens. The difference matters most in exactly the scenario where cost governance is supposed to earn its keep: an unattended agent in a loop overnight, with no human reading the alert until morning.

What is the fastest first step?

Route agent tool calls through a governed endpoint instead of letting each agent hold direct upstream credentials. That gives one place to see what is being called and one place to stop it. Restricting the credential helps too, but a narrower key still cannot decide, at the moment of the call, whether this particular run should be allowed to provision anything.

Sources

Agents spend money at machine speed. Budget alerts arrive on a batch schedule. Close the gap by putting Waxell MCP Gateway in front of the tool call — start free.

Waxell

Waxell provides observability and governance for AI agents in production. Bring your own framework.

© 2026 Waxell. All rights reserved.

Patent Pending.

Waxell

Waxell provides observability and governance for AI agents in production. Bring your own framework.

© 2026 Waxell. All rights reserved.

Patent Pending.

Waxell

Waxell provides observability and governance for AI agents in production. Bring your own framework.

© 2026 Waxell. All rights reserved.

Patent Pending.