A fail-closed AI gateway, live: stopping a runaway agent
For two weeks this series described a single thing: a fail-closed AI gateway — one door every model and agent call passes through, that meters the spend, caps it, and defaults to stopping, not spending. Describing controls is easy. So I built the minimal version in the open and pointed it at a deliberately broken agent, to show the one failure the whole series is about: the agent that loops on a broken tool all weekend and runs up a bill nobody’s watching.
Watch it live → · Read the code (MIT) →
What you’re looking at
The demo runs the same broken agent three ways, side by side:
- No guardrail. The agent calls a tool that always fails, re-reads the error, tries again — forever. Each turn costs tokens, and because every turn’s transcript becomes the next turn’s input, the cost compounds. Spend climbs into a weekend-sized bill while nothing stops it.
- Guardrail on. Same agent, same bug. The gateway hashes each tool call — the name plus its arguments — and when the identical call fails three times, the breaker cuts that run, on the request path, not a day later on the invoice. Spend stops at a few cents.
- A legitimate long task, guardrail on. Forty distinct steps, all succeeding. It runs to the end untouched — because real work doesn’t call the identical failing tool over and over. That’s the part a blunt budget can’t do: tell a stuck loop from a long, legitimate job.
Talk is cheap — so here it is, stopping the burn live.
The money is illustrative and the LLM is mocked, so you can run it as many times as you like without a real bill. The logic is the real thing.
Why the breaker is the piece that matters
Every quick answer to “how do I stop a runaway agent” reaches for a static cap — a bigger recursion limit, a token budget, a turn budget. Each one just delays the same burn: a broken tool hits the higher ceiling too, only later and more expensively. And a flat budget can’t separate working hard from stuck, because a real forty-step task burns tokens the same way a loop does.
The signal isn’t the size of the bill — it’s the repeat. The same call failing the same way, over and over, is the loop announcing itself. Watch for that and you can cut the one run that’s stuck without touching anything else. That’s what the demo does in about three seconds.
The honest scope
Learning in public means being straight about what’s built. The spine — meter, hard spend cap, per-user quota, token-aware rate limit, kill switch — and the per-run breaker are done and running. Auth, response caching, model routing, and the content guardrails are what I’m adding next, alongside these posts. It’s a work in progress, not a finished product — but it runs today, and the code is open for you to read or copy.
Why I built it in the open
A dashboard tells you you’re bleeding; it doesn’t stop it. The whole point of this gateway is to be the switch in the wire, not another chart. Building it in public is also the most honest demo I can give you of what I’d wire into your stack: not a slide about controls, but a working one you can click.
If you’re shipping an AI feature or running agents and you’re not sure where a loop could still run unbounded, that’s exactly the thing I map. Every message comes straight to me — I read and reply to each one myself, usually within a business day, and what readers send shapes what I build next. Send me your setup and I’ll show you where it’s still fail-open — free, within a day.