A fail-closed AI gateway you can copy
Fifteen lessons of individual controls. This is where they stop being a list and become one thing: a single gateway every LLM call passes through that is authenticated, metered, capped, rate-limited, cached, guarded — and defaults to closed. If you’ve read along, you’ve already built every piece. Here’s the whole architecture on one page.
The stack, as one door
The gateway is the spine; each control hangs on it, in order:
- Auth (Day 13). Identify the caller; hand them a per-user virtual key. Nothing anonymous reaches the model.
- Rate limits (Day 12). Tokens-per-minute, requests-per-minute, and concurrency — per identity, so no one caller can outspend their share.
- Spend caps + quotas (Day 4). A per-user daily budget and a global hard cap, with the deliberate fail-closed choice: hard-stop or degrade when a limit trips.
- The per-run breaker (the kill switch). A circuit breaker on the request path that halts a runaway run mid-flight — the identical-tool-call signal, the per-run budget, the fan-out cap.
- Cache + routing (Day 5). Cut the calls you actually make: response cache, prompt cache, cheap-model routing, batching.
- Guardrails (Days 6–10). Treat retrieved content as untrusted, grant least agency, moderate output, and never let the model take a binding action without a deterministic check in your code.
- Key management (Day 11). The real provider key lives server-side, in one place, scoped — your app only ever holds virtual keys.
- Observability. Log tokens, dollars, identity, and
run_idon every call. The dashboard isn’t a separate reporting tool bolted on afterward — it reads the same data the gateway enforces on. One source of truth for what you spent and what you stopped.
What “fail closed” means as a whole
Each control has its own trip condition. The system-level rule that ties them together is this: whenever anything is in doubt — a cap is hit, abuse is detected, the breaker fires, auth is missing, the budget is exhausted — the default action is to stop or degrade, never to run unbounded. That is the exact inversion of the Day 2 default, where every one of those conditions quietly meant “keep spending.” Fail-open is what you inherit; fail-closed is what you build.
You don’t write it all from scratch
Learning in public means being honest about the build, so: you assemble this from existing parts, not a blank file.
One authed, metered, capped door — and it defaults to closed.
- Self-hosted spine: LiteLLM. It already gives you virtual keys, per-key budgets,
tpm/rpmlimits, caching, model routing, and logging. You add your auth, your guardrails, and your per-run breaker around it. - Hosted parts: Cloudflare AI Gateway, Portkey, or Helicone cover rate limiting, caching, and observability; you still own the caps, auth, and abuse logic.
- The reference build — now live. I built the minimal, copyable version in the open: Guardrail — one gateway that meters every call, enforces a hard cap, and trips the per-run breaker on a stuck agent. The demo stops a runaway agent in front of you; the code is open source (MIT). Being honest about the build: the spine and the breaker are done — auth, caching, and the guardrails are what I’m adding next, alongside these posts. A work in progress, not a finished product I’m handing down — but it runs today.
The whole series in one sentence
Everything above compresses to this: every LLM call goes through one authenticated, metered, capped door that defaults to closed. If you take nothing else from two weeks of posts, take that sentence and the architecture under it.
Next in the series — the part the technical playbook can’t fix. All of this is the easy half. The hard half is that on a stretched team, these controls are nobody’s job — so next we leave the code and talk about why the caps still don’t get built.
If you’ve got pieces of this — a gateway here, a cap there — but no single fail-closed door, wiring them into one is exactly what I do. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your AI setup and I’ll map it against this architecture and show you what’s still fail-open — free, within a business day.