Putting a hard ceiling on AI spend (so it fails closed)

July 9, 2026

Day 3 gave us the door: one gateway every LLM call passes through. Today we bolt on the first thing that hangs there — the control that answers Day 2’s entire problem. It’s two separate controls, and the whole point is that you need both.

Two different ceilings

1. A global hard spend cap. The total ceiling. The gateway tracks cumulative spend over a window (say, per day or per month), and when it crosses your limit it stops serving — returns an error instead of forwarding the call. That’s the “fails closed” you’ve been promised since Day 2: the bill literally cannot exceed the number you set.

2. Per-user (per-key) quotas. A separate limit per identity. Each app, customer, or API key gets its own budget and its own rate limit, so no single one of them can consume the whole thing.

Teams reach for one and think they’re covered. They’re not:

• A global cap alone means one abusive user (or one leaked key running flat out) can burn the entire day’s budget in minutes — and now the cap trips for everybody. You’ve turned a cost problem into an outage for your real users.
• Per-user quotas alone don’t stop an aggregate blowout — a thousand legitimate users on a viral day, or a fleet of new signups, each politely under their quota, still sums past your budget.

Put both in and Day 2’s five failure modes finally hit a wall: the agent loop and retry storm burn through one key’s quota and stop; the leaked key and the abusive user are boxed to their own limit; the viral spike runs into the global cap instead of the void.

You already know this control

This is rate limiting and quota management — the exact discipline you’d put on any API. Token buckets, per-key budgets, requests-per-minute. The LLM route isn’t special; it’s just an unusually expensive API, which is precisely why it’s the one route people forget to limit. And the global cap is an old friend too: a circuit breaker. If you’ve built one for a flaky downstream, you’ve built this.

What it looks like at the gateway

Because everything already flows through one door (Day 3), this is configuration, not surgery. In a self-hosted LiteLLM proxy, each user/app gets a virtual key with its own ceiling:

// POST /key/generate — a virtual key per user/app, each with a hard ceiling
{
  "models": ["gpt-4o", "claude-sonnet"],
  "max_budget": 5.00,        // hard $ ceiling for THIS key
  "budget_duration": "1d",   // resets daily
  "rpm_limit": 60,           // requests/min — real-time burst guard
  "tpm_limit": 200000        // tokens/min
}
// …plus a global/team budget so the aggregate can't run away either.

Cloudflare AI Gateway gives you the rate-limiting half (requests per window) and the spend visibility; LiteLLM/Portkey give you hard per-key budgets. Either way the shape is the same: a ceiling per key, and a ceiling overall.

The one real decision: what happens when it trips

Fail-closed doesn’t have to mean a blank error. When a cap is hit, you choose:

• Hard stop (a 429) — correct for a learning account, a dev sandbox, or any non-revenue key. Better an error than a surprise invoice.
• Degrade — fall back to a cheaper model, or return a cached/canned “we’re at capacity” response — a reasonable choice for production, where a full stop is its own outage.

That’s the same deliberate call from the unlimited-liability post: you don’t inherit “unbounded” by default, but you do decide, per key, whether the ceiling stops or softens.

One honest caveat: budget caps evaluate over a window, so a hard burst can overshoot slightly before the cap catches up. That’s exactly what the real-time per-minute rate limit is for — it bounds the burst while the budget bounds the total. Belt and braces.

Next in the series — Day 5: caching and model routing — because the cheapest way to stay under a cap is to not spend the money in the first place.

Following along? Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your AI setup and I’ll tell you where it’s still fail-open — free, within a business day.