Rate limiting you already know — now for tokens

July 24, 2026

Yesterday we made the key safe. Today: how fast any one holder can spend through it. And here’s the good news — you already know how to do this. You’ve rate-limited APIs before: requests per minute, token buckets, per-key quotas. The LLM route needs the identical discipline. It just meters a different unit.

Requests-per-minute doesn’t bound cost

The trap is using your normal request-count limit and thinking you’re covered. You’re not, because cost per request varies enormously. One call might be 200 tokens. The next — a giant retrieved context plus max_tokens=4000 of output — might be 200,000. A limit of “60 requests per minute” says nothing about whether that’s 12,000 tokens or 12 million.

Same rate limiting you already do — new unit: tokens.

So a buggy agent loop or a deliberate abuser can stay politely under your request limit and still torch the budget. You have to limit the dimension that actually costs money: tokens (and, cumulatively, dollars).

The token-aware controls

All of these live at the gateway, keyed by the caller’s identity (that’s tomorrow’s lesson — auth is what makes “per user” possible):

TPM — tokens per minute, per user. The real throttle. Most gateways (LiteLLM and friends) let you set tpm_limit alongside rpm_limit. This is the one that bounds spend.
Per-user budget. The cumulative ceiling from Day 4 — $X/day per identity — sitting on top of the per-minute flow limit.
max_tokens on every call. Bound the output, the expensive half. This caps the damage of any single request.
Concurrency limit per user. An agent can fire many calls in parallel; cap the in-flight count so one identity can’t fan out into a hundred simultaneous requests.

The one real twist: you meter after the fact

A normal rate limiter knows the cost of a request up front. With tokens you often don’t know the exact cost until the call finishes — the output streams in and is counted as it goes. So it’s estimate-then-reconcile: pre-check the request against the limit using an estimate of the input tokens, let it run, then true up the real usage against the budget afterward.

The consequence is honest and worth stating: a single very large request can overshoot the limit before the meter catches it. That’s not a flaw you can fully remove — it’s why you don’t rely on the TPM limit alone.

The token bucket, keyed by identity

It’s the same token-bucket you’d write for requests — the only change is that the amount debited is a token count, not a fixed 1 per call:

import time

class TokenBucket:
    def __init__(self, tpm: int, burst: int | None = None):
        self.rate = tpm / 60.0          # tokens refilled per second
        self.capacity = burst or tpm    # burst ceiling
        self.state = {}                 # identity -> (available_tokens, last_ts)

    def allow(self, identity: str, est_tokens: int) -> bool:
        avail, last = self.state.get(identity, (self.capacity, time.monotonic()))
        now = time.monotonic()
        avail = min(self.capacity, avail + (now - last) * self.rate)  # refill for elapsed time
        if avail < est_tokens:                    # can't afford the estimate -> 429, don't call
            self.state[identity] = (avail, now)
            return False
        self.state[identity] = (avail - est_tokens, now)  # debit the ESTIMATE up front
        return True

    def reconcile(self, identity: str, est_tokens: int, actual_tokens: int) -> None:
        avail, last = self.state[identity]
        # true up once usage is known: refund the over-estimate, or debit the extra
        self.state[identity] = (avail + est_tokens - actual_tokens, last)

allow() refills lazily from the elapsed time (no background timer), then checks-and-debits your input estimate before the call — so a 200,000-token prompt is refused at the same limit that would wave through a thousand 200-token ones. After the response, reconcile() corrects the debit by the difference between estimate and actual, which is exactly where the balance can dip negative: that’s the honest overshoot, now bounded and visible rather than silent.

Belt and braces

Because of that overshoot, the token-aware controls work as a set, not individually:

the TPM limit bounds the ongoing flow,
max_tokens bounds any single call so one request can’t be arbitrarily huge,
the concurrency limit bounds the fan-out,
and the per-user budget is the cumulative backstop.

Together they turn “spend as fast as you can type” into a bounded, predictable envelope per user. Same token-bucket and quota algorithms you’ve always used — the only new idea is that the unit being bucketed is tokens, and it’s measured a beat late.

Next in the series — Day 13: AI endpoints are sensitive endpoints. Put auth on them. Everything here was “per user” — which only works if you actually know who the user is.

If your LLM route has a requests-per-minute limit and nothing on tokens, your rate limiting isn’t bounding your bill — and fixing that is part of what I do. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your setup and I’ll show you where one caller can outspend your limits — free, within a business day.