Cut your LLM bill without touching the feature: caching + model routing
Day 4 put a ceiling on the bill so a runaway can’t blow past it. That’s the safety half. This is the thrift half: the cheapest way to stay under a cap is to not spend the money in the first place — and you can usually cut a real chunk of an LLM bill without changing the feature at all. Four levers, and the reason the gateway came first is that all four live there, as configuration, instead of scattered through your app.
1. Response caching — don’t pay twice for the same answer
Most apps ask the model the same things over and over.
Four levers at the gateway, and the feature never changes.
- Exact-match cache. Hash the full request (prompt + params) → store the response. The next identical request is served for free. This alone kills a surprising amount: the same FAQ, the same button, the same regenerated summary.
- Semantic cache. Embed the query and match near-duplicates by similarity — so “how do I reset my password” and “password reset help” share one answer. Powerful, but it has teeth: a false match serves a subtly wrong answer. Use a conservative similarity threshold, and only for stable, factual responses — never for anything personalized.
2. Provider-side prompt caching — stop re-paying for your system prompt
If you send a 2,000-token system prompt or a big RAG context on every call, you’re paying for that same prefix thousands of times. Both major providers now let you cache it: Anthropic prompt caching and OpenAI’s cached-input pricing bill the repeated prefix at a fraction of the normal input rate. Input tokens are the cheap half (Day 1), but at volume the repeated prefix is most of your input cost — and this is close to free money, because the output is unchanged.
3. Model routing — stop sending easy calls to the flagship
Not every request needs your most expensive model. Default to a small/cheap model and escalate only when the task warrants it — routed by task type, by a cheap classifier, or by confidence (retry on the big model when the small one is unsure). Most traffic is easy, and a small model often runs at a tenth to a twentieth of the flagship’s price. This is usually the single biggest lever, and the one teams skip because “use the best model” feels safer than it is.
4. Batching — for anything that isn’t real-time
Embeddings, bulk classification, offline enrichment, nightly jobs — none of it needs a synchronous answer. Provider batch APIs are frequently ~50% cheaper for exactly this. If a human isn’t waiting on the response, batch it.
The discipline: you can’t claim a number you didn’t measure
“Cut ~40%” is only honest if you measured it, and your real number depends entirely on your repeat rate and how much easy traffic you’re overpaying for. So instrument it at the gateway (you already log tokens and dollars from Day 3): add cache-hit rate and per-route model mix, then compare before and after. No dashboard-as-vanity — a before/after on the actual bill.
The caveats nobody puts on the slide
This is where caching quietly breaks things:
- Cache invalidation is the hard part. Answers go stale when the underlying data changes. Cache facts, not fast-moving state.
- Never serve one user’s cached answer to another. Scope cache keys by user/tenant for anything user-specific — a shared cache is a data-leak (LLM02, coming up) waiting to happen.
- Routing down can degrade quality. Measure the quality, not just the cost — escalate on low confidence, and A/B the cheap model before you trust it on the hot path.
And the lever above all four
The cheapest token is the one you never spend. Before you tune any of this, ask whether the call needs to exist at all — sometimes the right move is to have the model write the code once instead of renting it by the task. Caps set the ceiling; caching and routing lower the floor; not-calling deletes the floor.
Next in the series — Day 6: the OWASP LLM Top 10 in plain English — because we’ve now covered the cost meter, and it’s time for the failure mode cloud never had: abuse.
If your LLM bill scales straight up with usage, some of that is repeat calls and easy traffic paying flagship prices — and finding it is part of what I do. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your AI setup and I’ll point at the cheapest wins first — free, within a business day.