Never call an LLM directly: the gateway pattern

July 8, 2026

Day 1 was what a single call costs. Day 2 was the five ways it goes unbounded — and the thread through all of them was that nothing stops itself. So the fix needs one thing before anything else: a place to put the stop. Today is that place, and it’s the foundation everything else in this series sits on.

You already know this pattern

You would never let every service in your system open its own connection straight to the database, each with its own copy of the password, each free to run whatever query it likes. You put it behind something — a connection pool, an API, a proxy — so there’s one controlled door. You’d never expose the database directly to the browser, either.

Somehow, with LLMs, that’s exactly what most apps do: import the OpenAI or Anthropic SDK, drop the key in, and call the provider directly, from wherever in the codebase happens to need it. Every one of those call sites is an ungoverned door — no cap, no meter, no cache, no guard, and often the raw key sitting right there.

The fix is the oldest pattern in infrastructure: put a chokepoint in front of it. If you’ve run nginx, an API gateway, or an outbound egress proxy, you already have the whole mental model. This is that — for LLM calls.

The gateway

A gateway is a single proxy that every LLM call in your system passes through on its way to the provider. Your app talks to the gateway; the gateway talks to OpenAI / Anthropic / Google. That’s it.

The reason it matters is simple: you can only enforce policy where the calls converge. Scattered direct SDK calls can’t be capped or measured as a whole — there’s no “as a whole.” Route them through one door and suddenly there is a place to stand. Everything else in this series lives at that door:

None of those are possible while your calls go out a dozen different doors. All of them are straightforward once they go out one.

The tools (the chokepoint matters more than which one)

Pick by how much you want to run yourself. The pattern is identical regardless.

It’s usually a one-line change

This is the part that makes it an easy sell to a busy team: adopting a gateway rarely means rewriting the feature. Most SDKs let you override the endpoint, so you point base_url at the gateway and leave the rest alone:

# before — your app calls the provider directly; no chokepoint, key in the app
client = OpenAI(api_key=KEY)

# after — same code, pointed at your gateway; now every call passes one governed door
client = OpenAI(api_key=KEY, base_url="https://your-gateway/openai")

One line to route through the door. Everything from Day 4 onward is configuration at that door, not surgery on your app.

Why this is Day 3 and not Day 14

Because the caps and guards everyone actually wants have nowhere to live until the chokepoint exists. Teams skip straight to “add a spend limit” and find there’s no single place to add it — so it never gets added. Build the door first. Then the lock, the meter, and the alarm all have somewhere to hang.

Tomorrow (Day 4): the first thing we bolt to the door — a hard spend cap and per-user quotas, so the fail-open default from Day 2 finally fails closed.

Following along? Join the list to get each lesson in your inbox. And if you’re already shipping an AI feature and want a read on where its ungoverned doors are, send it to me — free, within a business day.

Learning to cap AI cost & abuse yourself?

This series is becoming a hands-on course — the gateway, the spend caps, the abuse guardrails, built step by step. Join the waitlist: you'll get each lesson as it lands, and first access when it opens.

Double opt-in — one email to confirm. The lessons are free; the course is optional. No spam, unsubscribe anytime.