Stop renting the LLM by the task. Make it write the code.
Every practical AI-cost lever — caching, cheaper models, routing — is about spending less on the calls you make. There’s a lever above all of them, and a SaaS founder put it more sharply than I’ve managed to:
That reframes the whole thing. The cheapest token is the one you never spend, and this is how you never spend it.
Two ways to use a model for a repeatable job
Say you have a job to do ten thousand times — parse a file format, transform records, apply a validation rule.
- Make the LLM do it, each time. One metered API call per invocation. This is OPEX that scales with your usage — the uncapped meter itself. Every one of the 10,000 runs costs tokens, forever.
- Make the LLM write the code that does it, once. Now the token cost is a one-time capital expense, and the resulting function runs at ~zero marginal cost. The 10,000th run is free. The millionth is free.
Do the task, you rent it forever; write the code, you own it.
Most “our LLM bill is exploding” stories are option 1 applied to work that was always option 2 — deterministic, repeatable work pushed through a probabilistic, metered API because “the model can just handle it.” It can. You just pay rent on it every single time.
The hidden second tax: context cold-restarts
There’s a compounding cost the same thread flagged, and it makes option 1 worse than it looks. Complex, agentic tasks constantly refresh context — the agent re-reads its own earlier work to remember where it is. As tasks get longer, that overhead grows almost exponentially: you’re not just paying for the work, you’re paying, over and over, to re-load the state of the work. Doing repetitive things through the model stacks this tax on every run. A compiled function has no context to reload.
When to reach for which
This isn’t “never use the model.” It’s a line:
- Use the model to do the task when each instance genuinely needs fresh judgment: classify a novel input, summarize a unique document, handle fuzzy natural language that no rule captures. Here the per-call cost buys something a function can’t.
- Use the model to write the code when the task is deterministic and repeatable: parse this format, transform this shape, apply this rule, validate this constraint. Paying per invocation for something a regex or a function does for free is renting what you could own.
The trap, as the thread put it, is founders without the systems-engineering instinct “waving the magic LLM wand” at everything — turning work that should compile down to a cheap deterministic function into a recurring, metered, probabilistic API call.
Where this sits in the cost stack
Caps and quotas bound the bill. Caching and routing shrink the calls you make. This one deletes calls you never needed to make in the first place — which is why it belongs at the top of the list, before you tune anything downstream. Ask of every LLM call in a hot path: does this need a model, or does it need a function the model could have written once?
If your AI bill scales linearly with usage, some of that is deterministic work being rented by the call instead of compiled once — and finding it is part of what I do. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your AI setup and I’ll flag where you’re paying per call for something you could own — free, within a business day.