How I'd audit your AI feature in an afternoon
Fifteen lessons built the playbook. This is how I actually run it — against a real, shipped AI feature, in an afternoon. You don’t need a two-week engagement to know whether yours is a surprise bill or a liability waiting to happen. You need honest answers to seven questions, asked in order of how much they can hurt you. Here’s the audit.
The seven questions, most-dangerous-first
1. Can I see the meter? Is every LLM call logged with tokens-in, tokens-out, dollar cost, and who made it? Do you know your list-price cost with free credits stripped out? 🚩 Red flag: “we check the provider dashboard once a month.” You can’t cap a meter you can’t see, and a $0 credit hides your real burn.
2. Is there a ceiling? A global hard cap that stops serving at $X, plus per-user budgets so one key can’t drain the whole thing. 🚩 Red flag: “we have billing alerts.” Alerts email you; they don’t stop spend.
3. Is it on the request path? Does every call go through one gateway where the caps and limits actually live — or are there LLM SDK calls scattered through the app? For agents: is there a per-run breaker? 🚩 Red flag: direct SDK calls everywhere, and “we’d catch it on the next bill.” The kill switch belongs on the request path, not the invoice.
4. Where’s the key? Server-side only, scoped to the API/models you use, rotated — and not in client JS, a mobile app, a repo, or git history. 🚩 Red flag: a key anywhere the user can read it. That’s the $55k mistake.
5. Who’s calling? Real, revocable auth on the endpoint — an identity behind every call, not anonymous traffic. 🚩 Red flag: an open endpoint. If anyone can reach it, it’s someone’s free ChatGPT, and per-user caps are meaningless without a user to attach them to.
Most features fail three or four of these.
6. What can the model do? Least agency: can model output trigger a refund, a DB write, a tool call, an email — without a deterministic check in your code? Does the model read untrusted content (RAG, the open web) that could carry an injection? 🚩 Red flag: an agent with tool access and no gate. That’s how a jailbreak or a poisoned document becomes a $1 truck.
7. What happens when it trips? When a cap is hit or abuse is detected, does it fail closed — stop or degrade — or run unbounded? 🚩 Red flag: no defined behavior, which is fail-open. Fail-open is what you inherit; fail-closed is what you build.
What comes out
Not a forty-page report. A one-page map: each of the seven green or red, the two or three that would bite first, and the cheapest fix for each. An owner’s-eye view of where the money and the liability can escape — the kind of thing you can act on the same week.
Most features I look at fail three or four of these. And here’s the honest part, the same one the bridge lesson makes: it’s almost never because the team is bad. It’s because capping and guarding the meter was nobody’s job — so it lived on the backlog until it didn’t. The audit’s real value is making it someone’s job for an afternoon.
Do the first pass yourself
You don’t need me for the first pass — the free one-page Cap & Guard checklist is this audit as a self-serve list; run your feature down it and count the red flags. If you’d rather I did the pass and handed you the map, that’s the paid version of exactly this.
And if you want to see what “on the request path, fail closed” (question 3) actually looks like before you send me anything: I built a working gateway that stops a runaway agent live — watch it, code here. The audit tells you which of the seven you’re failing; that’s the shape of the fix.
Next in the series — Day 18: a teardown of a real public AI incident against these same seven questions, showing which red flag actually did the damage.
If you’ve shipped an AI feature fast and you’re not sure how many of those seven it fails, that’s the afternoon I’m offering. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your setup and I’ll run the seven questions against it — free, within a business day.