When your bot promises a $1 truck: jailbreaks and output liability
We’ve covered attackers who run up your bill and attackers who hijack your bot. This one closes the abuse module with a different kind of damage entirely: not what it costs, but what it says — and what that commits you to.
Three that actually happened
- A Chevrolet dealership’s support bot was talked into “agreeing” to sell a 2024 Tahoe for $1, and — when prompted — stating it was “a legally binding offer, no takesies-backsies.” It went viral in late 2023.
- Air Canada was held liable by a tribunal in early 2024 for a bereavement-fare refund policy its chatbot invented. The airline argued the chatbot was, in effect, a separate entity responsible for its own words. The tribunal rejected that outright.
- DPD’s customer-service bot was prompted into swearing and calling its own company the worst delivery firm in the world — screenshots everywhere within hours.
The costly thing your bot says isn't a number — it's a promise.
The through-line, and courts are now saying it explicitly: you are responsible for your bot’s output. “The AI said it, not us” is not a defense. Every word your bot emits is an official statement from your company.
Two risks, one root
- Jailbreaks. Getting the model past its brand and safety constraints — to make a commitment, produce disallowed content, insult, or go off the rails. This is the Day 7 problem wearing a different hat: text that out-argues your rules. You can’t fully prevent it.
- Output liability (LLM09 misinformation + LLM06 excessive agency). The bot states a wrong policy, invents a price, gives dangerous advice, or takes an action — and it binds you, legally or reputationally. This doesn’t even require an attacker; a confident hallucination to an honest customer is enough, as Air Canada found.
The defenses (same shape as the whole series)
You can’t make the model perfectly well-behaved, so you bound what its output can do and commit you to:
- Ground answers in approved sources. For anything that matters — price, policy, legal, refunds — retrieve the authoritative answer and have the bot relay it; don’t let it improvise. Free generation is for chit-chat, not commitments.
- Never let the model make a binding commitment or take a consequential action on its own. No “yes, we’ll sell/refund/agree” from the model alone. Anything that binds you gets a deterministic check and a human (least agency, again — it’s the load-bearing control in every one of these lessons).
- Output guardrails on the way out. A moderation/rules layer that scans responses for disallowed content, promises, PII, competitor mentions, or price/legal claims before they reach the user. Imperfect, but it catches a lot.
- Scope limits, refusal, and escalation. For high-stakes topics the bot should refuse and hand off to a human, not improvise.
- Red-team your own bot. Try to jailbreak it into promising a $1 truck before a customer does — the same break-your-own-demo discipline from Day 7.
- Disclaimers help a little. A “general info, not a binding quote” notice is worth having, but Air Canada’s disclaimer didn’t save it once the bot gave specific wrong information. It’s a seatbelt, not a force field.
The pattern, one last time for the module
Every abuse lesson has landed on the same place: don’t rely on the model behaving. Whether the threat is a runaway loop, an injection, a freeloader, or a jailbreak, the durable defense is deterministic boundaries around what the model’s output can reach, spend, and commit you to. A jailbroken bot that can’t bind you to anything is an embarrassment; a jailbroken bot that can is a lawsuit.
Next in the series — the security base (Module 3): keys, token-aware rate limiting, and auth on AI endpoints — the fail-closed plumbing underneath all of this, and the part my background is built on.
If your bot can quote a price, state a policy, or take an action, it can be talked into the wrong one — and putting deterministic boundaries around what it can commit you to is exactly what I do. Every message comes straight to me — I read and reply to each one myself, usually within a day, and what readers send shapes what I build next. It’s just me for now, so that’s genuinely true; it won’t be forever. Send me your bot and I’ll show you what it can currently promise on your behalf — free, within a business day.