Three numbers that explain everything

Output tokens cost roughly five times what input tokens cost. Cached context costs about a tenth. And between the cheapest and the strongest model class there is — depending on the provider — a factor of five to twenty-five.

These three ratios explain almost every surprising monthly bill. They also explain why so many optimisation attempts come to nothing: they start at the wrong end.

Load profile first, then the lever

Applications fall into two worlds. Context-heavy applications — retrieval, document analysis, long system prompts — burn money on the input side, because the sheer volume more than offsets the lower unit price. Response-heavy applications — chat, generation, drafting — burn it on the output side, precisely because output is so expensive per token.

The diagnosis takes ten seconds. Rule of thumb: output dominates when output tokens amount to more than a fifth of input tokens.

On the input side: send less, not more

The biggest lever is prompt caching. Anything that does not change between calls — policies, examples, schemas — need not be paid for again on every call. The one rule that matters: static before dynamic. The cache only takes hold on the unchanged beginning of the prompt. A timestamp or a user name in the wrong place invalidates it on every single call.

The second lever acts further upstream: precision in retrieval. The reflex when answers are weak is to add more context. That is wrong twice over — more expensive and worse, because noise dilutes the answer. Searching broadly and then narrowing to a few hits through reranking reverses both: twenty chunks of 800 tokens become four, 16,000 tokens become 3,200 — with a rising hit rate.

For very large documents, a two-stage approach pays off: a cheap model condenses the material against the question, and the strong model then decides on the short versions alone. 30,000 raw tokens become a good 4,000.

Two quiet cost drivers are easily overlooked here. First, the conversation history: without a cap, turn ten pays for turns one to nine all over again — every time. A sliding window plus a one-off summary keeps the input constant instead of growing linearly. Second, the tool definitions: schemas count as input on every call, even when no tool is used at all. Twenty tools at 500 tokens each is 10,000 tokens of base load per request — and again for every agent step.

On the output side: shorter is usually better

Here the most direct intervention is also the simplest: set a hard upper limit and specify a length in the prompt. That is a cost brake, not a loss of quality, because most answers are better short.

More effective still is the change of format. Where only a decision or a handful of fields are needed, there is no need for prose. Structured output instead of running text often saves sixty to ninety per cent of the output. And where a classification is enough, a code will do: two output tokens instead of a whole sentence — the plain text lives in your own code.

Two further details carry leverage. Reasoning tokens count as output, so they belong dosed rather than switched on across the board. And in interactive interfaces users frequently abandon a response — without cancellation logic the API keeps writing, and you pay for text nobody reads.

The biggest single lever sits one level up

No micro-optimisation beats model selection. The cheapest sufficient model as the default, escalation only where needed or where confidence is low — that is the factor of five to twenty-five from the introduction.

The important word is "sufficient". Right-sizing without proof is guesswork. A golden set of a few dozen reference cases that measures quality and cost together turns an assumption into a defensible decision.

Alongside that, three pragmatic additions are worth having: a response cache for recurring requests (the cheapest call is the one that never happens), a free preflight token count as a guard against accidentally enormous prompts, and batch processing for everything that is not needed in real time — typically around fifty per cent cheaper.

Cost is a test criterion, not a reporting topic

The step that turns isolated improvements into a stable cost level happens in the pipeline. Prompt changes are cost changes, and they can be handled exactly like performance regressions: a token budget as an assertion in the pull request, a cost gate that makes the diff visible and blocks when the increase is too steep, and, after deployment, a daily drift check against the baseline.

The effect is unspectacular, and that is precisely what makes it valuable: a forgotten retry loop surfaces the next day, not with the monthly bill.

What it all adds up to

A worked example from a triage use case with 120,000 requests per month. Built naively — strong model for everything, no caching, free-text answers — around 2,200 dollars a month. With routing to a cheaper model, caching on the system prompt, codes instead of prose and a budget guard: around 440 dollars. Eighty per cent less, at a demonstrably identical hit rate.

Three things to take away:

• Measure first, optimise second. Without the input/output ratio, every lever is a guess.

• Match the lever to the profile. Caching for input load, length and format control for output load.

• The cheapest sufficient model — with golden-set proof rather than gut feeling.

Getting started costs nothing: log input, output and cache tokens per call for one week. After that, the right lever is usually obvious.