Why is an LLM so expensive for agents?

An agent re-sends its whole context on every step, so input dominates. See why.

What is the cost of an LLM call versus an agent task?

One task is many calls; measure per resolved task. See why and baseline.

How do I calculate LLM cost per task?

Tag calls with the task id, sum tokens by bucket, divide by tasks resolved. See baseline.

What is LLM token cost made of?

Uncached input, cache writes, cache reads and output, reasoning included. See baseline.

Why is my prompt cache hit rate low?

The cache stops at the first change, and a timestamp near the top changes every call. See cache.

Do cached tokens count against rate limits?

Not on Anthropic; on OpenAI they count toward tokens per minute. See cache.

Does trimming context hurt quality?

It can. Clear stale results, keep their path, gate on the resolved rate. See trim.

Does routing to a smaller model save money?

Per step, often; per task, not always. See route.

When should I use the batch API?

For work nobody waits for, at a lower rate. See batch.

How do I stop a runaway agent bill?

A gateway budget, an eval gate and an alarm with an owner. See hold.

A free playbook, as a PDF and an EPUB

The LLM cost playbook

Measure cost per resolved ticket, then cut it with caching, trimming, routing and batching, behind an eval gate.

Look insidePages from the PDF, as they print.

Page 2: Contents
Page 3: Acme Inc's triage agent re-sends its whole context on every step of a ticket.
Page 6: An agent reads far more than it writes.
Page 14: Write the baseline down in cost/baseline.py.
Page 21: Write the prompt order down in triage/prompt.py.
Page 40: Switching models mid conversation throws away the cache.
Page 65: The cost levers on one page, in the order to pull them.

A free playbook for engineers: measure LLM cost per resolved ticket, then cut it with prompt caching, trimming, routing, batching and an eval gate.

1The bill one ticket runs up, and the route down3
2Why an agent costs more than a chat6
3Measure cost and quality per resolved ticket before you change anything12
4Put the stable prefix first so every step reads it from cache19
5Stop re-sending what the loop no longer needs27
6Cap what the agent writes and how hard it thinks33
7Send easy steps to a small model and keep hard tickets on the large one38
8Move the work nobody waits for to the batch API47
9Hold the line with an eval gate and a budget alarm54
10Hand the numbers to the team that keeps them61
11The cost levers on one page65
12Questions engineers ask about LLM cost66

What the playbook covers

1
The bill one ticket runs up, and the route down
Acme Inc is a made up company. Its support triage agent, acme-triage, resolves tickets well and costs too much per ticket. This playbook follows one refund ticket through seven changes, each measured against a baseline.
2
Why an agent costs more than a chat
Most of the refund ticket's bill is input: the context re-read on every step and the tool definitions sent with each call. Hidden reasoning adds to the output, so even a short reply can bill long.
3
Measure cost and quality per resolved ticket before you change anything
The provider's invoice says what the month cost, not what the refund ticket cost. Group each ticket's calls from your traces, split the tokens into buckets, and put the number next to whether the ticket was resolved. Step one on the map.
4
Put the stable prefix first so every step reads it from cache
Acme's ticket text sits near the top of the prompt, with a timestamp in the system message, so every step of the refund ticket writes the cache and none reads it. Move the stable parts first and the ticket last. Step two on the map: the prompt.
5
Stop re-sending what the loop no longer needs
By the refund ticket's last step, old search results and full policy pages from lookup_policy ride along on every call. Clear old tool results, compact long runs and read passages by path, so a cleared result can be read again. Step three on the map: the context.
6
Cap what the agent writes and how hard it thinks
The refund ticket's classify step needs only a label and its draft needs a short reply, yet both run at full reasoning with a loose limit. Set effort per call and a hard cap. Step four on the map: the output.
7
Send easy steps to a small model and keep hard tickets on the large one
Classifying and extracting do not need the model that drafts a hard refund reply. Send them to a small model as separate calls, let the label pick the loop model, and keep each loop on one model so it keeps its cache. Step five on the map: the model per step.
8
Move the work nobody waits for to the batch API
Nightly re-tagging of closed tickets and the nightly eval runs go through the same real-time endpoint as live tickets. Nobody waits for them, so batch them, and leave the refund ticket and every live ticket on the real-time path. Step six on the map: nightly jobs.
9
Hold the line with an eval gate and a budget alarm
Every cut so far was checked by hand on the refund ticket and the baseline. Make it a gate: the eval set runs on every prompt or model change, and an alarm fires when cost per ticket or the cache hit rate drifts. Step seven on the map: the eval gate and alarm.
10
Hand the numbers to the team that keeps them
The baseline, the eval set, the prices file and the alarm each need a named owner, or the bill climbs back. docs/cost-owner.md names one for each, and says what to do when the alarm fires.
11
The cost levers on one page
Every lever acme-triage pulled on the refund ticket, in the order to pull them, with the file that holds each.

Keep it on your deskIt comes with a one-page cheatsheet to print and pin up.

Get the playbook

The LLM cost playbook

The playbook is open in a new tab

Build it live, with an engineer reviewing your choices

Look insidePages from the PDF, as they print.

What the playbook covers

The bill one ticket runs up, and the route down

Why an agent costs more than a chat

Measure cost and quality per resolved ticket before you change anything

Put the stable prefix first so every step reads it from cache

Stop re-sending what the loop no longer needs

Cap what the agent writes and how hard it thinks

Send easy steps to a small model and keep hard tickets on the large one

Move the work nobody waits for to the batch API

Hold the line with an eval gate and a budget alarm

Hand the numbers to the team that keeps them

The cost levers on one page

Keep it on your deskIt comes with a one-page cheatsheet to print and pin up.