Caching Is the Cheapest Win
An agent is a loop, and a loop repeats itself. Every turn resends the system prompt, the tool definitions, the instructions, the examples, and most of the conversation so far. On a long run the same opening block of text is processed hundreds of times.
You pay for it every single time, unless you do one thing.
Prompt caching reuses the computed state of a prefix that has not changed. That is all it is, and on an agent workload it is routinely the largest cost reduction available, with no loss of quality, no change of model, and no clever prompting. It is the first thing to check when a bill looks wrong and usually the last thing anyone looks at.
In short: Put everything stable at the front and keep it byte-identical, then a cache hit costs a fraction of a fresh read. One varying token near the start, a timestamp or a shuffled tool list, silently destroys the whole thing.
Caching a Prefix, Not an Answer
The word cache makes people picture a lookup table of questions and answers. That is not this, and if it were it would be nearly useless, because the same question rarely arrives twice.
Recall from How a Language Model Works that a call has two phases. The model reads your whole input first, then generates the reply one token at a time. Reading is the part being cached. If the first several thousand tokens of this call are identical to the last one, that reading work does not have to happen again. The state is already computed. The model picks up where the new part begins and generates a completely fresh answer.
So the reply is never reused. Only the reading of the unchanged opening. That is why this is safe in a way an answer cache is not: nothing about the output changes, you simply stop paying twice to read the same instructions.
The Whole Trick Is a Stable Prefix
Everything practical follows from one property. A cache matches on an exact prefix. It works from the first token forward, and the moment something differs, everything after that point is uncached.
Which gives you the single rule of prompt design for cost:
Stable at the front, variable at the back.
| Front, cached | Back, fresh every call |
|---|---|
| System prompt and standing rules | The user's newest message |
| Tool definitions and schemas | Retrieved chunks for this question |
| Few-shot examples | The current timestamp, if you truly need one |
| Long reference documents | Anything that changes per turn |
Most prompts are assembled in whatever order the code grew, which means the variable parts are scattered through the stable ones. Reordering so that everything fixed sits together at the top is often a small refactor with a large invoice attached.
Four Ways to Destroy Your Own Hit Rate
This is where the money actually goes, and every one of these looks harmless in review.
A timestamp in the system prompt. "The current date is 2026-07-29 13:04:11." One line, near the front, different on every call, and now nothing after it can be cached. If the model needs the date, put it at the back, and use the day rather than the second.
A session or request id at the top. Same failure. Identifiers belong in your logs, not in the model's context.
A tool list that reorders itself. If tools come from a dictionary or a set, the order can change between runs. The text differs, so the prefix differs. Sort them, and keep the order fixed.
Editing the system prompt constantly. Every edit resets the cache for everyone. Do it deliberately, in batches, not as a running experiment against production traffic.
The tell for all four is a cache hit rate near zero while everyone is certain caching is enabled. Log the hit rate. If you are not measuring it, you do not have it.
This Is Not the Serving Engine's Cache
Shipping Agents covers prefix caching as something an inference engine does. The two are related and the difference is who is responsible.
A serving engine reuses computation across the requests it happens to receive. It is infrastructure, it optimises what arrives, and it works best when the traffic already has shared prefixes.
Prompt caching is the application side, and it is a contract you shape. You decide what sits at the front of every call, you keep it identical, and you mark what should be held. The engine cannot create a stable prefix for you. If your prompts vary at the top, both caches miss, and no amount of infrastructure fixes a prompt that is different every time.
That is the whole reason this is a chapter of its own: it is not something you turn on, it is something you design for.
Look at What You Resend Before You Change Anything Else
Go back to the bill. When an agent costs too much, the instinct is to reach for a smaller model, and that trade costs quality. The instinct after that is to tune infrastructure, and that costs a week.
Look at the repetition first. Count how much of each call is text you already sent. On a tool-heavy agent with a long system prompt, it is usually most of it, and making that block stable is an afternoon that changes the shape of the invoice without touching the output at all.
Cheapest win in the stack, and it is sitting in the first two thousand tokens of every call you make.
Next, the other way to change what a call costs: stop making it to someone else's machine.