How the cache works

Jargon gets defined on first use, and there's a glossary at the end. If you haven't read the first document, the one thing you need from it is that the model processes text left to right and each position can only see what came before it.

Everything later depends on knowing the shape of what you send, so start there. A request to the API is a JSON document assembled from discrete pieces, each called a content block:

{
  "tools":  [ {tool definition}, {tool definition}, ... ],   ← blocks
  "system": [ {a chunk of instructions}, ... ],              ← blocks
  "messages": [
     {"role":"user",      "content":[ {text block} ]},
     {"role":"assistant", "content":[ {text block}, {tool request} ]},
     {"role":"user",      "content":[ {tool result} ]},
     ...
  ]
}

A block is one indivisible item: a single tool definition, one chunk of system instructions, one message's text, one tool result, one image. A long conversation with tool use is easily a hundred blocks. The count matters later, so it's worth having the picture.

The server flattens all of this into one long sequence of tokens, in a fixed order: tools first, then system, then messages. That flattened sequence is what the model actually sees.

Two more terms, both used constantly below:

• A prefix is any opening stretch of that sequence, counted from the very first token. "The first 30,000 tokens" is a prefix. "Tokens 500 through 900" is not a prefix — it doesn't start at the beginning. This distinction turns out to be the whole ballgame.
• TTL is time-to-live: how long a saved thing survives before being discarded.

Both names are in common use and they mean different things:

They rest on the same underlying fact, which is why the naming collides. The next two sections cover that fact; everything after is about the billed one.

Serving a request splits into two phases with quite different characteristics.

Prefill is processing your prompt. All its tokens go through all the layers together — large matrix multiplies, GPU saturated, limited by arithmetic throughput. Along the way, every token's key and value vector at every layer gets computed and held in memory. Prefill is what you wait through before the first output token appears.

Decode then produces output one token at a time. Each new token computes its own query, key and value, attends over all the stored keys and values from before, and appends its own. Only one token wide, so the multiplies are thin and the hardware spends most of its time moving memory rather than computing. Memory-bound rather than compute-bound — and that gap is roughly why output tokens are priced about five times input tokens.

The stored keys and values are the KV cache. Without it, generating token n would mean recomputing attention over all n−1 previous tokens from scratch: quadratic work instead of linear. It isn't optional, isn't configurable, and never appears on your bill.

Here is the load-bearing fact, and it follows directly from the causal mask in the first document:

So if two requests begin with byte-identical text, the processed state for that shared opening stretch is identical too — not similar, identical. It can be computed once, saved, and handed to the second request. The expensive prefill over that stretch never runs again.

That is prompt caching. Three consequences fall straight out, and between them they explain nearly every caching surprise:

• It only works on prefixes. Reuse runs from the very start of the prompt forward, up to the first difference. A cached paragraph sitting in the middle of an otherwise-different prompt buys you nothing at all, because the state at that paragraph depends on everything before it.
• One character of difference discards everything after it. Change token 40 of a 50,000-token prompt and tokens 41 onward all have to be recomputed. A timestamp near the top of a system prompt costs you the whole cache.
• Order is the entire design problem. Static content first, volatile content last. Since the flattening order is fixed as tools, then system, then messages, that's also the order in which changes cascade.

Saving doesn't happen automatically. You mark a content block with cache_control, and the API saves the state of everything from the start of the prompt up to and including that block. That marker is called a cache breakpoint:

system=[
    {"type": "text", "text": WHOLE_STYLE_GUIDE,
     "cache_control": {"type": "ephemeral"}},     ← the breakpoint
]

("ephemeral" is currently the only type; it just means the entry expires rather than persisting.)

What gets stored is keyed by a hash of the whole prefix ending at that block. On a later request the API hashes the prefix at your breakpoint and looks for a stored entry with the same hash. Same hash means byte-identical opening text, which means the saved state is valid.

Three rules govern the rest, and misreading rule 2 is the usual cause of "I turned on caching and nothing happened":

1. Saves happen only at your breakpoint. Marking a block stores exactly one entry. Nothing is stored at any earlier position.
2. Lookups search backward for entries that earlier requests saved. If the hash at your breakpoint doesn't match anything, the API steps back one block at a time, hashing the shorter prefix at each position, looking for a hit. It is hunting for previously saved entries — not for stable-looking content.
3. That backward search covers 20 blocks. If nothing matches within 20 positions, it gives up.

The mistake almost everyone makes

Say your prompt has a large static system context in blocks 1–5, and block 6 holds a timestamp plus the user's message. You put the breakpoint on block 6, since it's last:

request 1   hash(blocks 1..6) saved            ← the hash includes the timestamp
request 2   hash(blocks 1..6) is different → no match
            step back: block 5, 4, 3, 2, 1
            — but nothing was ever saved at those positions
            result: full-price prefill, plus another useless save

You pay the 25% save premium on every single request and never once get a read. The backward search can't rescue you: it only finds things earlier requests saved, and saves only happen at breakpoints.

The fix is one line — move cache_control to block 5, the last block that stays identical across requests. Then every subsequent request matches on the first try.

Using more than one

You get four breakpoints, and they cost nothing — you're billed on tokens saved and tokens read, not on markers. Two reasons to use several: caching sections that change at different rates (tool definitions rarely, reference documents daily), and long conversations, where each turn appends blocks until your breakpoint drifts more than 20 blocks past the last save and the backward search can no longer reach it. A second breakpoint further back gives it something to find.

Or let the API place it

For a plain growing conversation, put cache_control at the top level of the request and the breakpoint advances by itself as turns accumulate:

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},   # automatic placement
    system=SYSTEM_PROMPT,
    messages=history,
)

It puts the breakpoint on the last cacheable block — so it walks into exactly the trap above whenever your last block is the one that varies. Right default for a chat transcript; wrong for a prompt with a volatile tail.

Five minutes by default. Reading an entry resets that clock at no charge, so an active session stays warm indefinitely without ever paying a second save. The clock starts when the request that touches the entry begins, not when its response finishes — so a four-minute streaming response leaves only about a minute of usable life once it completes.

When the hour is worth buying

A 1-hour TTL is available as an explicit opt-in, and costs double base input on the save:

"cache_control": {"type": "ephemeral", "ttl": "1h"}

The break-even arithmetic is simple. A 5-minute save carries a 0.25× premium and each read saves 0.9×, so one read pays it back. A 1-hour save carries a 1.0× premium, so it needs two reads before it's ahead. If your calls come more often than every five minutes, the standard cache refreshes for free and the hour buys nothing. It earns its keep in the awkward zone — gaps longer than five minutes but shorter than an hour. A side-agent that runs for eight minutes; a user who steps away for coffee mid-conversation.

If you mix the two in one request, all 1-hour breakpoints must come before any 5-minute ones.

Three prices, all quoted per million tokens, all derived from the model's base input price:

• save (5-minute) — 1.25× base. Paid once, when the entry is created.
• save (1-hour) — 2× base.
• read — 0.1× base. Paid every time an entry is reused.

The response reports three separate input counts, and the trap is that the one called input_tokens is not your total. It counts only the tokens after your last breakpoint.

cache_read_input_tokens       tokens reused from a saved entry
cache_creation_input_tokens   tokens saved to a new entry on this call
input_tokens                  tokens after the breakpoint, cached neither way

total input  =  the three added together

"usage": {
  "cache_read_input_tokens":     29800,   // reused, 0.1× price
  "cache_creation_input_tokens":   240,   // saved now, 1.25× price
  "input_tokens":                   62,   // after breakpoint, full price
  "output_tokens":                 503
}

Minimum cacheable length

Below the threshold, cache_control is silently ignored. No error, no caching, no indication except those two zeroed counters.

If you're just short, padding the cached region up to the threshold is usually worth it — reads are cheap enough that crossing the line repays the extra tokens quickly.

Because the flattening order is tools, then system, then messages, a change at one level invalidates that level and everything after it. ✘ means the cache for that section is lost.

and the non-obvious ones

• Unstable JSON key ordering. Some languages randomise map iteration when serialising. If your tool-use blocks emit keys in a different order each call, the bytes differ and every request misses. Go and Swift are the usual culprits.
• Anything derived from the clock. A date, a request ID, a session token near the top of a system prompt. Move it after the breakpoint or drop it.
• Concurrency. An entry only becomes available once the first response has started. Fire ten parallel requests sharing a new prefix and you pay ten saves. Let the first land, then fan out.
• Isolation boundaries. Caches never cross organisations. On the Claude API they're also isolated per workspace; Bedrock and Google Cloud isolate only at the organisation level. Splitting traffic across workspaces splits your cache too.

If the first user request of the day shouldn't have to absorb the cold-prefill wait, you can pay for the save ahead of time using max_tokens: 0. The API reads the prompt, saves the entry, and returns immediately with an empty response and no output tokens billed.

client.messages.create(
    model="claude-opus-5",
    max_tokens=0,
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": "warmup"}],
)

Put the breakpoint on the shared opening, not on the placeholder message — otherwise the entry is keyed to the placeholder and your real traffic won't match it. This is also why automatic placement is wrong here. Match your thinking and effort settings to the real requests too, since those get written into the prompt. And the TTL still applies: re-warm inside the window, or buy the hour.

1. Order the prompt: tools, system, static reference material, then volatile content. Nothing time-varying above the breakpoint.
2. Put cache_control on the last block that's byte-identical across the requests meant to share a cache.
3. Check the cached region clears your model's minimum length.
4. Look at cache_read_input_tokens on the second request. Both counters zero means caching never engaged.
5. Hold tool_choice, image presence, thinking config and effort constant across a session.
6. Reach for the 1-hour TTL only when gaps genuinely exceed five minutes but stay under an hour.
7. Add a second breakpoint further back once a conversation grows more than 20 blocks past the last save.
8. Let the first request of a new prefix finish before fanning out in parallel.

Prices and thresholds verified against Anthropic's documentation on 2 September 2026. They change; the mechanism doesn't.