Cruxly › AI engineering › LLM Backend Infrastructure

Prompt Caching: Paying for the Same Tokens Once

prompt layout is the cache key, and one timestamp can cost you all of it

Prompt: a support assistant sends every request with the same 20,000-token prefix — a long system prompt, tool definitions, and a product manual — followed by a few hundred tokens of the user's actual question. The bill is dominated by input tokens the model has already seen thousands of times. Cut it, and say exactly what could silently undo the saving.

30 minutes first. The naive answer is "turn on caching." The real answer is that caching only works if the prompt is laid out for it, and most production prompts are not.

What is actually being cached

• When a model reads a prompt, it computes internal state (the KV cache — see Inference Serving) for every token. Prompt caching lets the provider keep that state for a prefix and reuse it on the next request that starts with the same prefix.
• Two things get cheaper at once: cost (cached input tokens are billed at a fraction of the normal rate) and latency (the cached part skips prefill, so time-to-first-token drops on long prompts).
• The match is on the exact prefix. A single changed token at position 5,000 means everything after it is recomputed and billed in full — even if the remaining 15,000 tokens are identical.
• Matches are coarse, not per-token. Providers only cache prefixes above a minimum length, and only at certain boundaries (content-block or breakpoint boundaries on Anthropic; fixed-size steps on OpenAI). So a change near the very start leaves nothing reusable — the identical part before it is too short to count.

How the major providers expose it (as documented in 2026 — check current docs, the multipliers change):

Last verified 2026-10-31 against each provider's docs: Anthropic reads cost 0.05× on Claude Opus 5.5 and Sonnet 5.5 and 0.025× on Fable 5.1 (0.1× elsewhere); a single top-level cache_control field turns on automatic caching, which moves the breakpoint forward as the conversation grows. OpenAI documents, for GPT-5.6 and later (its examples use GPT-6.1 Sol), an explicit mode (prompt_cache_options.mode) that caches only developer-placed breakpoints, and a 12h retention option written at 2.5× the uncached input rate; its reads are 0.1× on most models and 0.05× on GPT-6.1 Sol and Terra, writes are 1.25× for the default 30-minute lifetime, and the minimum cacheable prompt is 1,024 tokens. Multipliers change, so re-check before quoting.

:::key The one fact that drives the whole design A cache hit needs the prompt to be byte-for-byte identical up to that point. Every other decision in this lesson follows from that. :::

Prompt layout is the cache key

Order the prompt from most stable to least stable. On Anthropic the cache prefix is built in a fixed order — tools, then system, then messages — and a change at one level invalidates that level and everything after it. Changing a tool definition invalidates the whole cache.

system:   "Today is 2026-10-06 14:32. You are helping {user_name}..."   ← changes every request
          + 15,000-token product manual
tools:    [list built from a dict — order differs between deploys]

tools:    fixed list, fixed order, defined once
system:   instructions + 15,000-token manual   ← identical for every user, every day
messages: "Context: today is 2026-10-06, user is {user_name}." + the question

The silent cache-killers that show up in real code:

• A timestamp or request ID in the system prompt. The most common one. Move it to the end.
• Per-user data before shared data. Personalisation belongs after the shared manual, not before it.
• Non-deterministic serialisation. Tool lists built from an unordered collection, or JSON with unstable key order, produce a different prefix on every deploy even though "nothing changed."
• Changing settings that are part of the prefix. Switching tool choice, adding an image, or changing tool definitions mid-conversation can invalidate cached sections, depending on the provider.
• A/B testing two system prompts. Each variant needs its own warm cache — twice the cache writes, and on low traffic each variant gets fewer hits and expires more often between them.

The arithmetic, and choosing the TTL

Caching costs a little extra on the first request (the write) and saves a lot on every later one (the reads). Work out when it pays off before claiming it does. Using Anthropic's multipliers, for a prefix of size P reused n times:

@@ 1 | 5-minute cache: write 1.25×, read 0.1× Cost with caching = 1.25P + 0.1P × (n − 1). Without = nP. It wins once n > 1.3 — from the second request onward.

@@ 2 | 1-hour cache: write 2×, read 0.1× Cost with caching = 2P + 0.1P × (n − 1). Wins once n > 2.1 — from the third request, but it survives quiet periods.

@@ 3 | The support assistant in the prompt 20,000-token prefix, steady traffic. Without caching every request pays for 20,000 input tokens. With a warm cache, each request pays about 20,000 × 0.1 = 2,000 token-equivalents for the prefix plus its own few hundred — roughly an 85–90% cut in input cost, and a shorter time-to-first-token.

Choosing the lifetime is a traffic question, not a preference:

• Requests arrive every few seconds or minutes → the short cache is refreshed by every hit and never expires. Cheaper writes, so use it.
• Requests are bursty with gaps longer than the short lifetime (internal tools, overnight batch, low-traffic tenants) → the short cache keeps expiring and you pay the write again and again. The longer cache costs more to write but stops that churn.

:::warn A cache that is written but never read is pure loss Marking a prefix that is only ever used once — a one-off document, a unique per-user prompt — costs 1.25–2× its normal price for no benefit. Cache what is reused, not everything. :::

Measuring it, and the alert that catches a regression

Never assume caching works — read it from the response. Every response reports how many input tokens were read from cache, how many were written, and how many were processed normally. If a prefix is shorter than the minimum, providers generally just don't cache it, with no error — the usage fields are the only signal.

• Track cache hit rate = cached input tokens ÷ all input tokens, per endpoint. On Anthropic that's cache_read ÷ (cache_read + cache_creation + input_tokens) — careful, input_tokens counts only the uncached tail, so dividing by it alone gives nonsense. For the support assistant (20,000 cached of ~20,300) it should sit near 98%. Don't confuse this with the cost cut (~88%): they are different numbers.
• Alert on a drop, not on a level. The classic incident is a deploy that adds a timestamp or reorders the tools: nothing breaks, latency rises a little, and the input bill quietly multiplies several times over. A sudden fall in hit rate right after a deploy is the signature.
• Watch request routing. A cache lives on the provider's infrastructure. On older OpenAI models, prompt_cache_key routes related requests to the same cache machine; newer models route automatically.

Three more traps that show up under real traffic:

• Parallel requests on a cold cache all pay the write. Anthropic's docs are explicit: a cache entry only becomes available after the first response begins. Fan out 50 requests at once and all 50 write. Send one first, then the rest.
• Multi-turn chats need the breakpoint to move. Each turn should cache up to the latest message so the next turn reads it. Automatic caching does this for you; with manual breakpoints, remember the limits — at most 4 breakpoints, and a lookback of 20 blocks, so a breakpoint placed too far past the last cached point misses.
• Output is never cached. Caching only discounts input. On a workload that generates long answers from short prompts, it barely moves the bill.

Google's Gemini works differently again: it has automatic (implicit) caching, plus explicit context caches you create and pay to store over time — a different cost curve from the write-surcharge model above.

Prompt caching versus the other caches in this track:

The first two are the same idea, one bought and one built. Semantic caching is a different trade: it skips the model entirely, which saves more but can be wrong.

How to say it in the interview

"The provider can reuse work for a prompt prefix it has seen before, but only an exact prefix match counts. So the design question is prompt layout: everything that never changes goes first, everything that changes per request goes last, and one dynamic value near the top — a timestamp, a user name — turns every request into a cache miss."

Checklist

• Explained that prompt caching reuses model state for an exact prefix, cutting both cost and time-to-first-token
• Laid the prompt out from most stable to least stable: tools, then system and shared documents, then per-request data
• Named the silent cache-killers: timestamps or IDs early in the prompt, per-user data before shared data, unstable serialisation, A/B prompt splits
• Did the break-even arithmetic instead of assuming caching always helps
• Chose the cache lifetime from the traffic pattern
• Measured hit rate from the response usage fields (over all input tokens) and alerted on drops after deploys
• Warmed the cache with one request before fanning out parallel requests
• Noted that caching discounts input only
• Distinguished prompt caching from semantic caching

Related lessons

• Multi-Provider Failover That Actually Fails Over
• Cost Attribution and Runaway Prevention