Prompt Caching for Production LLM Apps: Cutting Cost and Latency at the Token Layer

Prompt caching lets the provider reuse the processed state of a prompt prefix it has already seen, so repeated input tokens bill at about 0.1x the normal rate and time-to-first-token drops. To get it, order every prompt stable-first, dynamic-last, and set the cache breakpoint at the boundary. Nothing else about your app changes.

Our agent's system prompt runs to several thousand tokens: tool definitions, formatting rules, a few examples, policy text. It is stable across every request, yet without caching it is re-processed and re-billed on every call. At volume that becomes the largest line on the bill and a chunk of every request's latency, all spent re-reading text that never changed. The only real work in fixing it was structuring the prompt so the cache could do its job.

What prompt caching actually does

When you mark a prefix of your prompt as cacheable, the provider stores the model's internal representation (the processed KV state) of that prefix. On the next request with the identical prefix, it reuses that state instead of recomputing it. You're billed a steep discount for cached input tokens and you skip the compute for re-processing them, so both cost and time-to-first-token drop. As of October 2026 the numbers look like this:

The write premium is why sparse traffic can lose money: you pay 1.25x to store a prefix nobody reads back.

The critical mechanic: caching works on an exact-match prefix. The cache hits only for the contiguous span from the start of the prompt up to where it still matches byte-for-byte. The first token that differs ends the cacheable region.

The one rule: stable content first, dynamic content last

This is the whole discipline. Order your prompt so everything that's constant across requests comes first, and everything that varies comes last:

[ tool / function definitions ]  stable   (cacheable prefix)
[ system instructions          ]  stable
[ few-shot examples            ]  stable
[ policy / formatting rules    ]  stable   <- cache breakpoint here
-----------------------------------------
[ retrieved context            ]  varies
[ conversation history         ]  varies
[ user's current message       ]  varies

Put one dynamic token near the top (a timestamp, a request ID, the user's name interpolated into the system prompt) and you've poisoned the entire prefix. The cache matches up to that token and no further. We had a Current date: ... line at the top of the system prompt; moving it below the cache breakpoint took our hit rate from near zero to the large majority of requests.

Order matters inside the provider's own structure too. Anthropic's docs state the cached prefix covers tools, then system, then messages, in that order, so editing a tool definition invalidates everything after it.

Setting the cache breakpoint

With Anthropic's Messages API you tag the last stable block with cache_control. OpenAI caches eligible prefixes automatically and lets you place explicit breakpoints on newer models. The Anthropic version:

response = client.messages.create(
    model=MODEL,
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": POLICY + EXAMPLES,
        "cache_control": {"type": "ephemeral"},   # cache up to here
    }],
    messages=[{"role": "user", "content": dynamic_context + user_message}],
)
print(response.usage.cache_creation_input_tokens,
      response.usage.cache_read_input_tokens)

Everything from the start of the prompt through the marked block is cached; everything after is processed fresh each call. You want the breakpoint as far down as possible while staying before the first thing that varies. Anthropic allows up to 4 breakpoints, useful when tools change rarely but a document block changes daily, and also offers automatic caching via a single top-level cache_control that moves the breakpoint forward as a conversation grows.

TTL and hit rate

Caches expire. Anthropic's default is 5 minutes, refreshed at no cost on each hit, with a 1-hour option at a higher write price; OpenAI's GPT-5.6 and later default to a 30-minute minimum lifetime. The implication: caching pays off when requests sharing a prefix arrive close together in time. A high-traffic endpoint with a shared system prompt is the ideal case; the cache stays warm continuously. A low-traffic endpoint may see the cache expire between requests, so you pay the (small) cache-write premium without reaping the read discount.

Check the hit rate the provider reports (cache_read_input_tokens against cache_creation_input_tokens on Anthropic, cached_tokens on OpenAI). If both Anthropic fields are 0, the prefix was probably below the minimum length, and the docs note no error is returned. If you're mostly writing and rarely reading, either your prefix isn't stable (find the poisoning token) or your traffic is too sparse. For sparse-but-regular traffic, the longer TTL can flip the math.

Conversation history: cache the growing prefix

For multi-turn agents, the conversation history is append-only: each turn adds to the end, and the beginning stays identical. That's a perfect caching shape. Place the cache breakpoint at the end of the prior turn so each new request re-uses the entire conversation so far as a cached prefix and only processes the new turn. Long agent loops benefit the most, because the cached prefix grows while the per-turn fresh tokens stay small. If you are trimming history to fit a window, do it in large, infrequent steps; every trim rewrites the prefix (our notes on token budgeting for long-context prompts cover the trade-off).

What it bought us

• Cost: on cache hits, the stable prefix bills at the read rate (0.1x on most models) instead of full input price; on a system-prompt-heavy workload that is a large fraction of the total bill.
• Latency: time-to-first-token improved noticeably because the model skips re-processing the cached prefix. The bigger the stable prefix, the bigger the win.
• Behavior: no change to what the model is conditioned on. The prompt is identical, so caching is a cost and latency optimization, not a quality trade.

Prompt caching is distinct from semantic caching, which skips the model call entirely for similar questions. They stack, and both belong in a broader LLM cost optimization plan.

Questions people ask

What is prompt caching in LLM APIs?

Prompt caching stores the model's processed state for a prompt prefix so that a later request starting with the exact same tokens can reuse it. Cached tokens bill at a fraction of the input price (0.1x on most Anthropic and OpenAI models as of October 2026) and are not recomputed, which also cuts time-to-first-token. Any difference, even one character, ends the reusable prefix at that point.

How long does the prompt cache last?

On Anthropic's API the default lifetime is 5 minutes, refreshed for free every time the prefix is read, with an optional 1-hour TTL whose writes cost 2x base input. OpenAI's GPT-5.6 and later default to a 30-minute minimum, with a 12-hour option at 2.5x. Lifetimes are minimums, so steady traffic keeps a prefix warm indefinitely.

Why is my prompt cache hit rate so low?

Usually something dynamic sits above the breakpoint: a date, request ID, user name, or reordered tool list. Diff two consecutive prompts and look for the first changed token. Other causes are a prefix under the provider's minimum (512 tokens on current Claude models, 1,024 on recent OpenAI models), or traffic so sparse that entries expire between requests.

The discipline, restated

Prompt caching is almost free to adopt and almost free to ruin. Structure every prompt as [stable] then [dynamic], set the cache breakpoint at the boundary, keep dynamic tokens (dates, IDs, names) strictly below it, and watch the cache-read ratio. Done right, you stop paying full price to re-read the same instructions thousands of times a day for nothing.

Sources

• Anthropic docs: Prompt caching
• OpenAI API docs: Prompt caching