# Prompt Caching for Production LLM Apps: Cutting Cost and Latency at the Token Layer

URL: https://www.devopsness.com/blog/prompt-caching-production-llm-apps
Author: Kiril Urbonas
Published: 2026-06-27
Updated: 2026-10-31
Categories: AI
Tags: LLM, GPT, Anthropic, Cost Optimization, Production

Summary: A long, stable system prompt re-billed on every request is money on fire. How prompt caching works, where the cache boundary belongs, and the structuring discipline that got us a big cost and latency cut without changing behavior.

---
# Prompt Caching for Production LLM Apps: Cutting Cost and Latency at the Token Layer

Prompt caching lets the provider reuse the processed state of a prompt prefix it has already seen, so repeated input tokens bill at about 0.1x the normal rate and time-to-first-token drops. To get it, order every prompt stable-first, dynamic-last, and set the cache breakpoint at the boundary. Nothing else about your app changes.

> [!VERIFIED]
> Anthropic Messages API prompt caching and OpenAI prompt caching for GPT-5.6 and later, against the platform.claude.com and developers.openai.com prompt-caching guides. Checked October 2026.

Our agent's system prompt runs to several thousand tokens: tool definitions, formatting rules, a few examples, policy text. It is stable across every request, yet without caching it is re-processed and re-billed on every call. At volume that becomes the largest line on the bill and a chunk of every request's latency, all spent re-reading text that never changed. The only real work in fixing it was structuring the prompt so the cache could do its job.

## What prompt caching actually does

When you mark a prefix of your prompt as cacheable, the provider stores the model's internal representation (the processed KV state) of that prefix. On the next request with the *identical* prefix, it reuses that state instead of recomputing it. You're billed a steep discount for cached input tokens and you skip the compute for re-processing them, so both **cost and time-to-first-token** drop. As of October 2026 the numbers look like this:

| Provider | Cache read | Cache write | Default lifetime | Longer option | Minimum prefix |
|---|---|---|---|---|---|
| Anthropic (Claude API) | 0.1x base input (0.05x on Opus 5.5 and Sonnet 5.5, 0.025x on Fable 5.1 and Mythos 5.1) | 1.25x base input | 5 minutes, refreshed free on each hit | 1 hour at 2x base input | 512 tokens on current models, more on some older ones |
| OpenAI (GPT-5.6 and later) | 0.1x uncached input (0.05x on GPT-6.1 Sol and Terra) | 1.25x uncached input | 30 minutes minimum | 12 hours at 2.5x | 1,024 visible input tokens |

The write premium is why sparse traffic can lose money: you pay 1.25x to store a prefix nobody reads back.

The critical mechanic: caching works on an **exact-match prefix**. The cache hits only for the contiguous span from the start of the prompt up to where it still matches byte-for-byte. The first token that differs ends the cacheable region.

## The one rule: stable content first, dynamic content last

This is the whole discipline. Order your prompt so everything that's constant across requests comes first, and everything that varies comes last:

```
[ tool / function definitions ]  stable   (cacheable prefix)
[ system instructions          ]  stable
[ few-shot examples            ]  stable
[ policy / formatting rules    ]  stable   <- cache breakpoint here
-----------------------------------------
[ retrieved context            ]  varies
[ conversation history         ]  varies
[ user's current message       ]  varies
```

Put one dynamic token near the top (a timestamp, a request ID, the user's name interpolated into the system prompt) and you've poisoned the entire prefix. The cache matches up to that token and no further. We had a `Current date: ...` line at the top of the system prompt; moving it below the cache breakpoint took our hit rate from near zero to the large majority of requests.

Order matters inside the provider's own structure too. Anthropic's docs state the cached prefix covers `tools`, then `system`, then `messages`, in that order, so editing a tool definition invalidates everything after it.

## Setting the cache breakpoint

With Anthropic's Messages API you tag the last stable block with `cache_control`. OpenAI caches eligible prefixes automatically and lets you place explicit breakpoints on newer models. The Anthropic version:

```python
# app/llm.py
response = client.messages.create(
    model=MODEL,
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": POLICY + EXAMPLES,
        "cache_control": {"type": "ephemeral"},   # cache up to here
    }],
    messages=[{"role": "user", "content": dynamic_context + user_message}],
)
print(response.usage.cache_creation_input_tokens,
      response.usage.cache_read_input_tokens)
```

Everything from the start of the prompt through the marked block is cached; everything after is processed fresh each call. You want the breakpoint as far down as possible while staying before the first thing that varies. Anthropic allows up to 4 breakpoints, useful when tools change rarely but a document block changes daily, and also offers automatic caching via a single top-level `cache_control` that moves the breakpoint forward as a conversation grows.

## TTL and hit rate

Caches expire. Anthropic's default is 5 minutes, refreshed at no cost on each hit, with a 1-hour option at a higher write price; OpenAI's GPT-5.6 and later default to a 30-minute minimum lifetime. The implication: caching pays off when requests sharing a prefix arrive **close together in time**. A high-traffic endpoint with a shared system prompt is the ideal case; the cache stays warm continuously. A low-traffic endpoint may see the cache expire between requests, so you pay the (small) cache-write premium without reaping the read discount.

Check the hit rate the provider reports (`cache_read_input_tokens` against `cache_creation_input_tokens` on Anthropic, `cached_tokens` on OpenAI). If both Anthropic fields are 0, the prefix was probably below the minimum length, and the docs note no error is returned. If you're mostly writing and rarely reading, either your prefix isn't stable (find the poisoning token) or your traffic is too sparse. For sparse-but-regular traffic, the longer TTL can flip the math.

## Conversation history: cache the growing prefix

For multi-turn agents, the conversation history is append-only: each turn adds to the end, and the beginning stays identical. That's a perfect caching shape. Place the cache breakpoint at the end of the *prior* turn so each new request re-uses the entire conversation so far as a cached prefix and only processes the new turn. Long agent loops benefit the most, because the cached prefix grows while the per-turn fresh tokens stay small. If you are trimming history to fit a window, do it in large, infrequent steps; every trim rewrites the prefix (our notes on [token budgeting for long-context prompts](/blog/token-budgeting-long-context-prompts) cover the trade-off).

## What it bought us

- **Cost**: on cache hits, the stable prefix bills at the read rate (0.1x on most models) instead of full input price; on a system-prompt-heavy workload that is a large fraction of the total bill.
- **Latency**: time-to-first-token improved noticeably because the model skips re-processing the cached prefix. The bigger the stable prefix, the bigger the win.
- **Behavior**: no change to what the model is conditioned on. The prompt is identical, so caching is a cost and latency optimization, not a quality trade.

Prompt caching is distinct from [semantic caching](/blog/semantic-caching-for-llm-apps), which skips the model call entirely for similar questions. They stack, and both belong in a broader [LLM cost optimization](/blog/llm-cost-optimization-in-production) plan.

## Questions people ask

### What is prompt caching in LLM APIs?

Prompt caching stores the model's processed state for a prompt prefix so that a later request starting with the exact same tokens can reuse it. Cached tokens bill at a fraction of the input price (0.1x on most Anthropic and OpenAI models as of October 2026) and are not recomputed, which also cuts time-to-first-token. Any difference, even one character, ends the reusable prefix at that point.

### How long does the prompt cache last?

On Anthropic's API the default lifetime is 5 minutes, refreshed for free every time the prefix is read, with an optional 1-hour TTL whose writes cost 2x base input. OpenAI's GPT-5.6 and later default to a 30-minute minimum, with a 12-hour option at 2.5x. Lifetimes are minimums, so steady traffic keeps a prefix warm indefinitely.

### Why is my prompt cache hit rate so low?

Usually something dynamic sits above the breakpoint: a date, request ID, user name, or reordered tool list. Diff two consecutive prompts and look for the first changed token. Other causes are a prefix under the provider's minimum (512 tokens on current Claude models, 1,024 on recent OpenAI models), or traffic so sparse that entries expire between requests.

## The discipline, restated

Prompt caching is almost free to adopt and almost free to ruin. Structure every prompt as `[stable] then [dynamic]`, set the cache breakpoint at the boundary, keep dynamic tokens (dates, IDs, names) strictly below it, and watch the cache-read ratio. Done right, you stop paying full price to re-read the same instructions thousands of times a day for nothing.

## Sources

- [Anthropic docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
- [OpenAI API docs: Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)