LLM Cost Optimization in Production: What Actually Moves the Bill

Cut an LLM bill in this order: log cost per request and feature, fix wasteful loops, turn on prompt caching (cache reads cost 0.1x of input or less), route easy requests to cheaper models, trim prompts, and move background work to batch at 50% off. Measure first: it shows which two features own the bill.

Step one: actually measure

Before any optimization, put cost on every request log line: prompt tokens, cached tokens, completion tokens, model, computed cost, and the feature or endpoint that made the call. Every provider returns the token counts; you only have to keep them. A minimal wrapper with the Anthropic Python SDK:

# llm_cost.py
import anthropic, json, logging

PRICES = {  # USD per million tokens, Claude API list prices, October 2026
    "claude-sonnet-5-5": {"in": 2.00, "cache_write": 2.50, "cache_read": 0.10, "out": 10.00},
    "claude-haiku-5-5":  {"in": 0.10, "cache_write": 0.125, "cache_read": 0.01, "out": 0.50},
}
client = anthropic.Anthropic()

def call(feature: str, model: str, system: str, user: str) -> str:
    resp = client.messages.create(
        model=model, max_tokens=1024,
        system=[{"type": "text", "text": system, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": user}],
    )
    u, p = resp.usage, PRICES[model]
    cost = (u.input_tokens * p["in"] + u.cache_creation_input_tokens * p["cache_write"]
            + u.cache_read_input_tokens * p["cache_read"] + u.output_tokens * p["out"]) / 1e6
    logging.info(json.dumps({"feature": feature, "model": model, "cost_usd": round(cost, 6),
                             "in": u.input_tokens, "cache_read": u.cache_read_input_tokens,
                             "cache_write": u.cache_creation_input_tokens, "out": u.output_tokens}))
    return resp.content[0].text

The Haiku 5.5 prices above apply to prompts up to 100,000 tokens; longer prompts pay $0.50 / $2.50. Once this is in place you can ask the questions that matter: which feature is most of the spend, which prompts have 100x cost variance, what a session costs. In our experience the answer is almost always concentrated: a couple of features own most of the bill and the rest is rounding.

Lever 1: stop the loops (the first-month win)

Instrumentation usually surfaces bugs before it surfaces optimizations: a retry loop that re-prompts on transient errors without backoff, a worker that regenerates the same summary every minute because of a cache-key bug, a staging environment pointed at the production API key. These aren't optimization, they're firefighting, and they are often the biggest single delta. Retry on rate_limit_error (HTTP 429) and overloaded_error with exponential backoff that honours retry-after, and cap attempts.

Lever 2: provider prompt caching

Most production prompts repeat a long, stable prefix: system message, tool definitions, retrieval context for popular queries. Caching that prefix is the biggest structural win.

Anthropic's cache has a 5-minute default TTL that resets on every hit; a 1-hour TTL costs 2x base input to write instead of 1.25x. You get up to 4 explicit cache_control breakpoints per request, or one top-level cache_control for automatic caching. Put stable content first and anything per-user last, or every request writes a new cache entry and you pay the write premium for nothing. OpenAI caches automatically: on GPT-5.6 and later the minimum is 1,024 visible tokens, cached reads cost 0.1x on most models, and the default retention is 30 minutes. The structuring discipline is covered in depth in prompt caching for production LLM apps.

Application-level response caching goes further: if the same document was already summarized, return the stored summary and skip the call. Key it on a hash of prompt, retrieval context and model, and allow it only for idempotent operations (summarize, extract, classify), never for personalized output or stateful chat.

Lever 3: model routing

Not every request needs the frontier tier. Sonnet 5.5 costs 20x Haiku 5.5 per input token at the prices above. A router picks the model per request: classification and extraction go to Haiku 5.5, standard generation to Sonnet 5.5, hard multi-step reasoning to Opus 5.5. Start with rules ("endpoint X uses the small model"), then add a cheap classifier that predicts when the small model suffices. When to route down to Haiku 5.5 covers where that breaks, and multi-provider LLM routing with failover expands the routing layer across vendors.

The risk is routing a task to a model that can't handle it. Sample routed requests and score them against what the bigger model would produce, using LLM evals that predict production quality. If a route's quality drops below threshold, move it back up.

Lever 4: prompt compression and batching

Prompts grow by accretion. Cut repeated instructions ("be helpful, be accurate, be concise"), A/B test removing few-shot examples one at a time, and retrieve fewer, better documents: re-ranking with hybrid BM25 plus embeddings search usually lets you send three chunks instead of ten. Gate every prompt change on your eval suite and report the token delta on the PR.

For anything not user-facing (re-summarizing a corpus, bulk classification, offline evals), batch APIs charge half. Anthropic's Message Batches API takes up to 100,000 requests or 256 MB per batch, most batches finish within an hour, and anything unfinished at 24 hours expires unbilled. Results stay downloadable for 29 days. The batch discount stacks with prompt caching.

What we tried and dropped

Prompt compression libraries (LLMLingua and similar): the quality regression on our tasks outweighed the savings; hand edits did better. Distilling our own small model: the eval and serving burden didn't pay back at moderate volume, and a cheaper hosted tier now covers most of that ground. Streaming for cost: streaming is a latency feature, covered in LLM streaming UX with backpressure and cancellation, not a cost one. Slashing max_tokens to cap spend: truncated answers made users retry, which cost more.

Failure modes worth alerting on

• Both cache_read_input_tokens and cache_creation_input_tokens are 0. The prefix is below the model's minimum cacheable length (512 tokens on the 5.5 models, up to 4,096 on Haiku 4.5) and was silently not cached.
• Cache writes every request, reads never. A timestamp, user ID or shuffled tool list sits before the breakpoint. Move dynamic content after it.
• A 400 error after adding automatic caching. All 4 breakpoint slots were already used by explicit cache_control blocks.
• Batch results marked expired. The batch didn't finish within 24 hours; resubmit those requests.
• Token counts jump after a model upgrade. Claude 4.7 and later use a tokenizer that produces roughly 30% more tokens for the same text, so per-token price alone understates the change. Re-measure cost per request after every migration.

Where this breaks

Caching does little for short, unique prompts. Routing has a quality ceiling: edge cases still need the big model, so accept a baseline that always goes there. Batch is useless for interactive paths. And list prices move: models get retired (see the GPT-4o shutdown schedule), so keep the price table in config and re-check it quarterly rather than hard-coding last year's numbers.

Questions people ask

How do I reduce LLM API costs?

Log cost per request and per feature, then fix retry loops and misrouted environments before anything clever. Next, enable prompt caching with stable content first, route easy requests to a smaller model behind an eval gate, trim prompts and retrieval context, and move non-interactive jobs to a batch API; Anthropic's Message Batches API charges half the synchronous price.

How much does prompt caching save?

On cached tokens, a lot. As of October 2026, Anthropic charges 0.1x the base input price for cache reads on most models and 0.05x on Opus 5.5 and Sonnet 5.5, while 5-minute cache writes cost 1.25x. OpenAI charges 0.1x on most GPT-5.6+ models. Real savings depend on hit rate, so track cache read tokens per feature.

Is the batch API worth it?

For background work, yes. Anthropic's Message Batches API charges 50 percent of standard input and output prices, accepts up to 100,000 requests per batch, and most batches finish within an hour, with a 24-hour ceiling. It's wrong for anything a user is waiting on, but corpus summarization, classification backfills and offline evals fit well.

The call we'd make

Instrument first, fix the dumb bugs second, then turn on prompt caching with a cache-friendly prompt layout; those three usually account for most of the achievable savings. Add routing only once you have an eval suite that can catch a quality drop, and move every non-interactive job to batch. LLM cost work is ordinary infrastructure cost work: measure, find the big items, fix them, and hold a weekly review so the bill doesn't drift back.

Sources

• Anthropic: Claude API pricing
• Anthropic: Prompt caching
• Anthropic: Batch processing
• OpenAI: Prompt caching