Token caching, model routing, prompt compression, and the boring discipline of measuring. The levers that cut our LLM bill 60% without touching feature scope.
LLM Cost Optimization in Production — What Actually Moves the Bill
Our LLM bill grew 8x in six months. Some of that was feature growth; most was operational waste. We spent a quarter doing nothing but cost optimization and cut the bill 60% with no user-visible regression. This is what worked, what we tried and dropped, and the discipline that keeps the bill in line now.
Before any optimization, instrument. We added cost-per-request to our request logs:
- Prompt tokens
- Completion tokens
- Model name
- Total cost (computed at log time from the per-model pricing table)
- Feature / endpoint that made the request
Once this was in place we could ask the questions that matter: "Which feature is 80% of cost?" "Which prompts have 100x cost variance?" "What's the cost per user / per session?"
Before measurement, every conversation about cost was vibes. After, it was specific. Two features turned out to be 70% of the bill; the rest was rounding. That's where we focused.
A lot of LLM use is repeated work. The same prompts come in over and over (same system message, same retrieval context for popular queries, same user re-asking). Caching at multiple levels:
Provider-level prompt caching. Anthropic Claude (and now OpenAI) support marking sections of the prompt as cacheable. The provider keeps the KV cache for those sections; subsequent requests with the same prefix pay a fraction of the input-token cost (often 10x cheaper). The cache TTL is 5 minutes by default; if you have high enough QPS to keep it warm, the savings compound.
We marked our system message + retrieval context as cacheable. Hit rate ~70%. Input token cost dropped ~60% on cached requests.
Application-level response caching. Some prompts are deterministic enough to cache the response. If a user asks "summarize this document" and we've already summarized it, return the cached summary. We cache responses keyed by hash of prompt + retrieval-context + model. TTL 1 hour for fresh content, 7 days for stable content.
Hit rate ~25% on the use cases where it applies. The savings here are 100% (no LLM call at all).
Be careful: response caching is wrong for personalized output or stateful chat. We allow it only on idempotent operations (summarize, extract, classify).
Not every request needs the most powerful model. We added a router that picks the model based on the request:
- Simple classification → Claude Haiku 4.5 or equivalent small model.
- Standard generation → Claude Sonnet 4.6.
- Complex reasoning, long context → Claude Opus 4.7.
We started with rules ("if endpoint = X, use small model"), then added a learned classifier ("predict if Sonnet suffices; fall back to Opus if confidence < threshold"). The classifier itself is a cheap call.
Result: ~40% of requests now hit a smaller model. Quality on those requests is comparable; cost is roughly 5x lower.
The risk: routing to a model that can't handle the task. We added evaluation gates — every routing decision is sampled and the output evaluated against the result the big model would have produced. If quality drops below threshold for a route, we re-route to the bigger model and update the rules.
The prompts we'd grown over six months were enormous. System messages were 4K tokens. Retrieval context was 8K tokens. Many of those tokens weren't earning their keep.
What we cut:
- Redundant instructions. "Be helpful. Be accurate. Be concise. Don't make things up." Each repeated phrasing in the system message added tokens without changing output.
- Multi-shot examples that didn't move quality. Some examples were load-bearing; many were vestigial. We A/B tested removing them.
- Retrieval context. We were retrieving top-10 documents. Re-ranking + relevance filtering reduced this to top-3 most of the time. 70% fewer retrieval tokens.
Net prompt size dropped ~50%. Cost dropped proportionally on uncached requests. Quality (measured by eval suite) was unchanged.
For async/background work (re-summarizing a corpus, generating embeddings for a knowledge base, batch classification), batch APIs are usually 50% cheaper than synchronous ones.
We moved everything not user-facing to batch. This was a one-time refactor with permanent savings.
Note: batch APIs have higher latency (minutes to hours), so they're not a fit for user-facing requests. But "user-facing" is a smaller fraction of LLM work than you'd think.
Every time we audited, we found bugs:
- A retry loop that re-prompted on transient errors without exponential backoff. 5 calls per request.
- A worker that polled a queue and re-generated the same summary every minute because of a cache key bug.
- A test environment that was hitting prod LLM endpoints because of a misconfigured base URL. 20% of our spend for a while.
These aren't optimization; they're firefighting. But they were the single biggest delta in the first month. The instrumentation from step one is what surfaced them.
Prompt compression libraries (LLMLingua etc). Quality regression in our use cases didn't justify the savings. The hand-crafted prompt edits gave better results.
Distilling our own small model. Considered fine-tuning Llama or similar on our task. The eval/maintenance cost of running our own model didn't pay back at our scale (~10M LLM calls/month). At 10x that scale, it might.
Streaming for cost. Some teams claim streaming "feels" faster so users wait less and you save tokens. We didn't see this effect; users let long responses finish. Streaming is a latency feature, not a cost one.
Reducing output token max length. Cutting max tokens by 30% to "cap costs." Caused truncated responses. Users retried. Net cost increase.
What we do continuously now:
- Weekly cost review. 15-minute meeting. Look at total spend, per-feature spend, anomalies. Investigate anything > 20% week-over-week change.
- Per-feature cost dashboards. Each feature owner sees their feature's cost trend. Creates accountability.
- Eval gate on prompt changes. Any prompt change is run through the eval suite before merge. Both quality (must not regress) and cost (token count is reported on the PR).
- Quarterly model audit. Are we using the right model for each route? Costs change; models change; rerunning the routing analysis quarterly catches drift.
- Cost per request, per feature, per user. P50 and p95.
- Cache hit rate. Drops = something changed in the prompt structure that broke cache keys.
- Model usage distribution. Should match our routing intent.
- Failed request rate. Failed retries are wasted tokens; track them.
The biggest savings came from non-LLM bugs. Half of our reduction was "fix dumb things in code that was hitting LLM unnecessarily." The remaining half was thoughtful optimization. We expected the inverse.
Prompt caching is undersold. Provider prompt caching is a recent feature and many teams haven't integrated it. The savings are huge for high-QPS use cases. Restructure prompts so the long stable parts come first; pay attention to cache invalidation.
Routing has a quality ceiling. You can't route everything to a small model. The classifier itself isn't perfect; the cheaper models are worse at edge cases. Accept some baseline of "everything goes to the big model"; optimize the rest.
LLM cost optimization is mostly the same discipline as any other infra cost optimization: measure, find the big things, fix them, build operational rhythm to keep finding them. The specifics (prompt caching, model routing) are LLM-flavored, but the meta-pattern is timeless.
