Cutting LLM Inference Costs by 80%

A year ago we were spending around $11,200/month on LLM API calls (mostly OpenAI, some Anthropic). After working through it methodically, we're now at $2,300/month for roughly the same workload. This post is the seven changes that got us there, ranked by impact, with the "we tried this and it didn't help" notes too.

The starting point#

Workload breakdown when we started:

• ~3M requests/month split across 6 features
• Average input: 1,400 tokens; average output: 350 tokens
• 95% on gpt-4 (the original), 5% on gpt-3.5-turbo
• Bill: ~$11,200/month, dominated by GPT-4 calls

The biggest spend was a customer-facing assistant that answered questions about our product. ~60% of total cost.

Change 1: Use the right model for each task (largest impact)#

We were using gpt-4 for everything because someone benchmarked it once and it was best. Different tasks have different difficulty, and most don't need the most capable model.

We re-benchmarked each feature against several models:

The biggest savings came from realizing classification and RAG-with-good-context don't need GPT-4. They need consistent output, and gpt-4o-mini (or Claude Haiku) does that for a fraction of the cost.

The "agentic task runner" needed careful handling. Cheap models would sometimes get stuck; we built a fallback: if the cheap model returns "I'm not confident" or hits a retry limit, escalate to GPT-4 for that task. Most tasks (~85%) finish on the cheap model.

Estimated saving: ~$5,500/month. The biggest single change.

Change 2: Prompt caching#

Anthropic and OpenAI both added prompt caching: repeated prefixes (system prompt, few-shot examples) are billed at lower rates (50-90% off the cached portion).

We restructured our prompts to put the long stable parts first:

[CACHED — stable system prompt + examples + tool definitions, ~3000 tokens]
---
[NOT CACHED — user query + retrieved context, ~1500 tokens]

Before caching, every call billed for all 4500 input tokens. After caching, ~3000 tokens are at the cached rate (10% of normal for Anthropic's cache hits).

For our customer assistant, this dropped per-call cost by ~50%.

Estimated saving: ~$1,800/month.

Change 3: Stop sending unnecessary context#

Our RAG pipeline was retrieving 10 chunks and sending all 10 to the model. We added a re-ranker that scores the 10 against the query; we now send only the top 4 to the LLM.

Less context = fewer input tokens = less cost. Quality stayed flat (or improved slightly — less noise for the model to filter through).

Average input tokens dropped from ~2,200 to ~1,100. ~50% reduction in input cost on RAG queries.

Estimated saving: ~$1,400/month.

Change 4: Output length limits#

We had no max_tokens set. Some responses were 1500 tokens. Most should be 200.

We set per-task max_tokens based on the task:

• Classification: 50
• Short summary: 200
• Detailed response: 600
• Long-form: 1500 (rare)

Two effects: capped output cost, and forced the model to be concise (the prompts were updated to say "respond in N words"). Quality didn't suffer; users like shorter responses.

Estimated saving: ~$400/month.

Change 5: Streaming with early-stop on classification#

For classification tasks, we don't need the model to keep generating after it has produced the category. We:

1. Stream the response
2. As tokens arrive, parse for the category JSON field
3. Once we have a valid classification, close the stream

This stops the model mid-generation, avoiding tokens we don't use. For tasks where the model would otherwise generate explanations after the category, savings are real.

This works because OpenAI/Anthropic bill on tokens generated, even those not delivered. Closing the stream stops the meter.

Estimated saving: ~$300/month.

Change 6: Batch processing for non-urgent work#

For background tasks (re-summarizing old docs, generating internal search indexes), we use OpenAI's Batch API: half-price for processing within 24 hours.

Most "urgent" features stayed real-time. About 20% of our LLM volume moved to batch.

Estimated saving: ~$600/month.

Change 7: Caching responses for common queries#

For the customer assistant, ~12% of queries were near-duplicates of previously-asked questions. We added a query-similarity cache: if a new query is semantically very close to a recent answered query, return the cached answer.

Implementation:

• Embed the query
• Search a Redis-backed vector store of recent (query, answer) pairs
• If similarity > 0.92 and answer is < 24 hours old, return cached answer
• Otherwise call the LLM and store the result

Cache hit rate: ~12%. Each hit saves a full LLM call.

Estimated saving: ~$400/month.

What we tried and abandoned#

A few changes that sounded good but didn't deliver:

Self-hosting open-source models. We benchmarked Llama-3-70B on H100 instances. The throughput was OK but the cost (GPU rental) ended up similar to OpenAI's gpt-4o-mini for our patterns. Plus operational overhead. Not worth it for our scale; might be different at 10x our volume.

Distillation: fine-tuning a smaller model on GPT-4 outputs. Spent two weeks on this for one specific task. The fine-tuned model was 70% as good. The remaining 30% gap mattered for our use case (it was a customer-facing classifier where wrong answers hurt). Reverted.

Aggressive prompt compression (using a smaller model to compress context before passing to the bigger model). The compression itself costs tokens and loses information. Marginal at best.

Switching providers based on per-call cost. Tried routing each request to whichever provider was cheapest at that moment. The gain was small (most providers price similarly), and the operational complexity of multi-provider routing wasn't justified.

What we monitor now#

Visibility was as important as the changes:

• Cost per task type per day, broken down by model. Spikes show up immediately.
• Tokens per request distribution for each task. Outliers (a few requests using 50k tokens) often indicate a bug.
• Cache hit rate for the response cache.
• Per-user / per-customer cost. A small number of customers can drive disproportionate cost; visibility lets us address it.

The dashboards live in Grafana, fed from Datadog APM (we wrap every LLM call with span attributes for tokens and cost).

Specific incidents that drove changes#

The 500-token user prompt that cost $50. A user typed a request like "summarize this:" followed by 100k tokens of pasted text. Our token budget didn't catch it; the LLM call cost $50. We added per-call hard input-token caps; anything beyond gets truncated with an explanatory message.

The agent that looped. A bug in an agent caused it to repeat its own output back to itself, growing the context each iteration. After 80 iterations, one task had cost $200. Per-task cumulative-token caps now stop this.

The SaaS feature that 100x'd its volume overnight. A customer enabled a feature heavily, generating 100k LLM calls in a day. Our daily cost jumped 50x. We added per-customer rate limits to prevent runaway costs from individual customers.

What I'd tell someone starting#

The cheapest token is the one you don't send. Before optimizing model selection, check if you're sending unnecessary input.

Use the right model for each task. This is the biggest lever. Don't run GPT-4 on classification tasks.

Add observability first. You can't optimize what you can't see. Per-task cost dashboards make the wins obvious.

Set per-task token caps. Hard limits prevent surprise bills from edge cases.

Cache when possible. Both prompt caching (provider-side) and response caching (your-side) compound.

Don't chase exotic optimizations early. Self-hosting, fine-tuning, multi-provider routing — these are big projects with marginal payoff at small scale. Hit the easy wins first.

Most teams I've talked to pay 3-10x more for LLM inference than they need to. The optimizations aren't exotic. They're: pick the right model, send less context, set output limits, cache where you can, and watch the bill.