A practitioner's guide to how LLM API pricing works, how to estimate a workload's monthly bill, and the levers that actually cut it.
Most teams discover LLM cost the same way: a demo works, traffic grows, and the bill quietly becomes a line item someone in finance circles in red. The good news is that LLM pricing is more predictable than it looks once you understand the units. This post walks through how the pricing model works, how to estimate a workload before you ship it, and where the real savings live.
A note before the numbers: every rate below is directional. Provider pricing moves often, tiers get renamed, and new models land almost monthly. Treat the figures as ratios and rules of thumb, then verify current rates on the provider's own pricing page before you commit a budget.
Almost every hosted LLM charges per token, and tokens are billed in two separate buckets:
Input tokens: everything you send. Your system prompt, the user message, retrieved documents, prior conversation turns, and tool definitions all count.
Output tokens: everything the model generates back.
The two have different rates, and output is almost always more expensive, often 3x to 5x the input rate. As a directional example, a mid-tier frontier model might run around $3 per million input tokens and $15 per million output tokens. The gap exists because output is produced one token at a time in an autoregressive loop, so it is the compute-heavy part of the request. Input tokens are processed in a single parallel forward pass.
That asymmetry drives a lot of optimization decisions. A verbose prompt is annoying but cheap. A model that rambles for 800 tokens when 100 would do is where budgets actually bleed.
Three pricing mechanics matter beyond the base rates:
Context caching: if you send the same large prefix repeatedly (a long system prompt, a fixed knowledge base, a big document you ask many questions about), most providers let you cache it. Cached input tokens are billed at a steep discount, often 10% to 25% of the normal input rate. There is usually a small write cost to populate the cache and a short time-to-live.
Batch APIs: if you can tolerate latency measured in minutes or hours instead of milliseconds, batch endpoints typically cut the per-token rate roughly in half. Ideal for offline classification, enrichment, and evals.
Per-request minimums: some providers, especially for embeddings or specialty models, round up tiny requests. Rarely a headline cost, but it matters at high request volume with small payloads.
The formula is simple. For each request:
cost_per_request = (input_tokens / 1_000_000) * input_rate
+ (output_tokens / 1_000_000) * output_rate
monthly_cost = cost_per_request * requests_per_month
Here is a worked example as a small script you can adapt:
# Directional rates ($ per 1M tokens). Verify current numbers before you rely on them.
INPUT_RATE = 3.00
OUTPUT_RATE = 15.00
def request_cost(input_tokens, output_tokens):
return (input_tokens / 1_000_000) * INPUT_RATE + \
(output_tokens / 1_000_000) * OUTPUT_RATE
# A support-assistant workload:
avg_input = 1_500 # system prompt + retrieved context + user question
avg_output = 400 # the answer
requests_per_month = 500_000
per_req = request_cost(avg_input, avg_output)
monthly = per_req * requests_per_month
print(f"Per request: ${per_req:.5f}") # ~$0.01050
print(f"Monthly: ${monthly:,.0f}") # ~$5,250
That $5,250 is your baseline. Now the interesting question: how much of it is avoidable?
Right-size the model. Not every request needs your best model. Routing simple classification, extraction, and short factual answers to a smaller model can be 10x to 30x cheaper per token. A router that sends easy traffic to a cheap model and escalates only the hard cases is usually the single biggest win.
Prompt caching. In the example above, most of those 1,500 input tokens are a fixed system prompt and stable context. Cache them. If 1,200 of the 1,500 input tokens are cacheable at ~10% of the rate, your input cost on those tokens drops by roughly 90%, trimming a meaningful slice off the monthly total.
Batching. Move anything asynchronous (nightly enrichment, backfills, evals) to the batch API and halve those tokens.
Shorten prompts. Retrieval that dumps ten documents when three would answer the question is pure waste. Tighten your RAG top-k and trim boilerplate.
Cap max output. Set max_tokens deliberately. If answers should be two sentences, do not leave room for two pages. This directly caps your most expensive tokens.
Semantic caching. For workloads with repeated or near-duplicate questions, cache answers keyed on an embedding of the query. A cache hit costs an embedding lookup instead of a full generation.
Here is caching wired into the same estimate:
def cached_request_cost(cached_in, fresh_in, output_tokens, cache_rate=0.30):
return (cached_in / 1_000_000) * INPUT_RATE * cache_rate + \
(fresh_in / 1_000_000) * INPUT_RATE + \
(output_tokens / 1_000_000) * OUTPUT_RATE
# 1,200 cacheable input tokens, 300 fresh, same 400 output
per_req = cached_request_cost(1_200, 300, 400)
print(f"Monthly: ${per_req * 500_000:,.0f}") # meaningfully below $5,250
The cost gap between top frontier models and open-weight models served on commodity infrastructure is wide, often 5x to 20x per token. That does not automatically make open-weight cheaper in practice. Self-hosting adds GPU rental, ops time, autoscaling headaches, and the risk of paying for idle capacity. The rule of thumb: open-weight wins on cost at steady, high, predictable volume where you can keep hardware busy. Frontier hosted APIs win on spiky or low volume where you pay only for what you use and skip the ops burden. Many teams end up mixing both.
The bill on the pricing page is not the whole bill:
Retries and timeouts. Every failed-and-retried request that still consumed output tokens is billed. Aggressive retry logic on a flaky prompt can quietly double spend.
Embeddings. RAG means embedding every document once and every query forever. Cheap per token, but at scale it is a real line item.
Reranking. Reranker models charge per query-document pair, which multiplies fast with a large candidate set.
Agent loops. Multi-step agents re-send growing context on every turn. A ten-step agent can burn far more tokens than the single call you estimated.
A practical order of operations:
- Instrument first. Log input and output tokens per request so you know where money goes before you optimize.
- Cap
max_tokens everywhere and tighten prompts. Free, immediate.
- Turn on prompt caching for any stable prefix.
- Route easy traffic to a smaller model.
- Move async work to batch.
- Add semantic caching if you see repeated queries.
- Re-measure. Only consider self-hosting once volume is high and steady.
Start on a hosted frontier API, but instrument token usage from day one. Set max_tokens, enable prompt caching, and add a cheap-model router before traffic scales, not after the bill surprises you. Keep open-weight self-hosting in your back pocket for when volume is provably high and steady enough to keep the GPUs busy. The pattern that holds up: pay premium rates only for the hard requests, and make everything else cheap by default.
For the wider landscape of providers and infrastructure, see our guide to the best LLM APIs and AI infrastructure, and for a head-to-head on the major vendors, OpenAI vs Anthropic vs Gemini.
