Skip to main content
Token Budgeting for Long-Context Prompts: What to Cut First

Token Budgeting for Long-Context Prompts: What to Cut First

KU
· Founder & Chief Editor
• 7 min read•Updated •60 views

A huge context window is not a license to stuff everything in. Here's the order we trim long prompts in, and why retrieval is almost always first.

Key takeaways

  • Measure tokens per prompt segment first, then cut in this order.
  • Trim over-retrieval (lower k, add a score floor or reranker), prune stale few-shot examples, summarize old conversation history while keeping recent turns verbatim, and prompt-cache the stable prefix.
  • Cache reads bill at about a tenth of the normal input price on Anthropic and recent OpenAI models.

Cut long-context prompts in this order: over-retrieved RAG chunks first, then stale few-shot examples, then old conversation history (summarize it, keep recent turns verbatim), and finally prompt-cache the stable prefix you must keep. Measure tokens per segment before cutting anything. Retrieval is usually the biggest and safest cut.

Verified against

tiktoken (o200k_base via encoding_for_model("gpt-4o")), Anthropic Messages API token counting and prompt caching, OpenAI prompt caching for GPT-5.6 and later, against the tiktoken README and the Anthropic and OpenAI prompt-caching docs. Checked October 2026.

The way prompts get expensive is rarely a traffic spike. It's scope creep in the prompt itself: every engineer who touches it adds "just a little more context," and the average request quietly grows to tens of thousands of input tokens, a lot of which never changes the answer. Big context windows make this worse, because a window that can hold a whole manual feels free until the invoice arrives. Input tokens are cheaper than output, but at scale they dominate, because you pay for them on every call.

Measure before you cut

You can't budget what you can't see. Log token counts per prompt segment. Don't estimate with "4 characters per token"; it's wrong often enough to mislead you. Count for real with the tokenizer for your model.

token_audit.py
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")  # o200k_base

def count(text: str) -> int:
    return len(enc.encode(text))

segments = {
    "system": count(system_prompt),
    "few_shot": count(examples_block),
    "retrieved_docs": count(rag_context),
    "history": count(conversation_history),
    "user_msg": count(user_message),
}
print(segments)

tiktoken reflects OpenAI tokenization only. For Claude, call the Messages API's count_tokens endpoint (client.messages.count_tokens(...) in the Python SDK) with the same system prompt and messages you'd send; Anthropic documents the result as an estimate that can differ by a small amount.

An illustrative breakdown for an over-stuffed support assistant: system 1,200 tokens, few-shot 6,800, retrieved docs 18,000, history 7,400, user message 600. Laid out like that, the fat is obvious: retrieved docs and few-shot examples are about 73% of the prompt. Your numbers will differ; the shape usually doesn't.

Cut retrieval first, it's usually the worst offender

Most RAG pipelines over-retrieve. A k=12 set during a demo and never revisited is common. The way to find your real number is an offline eval: run your golden question set at several values of k and compare answer quality. Typically the curve flattens fast, and chunks past the first handful are dead weight. A harness for exactly this is in evaluating RAG retrieval offline.

retrieve.py
results = vector_store.similarity_search_with_score(query, k=5)
# drop weak matches; tune the floor on your eval set, it is model-specific
results = [doc for doc, score in results if score > SCORE_FLOOR]

Two changes: lower k, and add a score floor so weak matches don't get padded in. With 1,500-token chunks, going from 12 to 5 removes 10,500 tokens per call by arithmetic alone. Add a reranker and you can often go lower, because better ordering means you need fewer chunks; whether a cross-encoder is worth its latency is covered in reranking in RAG.

Few-shot examples: keep two, kill the rest

Few-shot blocks rot. People add an example every time they see a failure, and nobody removes one. Run your eval dropping examples one at a time; quality often plateaus at two or three well-chosen ones.

The counterintuitive bit: a bad example hurts more than a missing one. An example written in a verbose format steers every answer toward that format. Removing it can improve quality and cut tokens at the same time.

History: summarize the old, keep the recent verbatim

Conversation history grows without bound. Truncating to the last N messages loses context; keeping everything is expensive. The compromise: keep the last few turns verbatim and replace everything older with a running summary.

history.py
if len(history) > 6:
    old = history[:-6]
    summary = summarize(old)  # a cheap model call, cached
    history = [{"role": "system", "content": f"Earlier: {summary}"}] + history[-6:]

Run the summary on a cheaper model and cache it, so the amortized cost is small and history stays roughly flat no matter how long the chat runs. If you're routing that call down a tier, when to route down to a small model covers the tradeoffs.

Prompt caching pays for the stuff you must keep

Some context genuinely can't be cut: a long system prompt with policy rules, a stable few-shot block. For those, use provider prompt caching, and order the prompt so the stable parts come first.

The numbers, as of October 2026. On Anthropic, cache reads bill at 0.1x the base input price for most models (0.05x on Claude Opus 5.5 and Sonnet 5.5, 0.025x on Fable 5.1 and Mythos 5.1), 5-minute cache writes at 1.25x and 1-hour writes at 2x. The prefix is evaluated in the order tools, system, messages, and the minimum cacheable length ranges from 512 to 4,096 tokens depending on the model; shorter prompts simply aren't cached, with no error. On OpenAI, caching is on by default for supported models, cache reads for GPT-5.6 and later bill at 0.1x the input rate, and the minimum is 1,024 tokens. Our broader production setup is in prompt caching for production LLM apps.

The rule that matters on both: caches key on an exact prefix match, so anything dynamic (the user's question, retrieved docs, timestamps) must go last. A timestamp in the system prompt silently gives you a 0% hit rate.

Where this breaks

Every cut here is a bet that your eval set represents production traffic. If the golden set is small or stale, you'll trim the context that the long-tail questions needed and never see it in the score. Re-run the eval when the product changes, and watch user-reported failures for a week after each cut.

Questions people ask

What should I cut first when a prompt is too long?

Cut over-retrieval first. RAG pipelines commonly pull more chunks than the model needs, and lowering k plus adding a relevance score floor removes thousands of tokens with little quality risk, especially with a reranker. Then prune few-shot examples, then summarize old conversation history. Leave the system prompt for last; it's usually the smallest segment and the one prompt caching discounts best.

How do I count tokens accurately before sending a prompt?

Use the tokenizer for your model rather than a characters-per-token guess. For OpenAI models, tiktoken.encoding_for_model("gpt-4o") returns the o200k_base encoding. For Claude, call the Messages API count_tokens endpoint with the same system prompt, tools and messages you plan to send; Anthropic notes the count is an estimate. Log counts per prompt segment so you can see which part is growing.

How much does prompt caching save on input tokens?

As of October 2026, cache reads cost 0.1x the base input price on most Anthropic models and on OpenAI GPT-5.6 and later, with lower multipliers on some newer models. Anthropic charges 1.25x or 2x for cache writes depending on TTL, so savings depend on hit rate. Keep stable content at the front of the prompt; any change before the cache breakpoint invalidates everything after it.

The call we'd make

Cut in this order: over-retrieval, then stale few-shot, then unbounded history, then cache what's left. Retrieval is almost always where the waste hides, and it's the safest cut because a good reranker means fewer chunks is often better, not just cheaper. Don't optimize the system prompt first; it's the smallest slice and the one caching handles best anyway.

Sources

Explore topics:LLMAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Token Budgeting for Long-Context Prompts: What to Cut First. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/token-budgeting-long-context-prompts

Markdown

[Token Budgeting for Long-Context Prompts: What to Cut First](https://www.devopsness.com/blog/token-budgeting-long-context-prompts) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

604 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.