Token Budgeting for Long-Context Prompts: What to Cut First

Token Budgeting for Long-Context Prompts: What to Cut First#

Our support-assistant bill hit $9,400 in a single month and the finance team wanted a word. Traffic hadn't doubled. What had happened was scope creep in the prompt: every engineer who touched it added "just a little more context," and the average request had ballooned to 34,000 input tokens. Half of it was never read by the model in any way that changed the answer.

Big context windows tempt you into lazy prompting. A 180k or 200k window feels free until the invoice arrives. Input tokens are cheaper than output, but at scale they dominate, because you pay for them on every single call and they don't shrink.

Measure before you cut#

You can't budget what you can't see. First thing we did was log token counts per request segment. Don't estimate with "4 characters per token," it's wrong often enough to mislead you. Count for real.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def count(text: str) -> int:
    return len(enc.encode(text))

segments = {
    "system": count(system_prompt),
    "few_shot": count(examples_block),
    "retrieved_docs": count(rag_context),
    "history": count(conversation_history),
    "user_msg": count(user_message),
}
print(segments)

Our breakdown on a typical call: system 1,200, few-shot 6,800, retrieved docs 18,000, history 7,400, user message 600. The moment you see it laid out, the fat is obvious. Retrieved docs and few-shot examples were 73% of the prompt.

Cut retrieval first, it's usually the worst offender#

Most RAG pipelines over-retrieve. We were pulling top-12 chunks at 1,500 tokens each because someone set k=12 during a demo and nobody revisited it. We ran an eval: answer quality on our 200-question golden set was statistically identical at k=5 and k=12. Everything past the fifth chunk was dead weight.

results = vector_store.similarity_search_with_score(query, k=5)
# drop anything the reranker scores below the cliff
results = [r for r, score in results if score > 0.28]

Two changes here. Drop k from 12 to 5, and add a score floor so weak matches don't get padded in. That alone took retrieved docs from 18,000 tokens to about 6,500. Add a reranker (we use a small cross-encoder) and you can often go to k=3 without a quality hit, because the ordering gets better so you need fewer chunks.

Few-shot examples: keep two, kill the rest#

Few-shot blocks rot. People add an example every time they see a failure, and nobody ever removes one. We had eleven examples. We ran the eval dropping examples one at a time and found quality plateaued at two well-chosen ones. Going from eleven to two saved roughly 5,000 tokens per call.

The counterintuitive bit: a bad example hurts more than a missing one. Two of our eleven were actively steering the model toward a verbose format we didn't want. Removing them improved quality and cut tokens at the same time.

History: summarize the old, keep the recent verbatim#

Conversation history grows without bound. Truncating to the last N messages loses context; keeping everything is expensive. We compromised: keep the last 6 turns verbatim, replace everything older with a running summary the model maintains.

if len(history) > 6:
    old = history[:-6]
    summary = summarize(old)  # a cheap model call, cached
    history = [{"role": "system", "content": f"Earlier: {summary}"}] + history[-6:]

The summary runs on a cheaper model and gets cached, so the amortized cost is tiny. This capped history at around 1,800 tokens no matter how long the chat ran.

Prompt caching pays for the stuff you must keep#

Some context genuinely can't be cut, a long system prompt with policy rules, a stable few-shot block. For those, use provider prompt caching. Put the stable parts at the front so the cache prefix stays valid, and mark the boundary.

Cached input tokens run at a large discount versus fresh ones, often around a tenth of the price on the providers we use. The rule that matters: caches key on an exact prefix match, so anything dynamic (the user's question, retrieved docs) must go last. We reordered the prompt so system and examples sit up front and never change between calls. Cache hit rate went to 91%.

Where we landed#

Same golden-set quality score (within noise), average prompt down from 34,000 tokens to about 13,500, and with caching layered on top the effective billed input dropped further. The monthly bill went from $9,400 to roughly $3,100.

The call we'd make#

Cut in this order: over-retrieval, then stale few-shot, then unbounded history, then cache what's left. Retrieval is almost always where the waste hides, and it's the safest cut because a good reranker means fewer chunks is often better, not just cheaper. Don't optimize the system prompt first, it's the smallest slice and the one caching handles best anyway.