A million-token window doesn't retire your retrieval stack. Here's when to stuff the prompt, when to retrieve, and when to do both.
Long Context vs RAG — When to Use Which
Every few months someone declares RAG dead because a new model shipped a bigger context window. Then the invoice arrives, and the same team quietly puts retrieval back in. I've watched this cycle three times now. The framing is wrong: long context and retrieval-augmented generation aren't competitors. One decides what the model reads, the other decides how much reasoning room it gets over that material. Most systems I run in production use both.
A large window lets the model hold a whole document set in working memory at once. No chunking boundaries splitting an argument in half, no retrieval step that misses the paragraph that mattered. For a single 200-page contract or one long codebase module, stuffing everything in is genuinely simpler and often more accurate than building a pipeline.
The limits show up fast at scale.
Cost scales linearly with input tokens, every single call. That 900K-token prompt isn't a one-time load, you pay for it on every question. Prompt caching softens repeat reads but doesn't make them free.
Latency scales too. Time-to-first-token climbs with input size because the model has to process the whole prefix before it emits anything. Users feel a full-context prompt.
Then there's lost-in-the-middle. Models attend well to the start and end of a long input and get lazy about the middle. Push 500K tokens of retrieved noise at a model and the fact you needed on page 180 may as well not be there. Recall on a needle buried mid-context degrades measurably as the window fills.
And a stuffed prompt gives you no freshness and no citations. The model can't tell you which of the 40 documents an answer came from, and anything past its cutoff is invisible unless you inject it yourself.
Retrieval flips the default. Instead of sending everything, you send the handful of chunks that match the query. That single move fixes most of the above:
- Only relevant material reaches the model, so context stays small and the lost-in-the-middle problem mostly disappears.
- Citations come for free. You retrieved chunk #47 from
policy_2026.pdf, so you can show it.
- Freshness is a re-index away. Update the vector store and the next query sees new facts without touching the model.
- Cost is bounded by your top-k, not your corpus. A 10-million-token knowledge base costs the same per query as a 10-thousand-token one.
- It scales past any window. No model holds your entire wiki, but a retriever indexes all of it.
The tradeoff is that retrieval can miss. If your embedding model doesn't surface the right chunk, the LLM never sees it. That failure mode is why getting retrieval-augmented generation reliable is its own discipline.
Run each new use case through five questions:
- Corpus size. Fits in a window with room to spare? Long context is fine. Larger than any window, or growing? You need retrieval.
- Freshness. Static reference material tolerates a stuffed prompt. Data that changes daily wants an index you can update independently of the model.
- Citations. If users must trace answers to sources (legal, medical, compliance), retrieval gives you provenance that a giant prompt can't.
- Cost and latency budget. High query volume punishes big prompts. Retrieval keeps per-call tokens flat.
- Query patterns. Broad "summarize this whole document" questions favor full context. Narrow "what does clause 12 say" lookups favor retrieval.
The best systems combine the two.
Retrieve wide, reason long. Pull the top 30 chunks instead of the top 3, then let a long-context model reason over all of them at once. You get retrieval's precision on what enters the prompt and long context's headroom for synthesis across sources.
Cache the stable prefix. If a system prompt, schema, or reference doc stays constant across calls, put it at the front and cache it. You pay full price once, then a fraction on every reuse.
# stable instructions cached; only the query + retrieved chunks are fresh
messages = [
{"role": "system", "content": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}},
{"role": "user", "content": f"{retrieved_chunks}\n\nQuestion: {user_query}"},
]
Say you run a support assistant, 100K queries a month, over a 2M-token knowledge base. Assume input at $3 per million tokens.
Stuff everything (long context only):
2,000,000 tokens x $3 / 1,000,000 = $6.00 per query
$6.00 x 100,000 queries = $600,000 / month
That's before you hit the fact that 2M tokens exceeds most windows, and before latency makes it unusable.
Retrieve top-8 chunks (RAG):
8 chunks x ~500 tokens = 4,000 tokens
+ 1,000 tokens prompt/query overhead
= 5,000 tokens x $3 / 1,000,000 = $0.015 per query
$0.015 x 100,000 queries = $1,500 / month
Same model, same corpus, roughly 400x cheaper, and it fits comfortably in any window with room left to reason. Add embedding and vector-store costs and RAG is still a rounding error against the stuffed prompt.
The point isn't that retrieval always wins. For that occasional deep-dive over one 300K-token document, skip the pipeline and stuff it. The point is that the per-query economics decide the architecture, and at any real volume they decide it hard.
Default to retrieval for anything that scales: large corpora, high query volume, freshness needs, or citation requirements. Reach for full context when the material is small, static, and the questions are broad. And when you want both precision and reasoning room, retrieve to select and use the long window to think. Treat the window size as headroom for your retriever's output, not as a reason to delete the retriever.
