RAG vs Fine-Tuning — Picking the Right Tool, Honestly

RAG vs Fine-Tuning — Picking the Right Tool, Honestly#

The discussion online flattens this into a debate: "RAG vs fine-tuning, which wins?" Wrong framing. They solve different problems. Pick the wrong one and you spend three months building something that doesn't address your actual issue.

After running both in production for ~18 months, here's the framework we use.

What each one actually does#

RAG (Retrieval-Augmented Generation). At inference time, fetch relevant text from a knowledge base, stuff it into the prompt, ask the model to answer using that context. The model itself doesn't change. The knowledge is in the retrieval system.

Fine-tuning. Take a base model and continue training it on examples of input → output. The model's weights change. New behavior is baked in.

The core distinction:

• RAG = knowledge. "Here are the facts; answer the question using them."
• Fine-tuning = behavior. "Here's how you should answer; learn the pattern."

This is the single most important distinction; everything else falls out of it.

When RAG is the right answer#

The problem requires the model to know facts it doesn't know:

• Your company's documentation, internal policies, customer history.
• Recent news or events (post-cutoff).
• Domain-specific data (medical records, legal docs, product catalogs).

The signals that RAG fits:

• The "knowledge" changes frequently. Documentation gets updated weekly. A RAG-indexed corpus updates in real time; a fine-tuned model freezes its knowledge at training time.
• You can articulate the answer source. "The answer should come from this corpus." RAG returns citations naturally.
• The base model is already capable; it's just missing data.
• You need to debug "where did this answer come from?" — RAG shows the source documents.

We use RAG for: customer support (answers come from product docs + tickets), internal Q&A (answers from company wiki), compliance lookup (answers from policy documents).

When fine-tuning is the right answer#

The problem requires the model to behave differently:

• Specific output format (always JSON in this schema).
• Specific tone or style (terse, formal, friendly).
• Specific task pattern the base model is bad at (classifying with a custom taxonomy, translating to a low-resource language, generating SQL in a specific dialect).

The signals that fine-tuning fits:

• The "behavior" is stable. The taxonomy doesn't change weekly.
• Prompt engineering hits a ceiling. You've tried multi-shot examples + careful instructions; quality is still inconsistent.
• The task is fundamentally pattern-recognition, not knowledge retrieval.
• Latency or token cost matters. Fine-tuned models can use shorter prompts; cheaper per call.

We use fine-tuning for: classification of support tickets into our internal taxonomy, generation of structured extracts from documents, translation to a domain-specific style.

The hybrid pattern (what we actually run)#

In most production AI systems we've built, we use both. RAG for the knowledge, fine-tuning for the behavior.

Example: a support agent.

• RAG: retrieves relevant past tickets, product documentation, customer history. Injects into the prompt.
• Fine-tune: the model has been fine-tuned to respond in our support tone, escalate appropriately, format responses for the support tooling.

Each solves a problem the other can't. RAG can't make the model talk like our support team; fine-tuning can't keep up with weekly product changes.

The cost of each#

RAG operational cost:

• Embedding the corpus (one-time + on-update).
• Vector DB hosting (ongoing).
• Per-query retrieval (sub-100ms at our scale).
• Larger prompts → higher per-call inference cost.

Fine-tuning operational cost:

• Training data preparation (often the biggest hidden cost).
• Training compute (GPU-hours).
• Model hosting (if self-hosted) or per-token premium (if provider-hosted).
• Re-training cadence (every time you change behavior).

For an MVP, RAG is cheaper to start. Build the retrieval, point at the base model, ship. Fine-tuning has more upfront cost (data prep, training runs).

For sustained operation, fine-tuning can be cheaper. A fine-tuned model that uses 500-token prompts instead of 5000-token RAG prompts is 10x cheaper per call.

What we tried and dropped#

Fine-tuning to inject knowledge. Fine-tuned a model on our documentation. Worked OK on what was in the training set. New docs didn't help. The model "knew" what it knew at training time, period. Fine-tuning is not the right tool for knowledge.

RAG for behavior. Tried to push the model toward a specific output format by including format examples in the retrieved context. Worked sometimes; failed inconsistently. The model would format correctly when retrieval was clean and badly when retrieval included noisy context. Fine-tuning to bake in the format worked far better.

Massive context windows in lieu of retrieval. When context windows expanded to 1M tokens, the temptation was to stuff the entire knowledge base in the prompt. Two issues: cost (1M tokens per query is expensive) and quality ("needle in haystack" — models still struggle to use deeply-buried context). RAG with targeted retrieval beats raw context stuffing for accuracy.

Decision criteria, distilled#

Ask in order:

1. Is this a knowledge problem (model doesn't know facts) or a behavior problem (model can't act the way I need)? If knowledge → RAG. If behavior → fine-tuning. If both → both.

2. Does the data change frequently? If yes → RAG (fine-tuning freezes knowledge). If stable → either works.

3. Can I produce 1000+ high-quality training examples? If no, fine-tuning is unlikely to work well. RAG doesn't need labeled examples.

4. What's the inference cost / latency budget? Tight → fine-tuning (shorter prompts). Looser → RAG (longer prompts OK).

5. Do I need citations? If yes → RAG returns sources naturally; fine-tuned outputs don't.

The retrieval quality problem#

The biggest practical issue with RAG: retrieval quality is the ceiling. If retrieval brings the wrong documents, the model answers wrong. Time spent improving retrieval (better embeddings, hybrid search, re-ranking, query rewriting) usually beats time spent on model selection.

Pattern we use:

• Hybrid search (BM25 + dense embeddings).
• Re-ranking with a cross-encoder model on the top-50 candidates.
• Query rewriting (LLM rewrites the user's question into a better retrieval query).

Each adds latency and cost; each meaningfully improves quality. The cost-quality tradeoff is workload-specific.

Things that surprised us#

Fine-tuning small models is underused. A small fine-tuned model often outperforms a big base model on a narrow task. We have a fine-tuned 7B model that beats Sonnet at our specific classification task at 1/20 the cost.

RAG quality plateaus faster than you'd think. First 80% of RAG quality is "embed and retrieve." Next 15% takes 10x the effort (re-ranking, query rewriting, careful chunking). Last 5% is hand-tuning that may never converge.

Most "RAG vs fine-tuning" debates online conflate them. Articles compare a barely-tuned RAG against a well-tuned fine-tune (or vice versa) and conclude their preferred approach wins. The fair comparison requires investment in both.

What we monitor#

• RAG retrieval recall@k. What fraction of queries pull the actually-relevant doc in the top k? Drives quality.
• Fine-tuned model eval pass rate. How often does the model produce correct output? Compared against the base model.
• Cost per request, by approach. Sometimes the cheaper-feeling approach isn't.
• Latency p95. RAG adds retrieval latency; fine-tuned models may add inference latency.

What to read next#

• Hybrid search — BM25 + embeddings for RAG — the retrieval mechanism
• LLM evals that actually predict production quality — how to know if either is working
• LLM cost optimization in production — keeping the bill in line
• Embeddings drift detection — when "similar" stops — the long-term hazard of RAG indexes

RAG and fine-tuning are different tools. Pick by the problem shape; don't pick by what's trending. The hybrid approach is what most serious production AI systems converge to. The framework above is the one we use to decide where each one fits — we don't pretend it's the only one.