RAG vs Fine-Tuning: Picking the Right Tool, Honestly

Use RAG when the model lacks knowledge (your docs, recent data, anything that changes) and fine-tuning when it lacks a behavior (a strict output format, a tone, a narrow classification task). RAG changes what the model sees at query time; fine-tuning changes the weights. Picking one to fix the other's problem is the most expensive mistake in this space.

What each one actually does

RAG (Retrieval-Augmented Generation). At inference time you fetch relevant text from a knowledge base, put it in the prompt, and ask the model to answer from that context. The model is untouched. The term comes from Lewis et al. (2020), who paired a generator with a dense retriever and got more factual answers than the model alone.

Fine-tuning. You continue training a base model on input and output pairs, so the weights change and a new behavior is baked in.

The core distinction:

• RAG is knowledge. "Here are the facts; answer using them."
• Fine-tuning is behavior. "Here is how you answer; learn the pattern."

When RAG is the right answer

The model needs facts it does not have: your docs, policies, customer history, or anything newer than its training cutoff.

The signals that RAG fits:

• The knowledge changes. Docs update weekly. A re-indexed corpus is current within minutes; a fine-tuned model freezes what it knew at training time.
• You can name the answer source. "The answer must come from this corpus." RAG returns the chunks it used, so citations are free.

When fine-tuning is the right answer

The model needs to behave differently: always emit JSON in one schema, hold a specific tone, classify into a custom taxonomy, or write SQL in one dialect. OpenAI's guide lists the same shape of tasks (classification, nuanced translation, a specific format, fixing instruction-following failures) plus distilling a big model into a smaller one.

The signals that fine-tuning fits:

• The behavior is stable. The taxonomy does not change weekly.
• Prompting has hit a ceiling. Few-shot examples and careful instructions still produce inconsistent output.
• Prompt length is the cost driver. A tuned model can drop the long few-shot preamble.

The data is a JSONL file of conversations. This is the shape both OpenAI and TRL accept:

# train.jsonl (one JSON object per line)
{"messages": [{"role": "system", "content": "Classify the ticket. Reply with one label."}, {"role": "user", "content": "Invoice PDF shows the wrong VAT number"}, {"role": "assistant", "content": "billing/invoice-data"}]}
{"messages": [{"role": "system", "content": "Classify the ticket. Reply with one label."}, {"role": "user", "content": "SSO login loops back to the start page"}, {"role": "assistant", "content": "auth/sso"}]}

On open weights, a LoRA adapter with TRL is a few lines. This mirrors the PEFT example in the TRL docs, pointed at your own file:

# train_lora.py
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="train.jsonl", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
    args=SFTConfig(output_dir="ticket-classifier", learning_rate=1e-4),
    peft_config=LoraConfig(),
)
trainer.train()

$ pip install trl peft datasets
$ python train_lora.py

The 1e-4 learning rate follows TRL's own tip for adapters, which train only new parameters.

The hybrid pattern most teams converge on

Most serious production systems run both: RAG for the knowledge, a tuned model for the behavior.

Take a support agent. Retrieval pulls past tickets, product docs and customer history into the prompt; the model is tuned to answer in the team's voice, escalate correctly, and format replies for the ticketing tool. Retrieval cannot make the model sound like your support team, and fine-tuning cannot keep up with weekly product changes.

Where the money goes

RAG costs: embedding the corpus and its updates, vector store hosting, retrieval latency, longer prompts.

Fine-tuning costs: data preparation (usually the biggest hidden line), training compute, hosting or a per-token premium, and a retrain whenever the behavior changes.

RAG is cheaper to start. Over a long run the math can flip, because a prompt of 500 tokens costs a tenth of a 5,000-token RAG prompt at the same per-token rate. Do that arithmetic with your provider's current prices, not ours, and see LLM cost optimization in production for the other levers.

One 2026-specific warning: OpenAI's fine-tuning guide now says the platform is being wound down, is closed to new users, and accepts training jobs from eligible existing customers only until January 6, 2027. If you are starting fresh, plan for open-weights fine-tuning (TRL, LoRA) or another provider's tuning service.

The failure modes, and what each looks like

Fine-tuning to inject knowledge. You train on your docs, then the docs change and the model confidently returns the old answer, with no way to cite or correct it short of retraining. Symptom: correct answers in eval, stale answers a month later.

RAG for behavior. Format examples pushed into retrieved context work when retrieval is clean and break when noisy chunks arrive. Symptom: format compliance that varies with the query.

Stuffing the whole corpus into a long context. You pay for every token on every call, and models use buried context poorly. Liu et al. found accuracy is highest when the relevant passage sits at the start or end of the input and drops when it is in the middle. Targeted retrieval beats raw stuffing.

Blaming the model for a retrieval bug. If retrieval brings the wrong documents, the model answers wrong, and swapping models does nothing. Measure retrieval on its own first, with an offline retrieval evaluation harness, before touching the generator.

Too few or noisy training examples. OpenAI's guide is blunt: if 50 good examples have no impact, rethink the task or prompt before adding data. A thousand inconsistent labels teach the inconsistency.

Retrieval quality is the ceiling

Time spent improving retrieval usually beats time spent on model selection. The pattern that tends to pay off:

• Hybrid search, BM25 plus dense embeddings (how hybrid search works).
• Re-ranking the top 50 candidates with a cross-encoder.
• Query rewriting, where an LLM turns the user's question into a better retrieval query.

Each adds latency and cost, and each tends to lift quality. Track drift over time, too (embeddings drift detection).

Monitor retrieval recall@k, tuned-versus-base eval pass rate (LLM evals that predict production quality), cost per request and p95 latency.

Where this framework breaks

The knowledge/behavior split is a default, not a law. A domain vocabulary (medical codes, internal product names) is partly both, and a light fine-tune plus retrieval often beats either alone. A small static corpus that fits in a prompt may need neither. Without an eval set you cannot tell whether either worked, so build that first.

Questions people ask

Is RAG better than fine-tuning?

Neither is better in general, because they fix different problems. RAG supplies facts at query time and keeps them current, with citations. Fine-tuning changes how the model responds, locking in a format, tone, or narrow task. If the model gives wrong facts, add retrieval. If it gives the right facts in the wrong shape, tune it. Many production systems use both together.

Can fine-tuning teach a model new facts?

Partly, and badly. A model tuned on your documents can repeat what was in the training set, but it cannot update when the documents change, cannot cite sources, and tends to blend facts with confidence. For knowledge that changes or must be traceable, retrieval is the right tool. Use fine-tuning for stable behavior, and let RAG carry the facts.

How many examples do you need to fine-tune an LLM?

OpenAI's supervised fine-tuning guide requires at least 10 examples and suggests starting with about 50 well-crafted ones, then evaluating. Gains often appear with 50 to 100. If 50 good examples show no improvement, the guide says to rethink the task or prompt rather than add data. Quality and consistency of labels matter more than volume.

The call we'd make

Start with RAG and a strong base model, because most "the model is wrong" complaints are knowledge gaps and retrieval fixes them without training anything. Add fine-tuning only when evals show a stable behavior problem prompting cannot fix, and prefer open-weights LoRA now that OpenAI's tuning platform is closing to new work.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401)
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
• OpenAI API docs: Supervised fine-tuning
• Hugging Face TRL docs: SFT Trainer