When RAG answers go sideways, the model usually isn't the problem. Here's the top-to-bottom checklist we run to find where retrieval actually breaks.
Debugging RAG Retrieval — Why It Returns Garbage
The user asks a question, the answer comes back wrong, and the first instinct is to blame the model. So people swap GPT-4 for something bigger, rewrite the system prompt, and nothing changes. The reason is simple: if the wrong chunks reach the model, no amount of prompt tuning fixes the output. Retrieval failures are quiet. They don't throw. They just hand the model bad context and let it improvise.
The fix is to stop guessing and walk a checklist. Retrieval has a small number of failure points, and they stack in a predictable order. Go top to bottom, prove each layer works before moving down, and the bug usually surfaces in ten minutes instead of an afternoon.
Before anything else: is the answer even in your data? I have watched teams tune similarity thresholds for a week over a fact that was never ingested. Grep the raw source for a keyword you'd expect in the answer. If it isn't there, retrieval is working perfectly and your ingestion pipeline is the bug.
If the text is present but retrieval still misses it, the next suspect is chunking. A 300-token chunk window loves to split an answer across a boundary. The definition lands in chunk 7, the example that makes it findable lands in chunk 8, and neither chunk alone scores high enough to surface. Pull the raw chunks around your answer and read them like a human. If the answer is fragmented, widen the window or add overlap before touching anything else.
The embedding model has to actually understand your domain. General-purpose embeddings trained on web text do fine on plain prose and fall apart on legal clauses, medical shorthand, or internal SKU codes. If your corpus is full of jargon the model never saw, semantically related passages land far apart in vector space. A quick tell: embed two passages you know are about the same thing and check their similarity. If it's low, the model is wrong for the domain.
Then there's asymmetry. A query is five words. The matching document is four hundred. Those live in different regions of the vector space even when they mean the same thing, and short-to-long matching is where a lot of recall quietly dies. Two cheap fixes: expand the query with a few likely synonyms, or use HyDE, where you have the LLM draft a hypothetical answer and embed that instead. A fake answer sits much closer to the real document than the terse question does.
Distance metric and normalization bugs are the silent killers. If your vectors aren't normalized and you're using cosine similarity, or you mixed cosine and dot-product between indexing and query time, the scores are noise. Confirm the metric matches on both sides and that vectors are unit length if the metric assumes it.
Metadata filters are the next trap. A source_type = 'faq' filter looks harmless until the ingestion job wrote that field as 'FAQ', and now every candidate gets excluded before scoring even runs. Filters fail closed and silent. Temporarily strip every filter and see if the right chunk reappears.
Index staleness bites after updates. You reindexed the documents but the vector store still serves the old build, or the upsert half-completed. If a freshly added document never retrieves, query the store directly by its ID and confirm the vector is actually there.
Finally, top-k. If you retrieve the top 3 and the right chunk sits at rank 6, you'll never see it. Bump k to 20 during debugging so you can tell "not retrieved at all" from "retrieved but ranked too low." Those are different bugs with different fixes.
Most of this collapses into one habit: dump what retrieval returned, with scores, every time. Stop reasoning about the vector store in the abstract and read what it handed back.
def debug_retrieve(query, k=20):
results = store.search(query, k=k)
print(f"query: {query!r}\n")
for rank, r in enumerate(results, 1):
preview = r.text[:120].replace("\n", " ")
print(f"[{rank:2d}] score={r.score:.4f} id={r.id}")
print(f" source={r.metadata.get('source')} "
f"type={r.metadata.get('source_type')}")
print(f" {preview}...\n")
return results
Run that against a query you know the answer to. If the correct chunk is nowhere in twenty results, the problem is upstream: ingestion, chunking, or embedding. If it's there at rank 12 with a low score, the problem is ranking: metric, asymmetry, or reranking. The score distribution tells you which. Scores clustered tight around one value mean the embeddings aren't separating anything, which points back at model fit.
Once you have a handful of known-answer queries, turn them into a recall check so a regression can't sneak back in.
def recall_at_k(cases, k=10):
hits = 0
for c in cases:
ids = {r.id for r in store.search(c["query"], k=k)}
if c["expected_id"] in ids:
hits += 1
else:
print(f"MISS: {c['query']!r} -> expected {c['expected_id']}")
print(f"recall@{k}: {hits}/{len(cases)} = {hits/len(cases):.2%}")
# cases = [{"query": "how do refunds work", "expected_id": "doc_412#c3"}, ...]
Twenty labeled cases catch more than a fancy eval harness you never finish building. Keep them in the repo and run them on every ingestion or embedding change.
The repeatable loop is short: log the query, dump the retrieved chunks with their scores, compare against the chunk you expected. If the expected chunk is missing, walk up the checklist from corpus to embedding. If it's present but ranked low, walk down from metric to reranking. Fix one layer, rerun the recall check, repeat. This same discipline is the foundation of retrieval-augmented generation that people actually trust, because retrieval quality caps every answer above it.
Instrument retrieval before you tune anything. A logger that prints the top-20 chunks with scores and metadata, plus twenty labeled recall cases, catches nearly every "RAG returns garbage" report faster than swapping models ever will. Build those two things first, and treat every bad answer as a retrieval bug until the dump proves otherwise.
