RAG Retrieval Evaluation: Building an Offline Eval Harness Before You Ship

Evaluate RAG retrieval offline with a labeled golden set and two deterministic metrics, recall@k and MRR, computed with no model in the loop. Score generation separately with claim-level faithfulness. Run both in CI on every change to embeddings, chunking, or reranking, slice the results by query type, and block merges that drop any slice below its floor.

Split retrieval evaluation from generation evaluation

Our first eval was a single model-judged quality score. When it dropped after a chunking change, we could not tell which half had broken. A single score multiplies two failure modes together and hands you a product you cannot factor back apart. A bad answer built from perfect context is a generation bug. A confident answer built from missing context is a retrieval bug. They need different fixes, so they need different measurements.

1. Retrieval eval: given a query, did we fetch the chunks that contain the answer? Recall@k and MRR. Fast, deterministic, no LLM call.
2. Generation eval: given the retrieved chunks, is every claim in the answer supported by them, and does the answer address the question? Faithfulness and answer relevance. Slower, needs an LLM judge.

Most iteration happens in the first harness, because it runs in milliseconds.

Build a golden set before anything clever

Each row is a query, the ids of the chunks that genuinely answer it, and a reference answer written by someone who knows the domain. The reference answer is what lets you score context recall at the claim level later, the way Ragas does.

• Mine production logs. Real user queries beat invented ones. Sample them.
• Annotate the answer location. For each query, mark which chunk contains the answer. This is the tedious part, and a small honest set beats a large sloppy one.
• Cover the failure modes you've seen. Acronyms, multi-hop questions, near-duplicate documents, time-sensitive queries.
• Label the negatives. Include "not in the docs" queries where the correct answer is a refusal. A pipeline that confidently answers questions it should decline is failing quietly.

golden = [
    {"query": "how do I rotate the signing key",
     "relevant_chunk_ids": ["doc_42#3", "doc_42#4"],
     "reference_answer": "Rotate it in the admin console, then redeploy the verifier.",
     "slice": "lookup"},
    {"query": "what's the default retention period",
     "relevant_chunk_ids": ["doc_17#1"],
     "reference_answer": "Logs are kept until a retention policy is set.",
     "slice": "lookup"},
    # ...
]

Measure retrieval with no model in the loop

def recall_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    hits = len(set(top_k) & set(relevant_ids))
    return hits / len(relevant_ids) if relevant_ids else 0.0

def reciprocal_rank(retrieved_ids, relevant_ids):
    for i, rid in enumerate(retrieved_ids, start=1):
        if rid in relevant_ids:
            return 1.0 / i
    return 0.0

• Recall@k answers "is the answer even in the context we'll send the model?" Whatever fraction of queries miss, the model cannot answer correctly no matter how good it is. This is the single most important retrieval number.
• MRR (mean reciprocal rank) answers "how high up is the right chunk?" Position matters because chunks buried in the middle of a long context get used less.

When recall is high but MRR is low, the retriever finds the answer but ranks it poorly, and a cross-encoder reranker is usually the cheapest fix.

Judge faithfulness, but make the judgment mechanical

Faithfulness has no ground-truth string to diff against, so an LLM has to judge it. The trick is to extract individual claims and check each one against the context instead of asking the judge for an overall grade. Ragas defines faithfulness exactly this way: the number of claims supported by the retrieved context divided by the total number of claims, on a 0 to 1 scale.

You are checking whether an answer is grounded in the provided context.

CONTEXT: {context}
ANSWER: {answer}

Break the ANSWER into individual factual claims. For each claim, decide
if it is directly supported by the CONTEXT. A claim is UNSUPPORTED if it
adds facts not present in the CONTEXT, even if those facts are true.

Return JSON only: {"claims": [{"text": "...", "supported": true|false}]}

One hallucinated sentence in a two-claim answer scores 0.5 instead of a blunt pass or fail, and the per-claim JSON tells you which sentence to look at. Run the judge at temperature zero with a model at least as capable as the generator, and check it against a handful of human labels on a regular schedule so it does not drift. This is the same discipline that makes production RAG reliable instead of merely demo-ready.

Run it on every change, sliced by query type

def evaluate(retriever, golden, k=8):
    by_slice = defaultdict(lambda: {"recall": [], "rr": []})
    for case in golden:
        ids = retriever.search(case["query"], k=k)
        s = by_slice[case["slice"]]
        s["recall"].append(recall_at_k(ids, case["relevant_chunk_ids"], k))
        s["rr"].append(reciprocal_rank(ids, case["relevant_chunk_ids"]))
    return {name: {"recall@k": mean(v["recall"]), "mrr": mean(v["rr"])}
            for name, v in by_slice.items()}

We run the retrieval harness on every pull request that touches the retrieval stack and the faithfulness harness nightly. The gate is a floor per slice, not a target on the average. A change that improves the aggregate while sinking multi-hop queries is a regression for your hardest users, and the per-slice floor is the clause that catches it.

What slicing caught that spot checks missed:

• An embedding upgrade that improved average recall but hurt acronym queries specifically. Invisible in spot checks, obvious once metrics were broken down by query type.
• A chunking change to bigger chunks that raised recall but pushed the relevant text to the bottom of each chunk, lowering faithfulness downstream. Retrieval metrics alone would have shipped it; the generation harness flagged it. Our notes on chunking strategies cover the tradeoff.
• A reranker that improved MRR but added enough latency to make the gain a real tradeoff. Now it is a measured decision, not a guess.

Keep the raw per-query rows from every run. When a slice drops, the diff of those rows tells you which queries flipped and which half of the pipeline to open up.

Questions people ask

What is the difference between recall@k and MRR in RAG evaluation?

Recall@k is the share of relevant chunks that appear in the top k results, so it tells you whether the evidence reached the model at all. MRR averages one over the rank of the first relevant chunk, so it tells you how high that evidence sits. High recall with low MRR means the retriever finds the answer but ranks it badly, which a reranker usually fixes.

How do you measure faithfulness in a RAG pipeline?

Split the answer into individual factual claims, then ask an LLM judge whether each claim is supported by the retrieved context. Faithfulness is supported claims divided by total claims, between 0 and 1. Run the judge at temperature zero, use a model at least as capable as the generator, and spot-check it against human labels regularly.

How big should a RAG golden set be?

Big enough to cover every query type your users send, with several rows per slice, and no bigger than you can label honestly. A few dozen carefully labeled queries with chunk ids and reference answers are more useful than thousands of unlabeled ones. Include refusal cases, and grow the set from production failures as they surface.

The decision, concretely

• Shipping a chunking, embedding, or reranking change? Gate on recall@k and MRR per slice; a faithfulness dip afterward usually traces back to retrieval.
• Suspecting hallucination? Gate on claim-level faithfulness, not a single "is this answer good" score that conflates grounding with fluency.
• Building your first eval? Spend the first week on honest golden rows, not on harness tooling.

The call we'd make

Never ship a RAG change against a single quality score. Split it: deterministic retrieval metrics with no model in the loop, and a judged faithfulness score where the claim-extraction prompt does the real work. Label the golden set first, gate CI on per-slice floors, and keep per-query rows so every regression points at a specific half of the pipeline.

Sources

• Ragas docs: Faithfulness
• Ragas docs: Context Recall