RAG Evaluation: Split Retrieval From Generation, or You're Debugging by Vibe

The first version of our RAG evaluation was a single number: answer quality, zero to five, judged by a model. It went up, it went down, and it told us nothing useful. When the score dropped from 4.1 to 3.6 after a chunking change, we could not say whether retrieval had gotten worse or the generator had started making things up. Untangling the two took a week, and that turned out to be the whole point.

A RAG answer is the product of two independent systems. Retrieval finds chunks; generation writes an answer from those chunks. An aggregate score multiplies both failure modes together and hands you the product, which you cannot factor back apart. A bad answer built from perfect context is a generation bug. A confident answer built from missing context is a retrieval bug. They need different fixes, so they need different measurements.

Split the eval in two#

Once you accept the split, the metrics organize themselves.

On the retrieval side, the question is whether you surfaced the chunks that actually contain the answer. Two numbers cover most of it. Context recall is the share of relevant chunks that made it into the top-k. Context precision is how much of what you retrieved was actually on-topic, and how high up. Recall protects you from missing evidence; precision protects you from burying the right chunk under six irrelevant ones that distract the generator.

On the generation side, two more questions. Faithfulness is whether every claim in the answer is supported by the retrieved context, meaning the model is grounded rather than improvising. Answer relevance is whether the answer actually addresses the question rather than being technically-true filler. Faithfulness is the one that catches hallucination, and it is the metric worth watching hardest, because a fluent wrong answer is worse than an honest "I don't know."

Build a golden set before anything else#

None of this works without labels, and the golden set is the boring part that matters most. Each row is a query, the ids of the chunks that genuinely answer it, and a reference answer written by someone who knows the domain.

[
  {
    "query": "What is the default connection pool size in the gateway?",
    "relevant_chunk_ids": ["cfg-gateway-0007", "cfg-gateway-0009"],
    "reference_answer": "The gateway defaults to a pool of 20 connections, configurable via GATEWAY_POOL_SIZE."
  }
]

Start with 40 to 50 rows hand-picked to cover the query types users actually send: lookups, comparisons, multi-hop questions, and the "not in the docs" cases where the correct answer is a refusal. Fifty good rows beat five hundred lazy ones. Label the negatives too; a pipeline that confidently answers questions it should decline is failing quietly.

Measure retrieval with no model in the loop#

With chunk labels in hand, retrieval metrics are plain arithmetic:

def recall_at_k(retrieved_ids, relevant_ids, k):
    top = retrieved_ids[:k]
    hits = sum(1 for cid in relevant_ids if cid in top)
    return hits / len(relevant_ids) if relevant_ids else 0.0

def reciprocal_rank(retrieved_ids, relevant_ids):
    for rank, cid in enumerate(retrieved_ids, start=1):
        if cid in relevant_ids:
            return 1.0 / rank
    return 0.0

Mean reciprocal rank rewards putting the right chunk first, which is what the generator's attention actually favors. Recall at k tells you whether the evidence was in the window at all. When recall is high but MRR is low, the retriever finds the answer but ranks it poorly, and a reranker is the cheapest fix available.

Judge faithfulness, but make the judgment mechanical#

Faithfulness has no ground-truth string to diff against, so an LLM has to judge it. The trick is extracting individual claims and checking each one against the context, rather than asking the judge for a vibe:

You are checking whether an answer is grounded in the provided context.

CONTEXT: {context}
ANSWER: {answer}

Break the ANSWER into individual factual claims. For each claim, decide
if it is directly supported by the CONTEXT. A claim is UNSUPPORTED if it
adds facts not present in the context, even if those facts are true.

Return JSON only: {"claims": [{"text": "...", "supported": true|false}]}

Faithfulness becomes supported claims over total claims, so one hallucinated sentence in a five-claim answer scores 0.8 rather than a blunt pass or fail. Run the judge at temperature zero with a model at least as capable as the generator, and sanity-check it against a handful of human labels every few weeks so it does not quietly drift. This is the same shape of discipline that makes production RAG reliable rather than merely demo-ready.

Run it in CI, sliced by query type#

Wire the harness into a nightly job and a pre-merge gate. The gate is a floor, not a target: recall at 5 above 0.85, mean faithfulness above 0.9, no single query type below 0.75. A change that improves the average while tanking multi-hop queries gets caught by that last clause specifically.

Slicing by query type is where the real signal lives. Averaging over everything says the pipeline is "fine." Slicing showed lookups at 0.95 faithfulness and multi-hop at 0.68, which is an entirely different bug from a uniform sag, and it points straight at chunking or reranking rather than the generator.

The decision, concretely#

• Shipping a chunking or reranking change? Gate on context recall and precision specifically; a faithfulness dip afterward almost always traces back to retrieval, not the generator.
• Suspecting hallucination? Gate on faithfulness with the claim-extraction prompt, not a single "is this answer good" score that conflates grounding with fluency.
• Building your first eval? Fifty honest golden rows beats a large sloppy set. Spend the first week there, not on harness tooling.
• Already have an aggregate score in production? Keep it as a headline number if stakeholders expect one, but never gate a deploy on it alone.

The call we'd make#

Never ship a RAG change against a single quality score. Split it: deterministic retrieval metrics that need no model in the loop, and a judged faithfulness score where the claim-extraction prompt does the real work. Build fifty honest golden rows before anything clever, gate on per-slice floors in CI, and keep the raw per-query rows so a regression tells you which half of the pipeline to open up. It is more plumbing than a single magic number, and it is the difference between knowing your system works and hoping it does.