A single answer-quality score hides where your RAG pipeline actually breaks. Split retrieval eval from generation eval and measure each one honestly.
RAG Evaluation Metrics — Faithfulness and Context Precision
The first version of our eval was a single number: "answer quality, 0 to 5, judged by a model." It went up, it went down, and it told us nothing. When quality dropped from 4.1 to 3.6 after a chunking change, we couldn't say whether retrieval got worse or the generator started making things up. So we spent a week untangling the two, and that turned out to be the whole point.
A RAG answer is the product of two independent systems. Retrieval finds chunks. Generation writes an answer from those chunks. An aggregate score multiplies both failures together and hands you the product, which you can't factor back apart. A bad answer from perfect context is a generation bug. A confident answer from missing context is a retrieval bug. They need different fixes, so they need different measurements.
Once you accept the split, the metrics organize themselves.
Retrieval side, you're asking: did we surface the chunks that actually contain the answer? Two numbers cover most of it. Context recall is the share of relevant chunks that made it into the top-k. Context precision is how much of what we retrieved was actually on-topic, and how high up. Recall protects you from missing evidence; precision protects you from burying the good chunk under six irrelevant ones that distract the generator.
Generation side, you're asking two more things. Faithfulness is whether every claim in the answer is supported by the retrieved context, i.e. is the model grounded or is it improvising. Answer relevance is whether the answer actually addresses the question rather than being technically-true filler. Faithfulness is the one that catches hallucination, and it's the metric we watch hardest, because a fluent wrong answer is worse than an honest "I don't know."
None of this works without labels. The golden set is boring to make and it's the only part that matters. Each row is a query, the ids of the chunks that genuinely answer it, and a reference answer written by someone who knows the domain.
[
{
"query": "What is the default connection pool size in the gateway?",
"relevant_chunk_ids": ["cfg-gateway-0007", "cfg-gateway-0009"],
"reference_answer": "The gateway defaults to a pool of 20 connections, configurable via GATEWAY_POOL_SIZE."
}
]
We started with 40 rows hand-picked to cover the query types users actually send: lookups, comparisons, multi-hop, and the "not in the docs" cases where the correct answer is a refusal. Fifty good rows beat five hundred lazy ones. Label the negatives too; a pipeline that answers questions it should decline is failing quietly.
With chunk labels, retrieval metrics are plain arithmetic, no model in the loop. Here's recall@k and reciprocal rank:
def recall_at_k(retrieved_ids, relevant_ids, k):
top = retrieved_ids[:k]
hits = sum(1 for cid in relevant_ids if cid in top)
return hits / len(relevant_ids) if relevant_ids else 0.0
def reciprocal_rank(retrieved_ids, relevant_ids):
for rank, cid in enumerate(retrieved_ids, start=1):
if cid in relevant_ids:
return 1.0 / rank
return 0.0
def eval_retrieval(golden, retrieve_fn, k=5):
recalls, rrs = [], []
for row in golden:
got = retrieve_fn(row["query"])
recalls.append(recall_at_k(got, row["relevant_chunk_ids"], k))
rrs.append(reciprocal_rank(got, row["relevant_chunk_ids"]))
return {
"recall@k": sum(recalls) / len(recalls),
"mrr": sum(rrs) / len(rrs),
}
Mean reciprocal rank rewards putting the right chunk first, which is what the generator's attention actually favors. Recall@k tells you whether the evidence was in the window at all. When recall is high but MRR is low, your retriever finds the answer but ranks it poorly, so a reranker is your cheapest win.
Faithfulness has no ground-truth string to diff against, so we use an LLM as a judge. The trick is to make the judgment mechanical: extract the claims, check each against the context, and score the fraction supported. Don't ask for a vibe.
You are checking whether an answer is grounded in the provided context.
CONTEXT:
{context}
ANSWER:
{answer}
Break the ANSWER into individual factual claims. For each claim, decide
if it is directly supported by the CONTEXT. A claim is UNSUPPORTED if it
adds facts not present in the context, even if those facts are true.
Return JSON only:
{"claims": [{"text": "...", "supported": true|false}]}
The harness computes faithfulness as supported claims over total claims, so one hallucinated sentence in a five-claim answer scores 0.8, not a blunt pass/fail. We run the judge with temperature 0 and a model at least as capable as the generator, and we sanity-check the judge against a handful of human labels every few weeks so it doesn't quietly drift.
def faithfulness(context, answer, judge_fn):
verdict = judge_fn(context=context, answer=answer) # returns parsed JSON
claims = verdict["claims"]
if not claims:
return 1.0
supported = sum(1 for c in claims if c["supported"])
return supported / len(claims)
def run_eval(golden, retrieve_fn, generate_fn, judge_fn, k=5):
rows = []
for row in golden:
chunks = retrieve_fn(row["query"])
context = fetch_text(chunks[:k])
answer = generate_fn(row["query"], context)
rows.append({
"type": row.get("type", "general"),
"recall@k": recall_at_k(chunks, row["relevant_chunk_ids"], k),
"faithfulness": faithfulness(context, answer, judge_fn),
})
return rows
This is the RAGAS-shaped pattern: a golden set in, per-row metrics out, one deterministic retrieval score and one judged generation score per query. Keep the raw rows around, not just the means.
We wired the harness into a nightly job and a pre-merge gate. The gate is a floor, not a target: recall@5 above 0.85, mean faithfulness above 0.9, and no single query type below 0.75. A PR that improves the average while tanking multi-hop queries gets caught by that last clause.
Slicing by query type is where the signal lives. Averaging over everything told us the pipeline was "fine." Slicing showed lookups at 0.95 faithfulness and multi-hop at 0.68, which is a completely different bug than a uniform sag. This kind of per-slice discipline is the backbone of retrieval-augmented generation that you can actually trust in production.
Never ship a RAG change against a single quality score. Split it: deterministic retrieval metrics that need no model, and a judged faithfulness score with the claim-extraction prompt doing the heavy lifting. Build fifty honest golden rows before you build anything clever, gate on per-slice floors in CI, and keep the raw per-query rows so a regression tells you which half of the pipeline to open up. It's more plumbing than a magic number, and it's the difference between knowing your system works and hoping it does.
