A RAG system that invents facts erodes trust fast. Here is how we ground answers, force citations, and catch the fabrications before users do.
Hallucination Detection — Grounding and Citations for RAG
The first hallucination that hurt us in production was a confident, well-formatted answer about a refund policy that did not exist. Retrieval had pulled the right documents. The model read them and then wrote something adjacent to the truth but not in the text. Nobody caught it until a support agent quoted it back to a customer.
In a RAG setup, a hallucination has a narrower, more testable meaning than "the model made something up." It means the answer contains a claim not supported by the retrieved context. That definition is useful because it is checkable: you have the context, you have the answer, and you can ask whether one entails the other. You do not need ground truth about the world. You need agreement between two strings you already hold.
Most fabrications start with a prompt that invites them. If you ask "answer the question" and paste some documents, the model treats the documents as a hint and its parametric memory as the rest. The fix is to move the boundary explicitly.
Three instructions carry most of the weight. Answer only from the provided context. Cite the source of every claim. Refuse when the context does not contain the answer. That refusal path is the part people skip, and it is the part that stops the worst failures.
You answer questions using ONLY the numbered context passages below.
Rules:
- Every factual sentence must be supported by at least one passage.
- After each sentence, cite the passage id(s) in brackets, e.g. [3].
- If the context does not contain enough information to answer,
reply exactly: "I don't have enough information to answer that."
- Do not use outside knowledge. Do not guess.
Context:
[1] {chunk_1}
[2] {chunk_2}
[3] {chunk_3}
Question: {question}
The refusal string is deliberately fixed. A literal marker is easy to detect downstream, route to a human, or count in metrics. If you let the model phrase its own uncertainty, you get twelve variants and none of them are greppable.
We wrote more about how these pieces fit into a reliable pipeline in our notes on retrieval-augmented generation.
Bracketed passage ids are a start. They let a reader jump to the source, and they give you a cheap signal: an answer sentence with no citation is a candidate hallucination. But passage-level citations still let the model cite chunk 3 while paraphrasing something chunk 3 never said.
The stronger pattern is span attribution. Ask the model to return, for each claim, the exact substring of the context it relied on. Then you can verify that the substring actually appears in the chunk.
def verify_spans(answer_claims: list[dict], chunks: dict[int, str]) -> list[dict]:
"""Each claim: {"text": ..., "chunk_id": int, "quote": str}."""
results = []
for claim in answer_claims:
source = chunks.get(claim["chunk_id"], "")
supported = claim["quote"].strip() in source
results.append({
"claim": claim["text"],
"chunk_id": claim["chunk_id"],
"span_found": supported,
})
return results
Exact-match on the quote is strict and it should be. If the model cannot point at a literal span, that is worth flagging even when the claim happens to be true. In the UI we highlight the quoted span inside the source chunk so a user can confirm attribution in one glance.
String overlap is the crudest signal and still catches a surprising amount. If an answer sentence shares almost no tokens with any retrieved chunk, it probably came from somewhere else. It is a filter, not a verdict.
The check we trust more is entailment. Treat each answer sentence as a hypothesis and the retrieved context as the premise, then ask whether the premise supports the hypothesis. A natural-language-inference model gives you entailment / neutral / contradiction per sentence. Anything that is not entailment gets a second look.
When we do not want to host an NLI model, we use an LLM as a faithfulness judge. Keep the judge prompt narrow: it grades support, not helpfulness or style.
JUDGE_PROMPT = """You are checking whether an ANSWER is fully supported
by the CONTEXT. Do not judge whether the answer is helpful or correct in
general; only whether every claim is backed by the context.
CONTEXT:
{context}
ANSWER:
{answer}
Return JSON:
{{"supported": true|false,
"unsupported_claims": ["..."],
"verdict": "one sentence explaining the decision"}}"""
def faithfulness_check(client, context: str, answer: str) -> dict:
resp = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
messages=[{
"role": "user",
"content": JUDGE_PROMPT.format(context=context, answer=answer),
}],
)
return json.loads(resp.content[0].text)
Two things matter here. Run the judge at temperature 0 so the verdict is stable across reruns. And feed it the same context the answering model saw, not a fresh retrieval, or you are grading a different question.
Confidence signals help as a cheap prefilter. Low token probabilities on named entities and numbers correlate with fabrication, and answers where the model hedged in its own words are worth routing to the judge. None of these are proof on their own. Stack them: overlap flags a candidate, entailment or the judge confirms it.
You cannot improve a number you do not track. Build a golden set of maybe 100 to 200 question-and-context pairs, each labelled by a human as answerable or not, with the supported answer written out. Run your pipeline over it and count how often the generated answer contains an unsupported claim. That fraction is your hallucination rate.
Watch two failure modes separately. On answerable questions, measure faithfulness: did the answer stay inside the context. On unanswerable questions, measure the refusal rate: did the system correctly say it did not know instead of inventing something. A system that never refuses will look great on the first metric and fail every user who asks about something you do not cover.
Re-run the set on every prompt change and every model swap. We caught a regression once where a new model version was more fluent and more willing to fill gaps: faithfulness on the golden set dropped four points before anything shipped.
Ground first, detect second. A citation-enforcing prompt with a hard refusal string removes more hallucinations than any downstream classifier, and it costs you one prompt rewrite. Then add span verification because it is cheap and deterministic, and an entailment or judge check on the sentences that fail the cheap checks. Track faithfulness and refusal separately on a golden set you actually maintain. Skip the fancy confidence math until the basics are in place: most teams do not have a detection problem, they have a prompt that quietly invited the model to guess.
