# Embedding Quality in RAG: How We Cut Hallucinations by 60%

URL: https://www.devopsness.com/blog/embedding-quality-in-rag-how-we-cut-hallucinations-by-60-2026-04-06
Author: Kiril Urbonas
Published: 2026-04-06
Updated: 2026-04-06
Categories: AI
Tags: Python, LLM, GPT

Summary: Six months running RAG in production taught us that the retrieval step matters far more than the model. Concrete techniques that moved the needle, with before/after numbers.

---
# Embedding Quality in RAG: How We Cut Hallucinations by 60%

We run a customer-facing RAG system over ~280k internal documents. After six months in production, we did a quarter-long quality push and brought our **answer-with-source-mismatch rate from 14.2% to 5.7%** without changing the LLM. Every win came from the retrieval side.

## The Baseline We Inherited

- Embeddings: `text-embedding-3-small` (1536-dim)
- Chunking: 512 tokens, 64-token overlap
- Vector store: pgvector with HNSW index
- Top-k: 8 chunks per query
- LLM: `gpt-4o-mini` with a vanilla "answer using only the context below" prompt

Quality eval against 1,200 hand-labeled QA pairs:
- **Exact answer present in retrieved chunks**: 76%
- **Answer correct (LLM-graded)**: 71%
- **Hallucination rate**: 14.2%

## Win #1: Re-Chunk by Semantic Boundaries (+8% recall)

Fixed-token chunking cut sentences in half. We switched to **semantic chunking** using a recursive splitter that respects markdown headings, paragraph breaks, and code fences.

```python
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

headers = [("##", "h2"), ("###", "h3")]
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
recursive = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=80,
    separators=["\n\n", "\n", ". ", "? ", "! ", " "],
)

def chunk(doc):
    sections = md_splitter.split_text(doc.text)
    chunks = []
    for s in sections:
        for piece in recursive.split_text(s.page_content):
            chunks.append({
                "text": piece,
                "metadata": {**doc.metadata, **s.metadata},
            })
    return chunks
```

Recall@8 jumped from 76% → 84%. Simple, free.

## Win #2: Hybrid Search (BM25 + Dense), Then Re-Rank (+11% recall)

Pure vector search misses queries with rare keywords (model numbers, product codes). We added BM25 in parallel and combined the two with **reciprocal rank fusion**:

```python
def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)
```

Then we re-rank the fused top-25 with a cross-encoder (`bge-reranker-v2-m3`) and take the top-8. Adds ~120ms but recall@8 went to 95%.

## Win #3: Query Rewriting (+4% answer accuracy)

Users ask "what's the latency for X?" but our docs say "p99 response time." A small LLM call rewrites the user's query into 2-3 alternative phrasings:

```python
REWRITE_PROMPT = """Generate 3 alternative phrasings of this query that an internal doc might use.
Output JSON: {"queries": ["...", "...", "..."]}.
Query: {q}"""

alts = json.loads(llm_small(REWRITE_PROMPT.format(q=user_query)))["queries"]
all_results = [retrieve(q) for q in [user_query, *alts]]
fused = rrf(all_results)
```

Costs ~$0.0001 per query (3¢/1k queries). Worth it.

## Win #4: Stricter Prompt + Citation Requirement (-6% hallucination)

The biggest hallucination reduction came from a prompt change, not retrieval:

```
You are answering using ONLY the numbered context snippets below.
For every factual claim, append a citation like [3] referring to the snippet.
If the answer is not in the context, reply exactly:
  "I don't have that information."
Do not paraphrase claims that lack a citation.

Context:
[1] {chunk_1}
[2] {chunk_2}
...
```

Adding the **"do not paraphrase claims that lack a citation"** sentence dropped hallucinations by ~6 percentage points on its own.

## Win #5: Reject Low-Confidence Retrievals

If the **top-1 re-ranker score is below 0.35**, we don't even call the LLM — we return the "I don't have that information" answer directly. This caught most of the residual hallucinations on out-of-scope queries.

```python
top_score = reranker_scores[0]
if top_score < 0.35:
    return {
        "answer": "I don't have that information.",
        "confidence": "low",
        "sources": [],
    }
```

## The Final Numbers

| Metric | Before | After |
|---|---:|---:|
| Recall @ 8 | 76% | **95%** |
| Answer correct (LLM-graded) | 71% | **88%** |
| Hallucination rate | 14.2% | **5.7%** |
| p95 latency | 1.8s | 2.1s |
| Cost per query | $0.0021 | $0.0029 |

## What Didn't Work

- **Larger embedding model** (`text-embedding-3-large`): +1% recall, +2× cost. Not worth it.
- **Fine-tuning embeddings** on our domain: marginal gains, weeks of work.
- **Larger top-k (16, 32)**: noise crowded out signal once we had a re-ranker.

## Best Practices We Recommend

1. **Build an eval set first.** 100-200 hand-labeled QA pairs is enough to make decisions on. Without numbers you'll burn weeks chasing intuition.
2. **Fix retrieval before touching the model.** A bigger model hides retrieval bugs; it doesn't fix them.
3. **Always re-rank.** Cross-encoders are slow but consistently lift quality more than any other change.
4. **Make "I don't know" easy.** Most hallucinations come from the model trying to be helpful when it shouldn't.
5. **Track hallucination rate, not just accuracy.** Wrong-with-confidence is worse than refusing to answer.