# Real-World RAG Incidents: Lessons from a Production Rollout

URL: https://www.devopsness.com/blog/real-world-rag-incidents-lessons-from-a-production-rollout
Author: Kiril Urbonas
Published: 2025-03-01
Updated: 2025-03-01
Categories: AI
Tags: Python, LLM, GPT

Summary: A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.

---
# Real-World RAG Incidents: Lessons from a Production Rollout

When we first rolled out a RAG-based assistant for our internal SRE team, nothing in the vendor docs really prepared us for the messy parts.

## Incident: Cached Wrong Answers

The first painful incident happened on a Monday morning. A runbook query returned an **outdated PostgreSQL failover procedure** because:

- We cached answers aggressively to save tokens.
- The underlying runbook in Git had been updated over the weekend.
- Our invalidation logic only watched the vector store, not the source repo.

### How We Fixed It

- We changed our cache key to include the **document commit hash**.
- We added a background job that compares Git commits against stored vectors.
- We updated the runbook template to show the **last updated date** in the answer.

## Incident: Embeddings Going Silent

Two weeks later, we saw a spike in “no relevant context found” errors during incident calls. The vector DB was healthy; the problem turned out to be:

- A new data source with **HTML-heavy content**.
- We were chunking purely by character count.
- The relevant text was split across three different chunks.

### Changes We Made

- Switched to **semantic + heading based chunking** with overlap.
- Added a metric for “chunks per query” and “distance of top-1 match”.
- Logged a sample of low-quality retrievals for manual review.

## Checklist for RAG in Production

- Track **cache hit rate**, **LLM error rate**, and **no-context rate**.
- Store the **retrieved chunk IDs** alongside each answer.
- Regularly sample answers and review them with the owning team.

The marketing pages sold RAG as magic. In reality it behaves more like a database: if you don’t design for drift, invalidation, and observability, it will betray you at the worst moment.