Real-World RAG Incidents: Lessons from a Production Rollout

When we first rolled out a RAG-based assistant for our internal SRE team, nothing in the vendor docs really prepared us for the messy parts.

Incident: Cached Wrong Answers

The first painful incident happened on a Monday morning. A runbook query returned an outdated PostgreSQL failover procedure because:

• We cached answers aggressively to save tokens.
• The underlying runbook in Git had been updated over the weekend.
• Our invalidation logic only watched the vector store, not the source repo.

How We Fixed It

• We changed our cache key to include the document commit hash.
• We added a background job that compares Git commits against stored vectors.
• We updated the runbook template to show the last updated date in the answer.

Incident: Embeddings Going Silent

Two weeks later, we saw a spike in “no relevant context found” errors during incident calls. The vector DB was healthy; the problem turned out to be:

• A new data source with HTML-heavy content.
• We were chunking purely by character count.
• The relevant text was split across three different chunks.

Changes We Made

• Switched to semantic + heading based chunking with overlap.
• Added a metric for “chunks per query” and “distance of top-1 match”.
• Logged a sample of low-quality retrievals for manual review.

Checklist for RAG in Production

• Track cache hit rate, LLM error rate, and no-context rate.
• Store the retrieved chunk IDs alongside each answer.
• Regularly sample answers and review them with the owning team.

The marketing pages sold RAG as magic. In reality it behaves more like a database: if you don’t design for drift, invalidation, and observability, it will betray you at the worst moment.