Real-World RAG Incidents: Lessons from a Production Rollout
Last reviewed . Tool versions and defaults may have changed since, so check the commands against current docs.
A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.
On this page
When we first rolled out a RAG-based assistant for our internal SRE team, nothing in the vendor docs really prepared us for the messy parts.
Incident: Cached Wrong Answers
The first painful incident happened on a Monday morning. A runbook query returned an outdated PostgreSQL failover procedure because:
- We cached answers aggressively to save tokens.
- The underlying runbook in Git had been updated over the weekend.
- Our invalidation logic only watched the vector store, not the source repo.
How We Fixed It
- We changed our cache key to include the document commit hash.
- We added a background job that compares Git commits against stored vectors.
- We updated the runbook template to show the last updated date in the answer.
Incident: Embeddings Going Silent
Two weeks later, we saw a spike in “no relevant context found” errors during incident calls. The vector DB was healthy; the problem turned out to be:
- A new data source with HTML-heavy content.
- We were chunking purely by character count.
- The relevant text was split across three different chunks.
Changes We Made
- Switched to semantic + heading based chunking with overlap.
- Added a metric for “chunks per query” and “distance of top-1 match”.
- Logged a sample of low-quality retrievals for manual review.
Checklist for RAG in Production
- Track cache hit rate, LLM error rate, and no-context rate.
- Store the retrieved chunk IDs alongside each answer.
- Regularly sample answers and review them with the owning team.
The marketing pages sold RAG as magic. In reality it behaves more like a database: if you don’t design for drift, invalidation, and observability, it will betray you at the worst moment.
Cite this article
Kiril Urbonas (2025). Real-World RAG Incidents: Lessons from a Production Rollout. DevOpsNess. https://www.devopsness.com/blog/real-world-rag-incidents-lessons-from-a-production-rollout
[Real-World RAG Incidents: Lessons from a Production Rollout](https://www.devopsness.com/blog/real-world-rag-incidents-lessons-from-a-production-rollout) (Kiril Urbonas, DevOpsNess, 2025-03-01)
More from AI
Explore more articles in this category
vLLM vs SGLang vs TGI vs TensorRT-LLM: Which LLM Server in 2026
Hugging Face TGI is archived and read-only, so the real 2026 choice is vLLM, SGLang or TensorRT-LLM. A feature table, a launch command for each, and the one we default to.
Monitoring ML Models in Production: Data, Concept and Prediction Drift, With Code
Three kinds of drift, two tests that catch most of it, one Prometheus exporter, and the alert rules that do not page you every Monday morning.
MLOps Tools in 2026: 7 Platforms Compared, and the Stack We'd Pick
MLflow, Kubeflow, Weights & Biases, SageMaker, Vertex AI, ZenML and Metaflow compared on tracking, registry, pipelines, serving, self-hosting and pricing, with a default stack for each team size.
You might have missed
Evergreen posts worth revisiting.