Embeddings Drift Detection: When "Similar Enough" Stops Being Similar

Detect embedding drift by logging the top-1 similarity score of every live query and comparing each day's distribution with a baseline week: a left shift in p50 and p10 is the leading signal. Confirm with top-k spread, refusal rate and thumbs-down rate. The vectors rarely change; the queries and the corpus do.

We run RAG features over a few hundred thousand chunks. A deterministic model like bge-large-en-v1.5 produces the same vector for the same input, but retrieval quality still drifts as the corpus, the queries and the product change. We once had a regression grow for weeks before any metric caught it. This post is the set of signals we built afterwards.

Two kinds of drift

Index drift. The documents and the live queries grow apart. The index was built on last year's docs; today's questions are about things the docs don't cover. Nearest-neighbor search always returns something, so the top match stays plausible-looking while relevance drops. Quality falls without any document changing.

Model drift. The embedding model was updated or replaced. Vectors from a different model live in a different space, often with a different dimension (bge-large-en-v1.5 is 1024, text-embedding-3-small defaults to 1536), and comparing them across models gives noise. You usually know about this one, because you deployed it.

Index drift is the slow, sneaky one, so most of this post is about it. If embeddings are new to you, start with what embeddings are.

The signals we monitor

Top-1 score distribution. For every live query, record the similarity of the top retrieved chunk and track p50, p25 and p10 daily. Do not hard-code an absolute "good match" threshold borrowed from someone else's model. The BGE model card says its similarity scores sit roughly in the interval [0.6, 1] and that a score above 0.5 "does not indicate that the two sentences are similar"; it recommends picking thresholds from your own data. Compare against your own baseline instead. This is the leading indicator, and it is the one that would have caught our regression.

Top-k spread. Healthy retrieval has a clear winner. Drift shows up as a flatter list where score[0] - score[4] shrinks, meaning nothing in the index is a strong match.

Citation rate. What fraction of answers cite at least one retrieved chunk? A drop means the LLM isn't finding usable context.

Refusal rate. "I don't have that information in the documentation." A rising rate usually means the index no longer covers what users ask.

User feedback. Thumbs-down rate and reformulation rate (the user asks again, differently). Lagging, but reliable.

Chart all five together. One number moving alone is often noise; three moving together is signal.

A drift check you can run daily

Log (timestamp, query_id, top_scores[]) for every retrieval, then compare today against a baseline window. A two-sample Kolmogorov-Smirnov test catches distribution shifts that a mean hides:

import numpy as np
from scipy.stats import ks_2samp

def summarize(top1):
    return {p: round(float(np.percentile(top1, p)), 3) for p in (10, 25, 50)}

def drift_report(baseline_top1, today_top1, today_topk):
    ks = ks_2samp(baseline_top1, today_top1, alternative="greater")
    spread = np.median([s[0] - s[min(4, len(s) - 1)] for s in today_topk])
    return {
        "baseline": summarize(baseline_top1),
        "today": summarize(today_top1),
        "ks_statistic": round(float(ks.statistic), 3),
        "p_value": float(ks.pvalue),
        "median_top_k_spread": round(float(spread), 3),
    }

Feed it a baseline week of top-1 scores and one day of results from your query log; drift_report(...) returns a dict you can push to your metrics backend. With alternative="greater", the test asks whether today's scores sit lower than the baseline's. With thousands of queries a day, p-values get tiny for trivial shifts, so alert on effect size: page someone when the KS statistic and a p10 drop both cross thresholds you calibrated on a few weeks of normal traffic, not on the p-value alone.

What the regression looked like

Mean top-1 score slid gradually for about six weeks. Nobody noticed because latency was unchanged, citation rate barely moved (the LLM still cited chunks, just the wrong ones), and thumbs-down crept up slowly enough to stay under the alert line.

The cause: a feature had launched and users were asking about it, but the docs hadn't been updated. Existing chunks were the nearest match for the new queries without being relevant. The fix was content, not ML: group the low-scoring queries by topic, write the missing docs, re-embed. The fix took a day. Detection took six weeks because nobody was watching the distribution.

Re-embedding cadence

• Static docs: embed once; re-embed only when the model changes.
• Frequently edited docs (product docs, runbooks): re-embed changed documents nightly via change-data-capture on the content store.
• Curated external content: re-embed when the curation runs.

Re-embedding is cheaper than most teams assume. At OpenAI's October 2026 list price of $0.02 per 1M input tokens for text-embedding-3-small, 280,000 chunks averaging 800 tokens (224M tokens) cost about $4.50 to re-embed in full. Self-hosted bge-large-en-v1.5 costs GPU hours instead; note its 512-token limit when you chunk.

Re-embed proactively when the model changes (always), when more than about 20% of the corpus changed, or after a big shift in what users ask. When the dashboard flags a regression, investigate first: as above, the fix is often missing content, and re-embedding the same docs changes nothing.

The shadow index pattern

Never cut over to a new embedding model blind. Run the candidate as a shadow index for a couple of weeks: embed live queries with both models, query both indexes, store both results, and compare:

• Recall@5 on a labeled eval set. Our offline RAG retrieval harness is how we build and score that set.
• Top-1 score distribution, compared with the candidate's own baseline, since raw scores across models are not comparable.
• Answer quality from an LLM judge on the same queries.

The shadow doubles embedding spend and vector storage for the period. Cut over behind a feature flag that selects the index version, keep the old index for a week as fallback, then decommission it.

Failure modes and the errors you'll see

Where this breaks

Score-distribution monitoring assumes a dense-only retriever. With hybrid BM25 plus embeddings or a reranker, track the final fused or reranked score as well, because the dense score alone can drift while answers stay fine. Very low traffic (a few dozen queries a day) makes daily distributions too noisy; use weekly windows. And none of these signals replaces a labeled eval set; they tell you when to run it.

Questions people ask

What is embedding drift in RAG?

Embedding drift is the gradual mismatch between what users ask and what your vector index covers, or a change in the embedding model that puts new vectors in a different space. Nearest-neighbor search still returns a top result, so nothing errors, but relevance falls. It shows up as lower top-1 similarity scores, flatter top-k results, more refusals and more thumbs-down feedback.

How do you detect embedding drift?

Log the top-1 similarity score of every live query and compare each day's distribution with a baseline window using percentiles and a two-sample KS test from SciPy. Alert on effect size, such as a p10 drop, rather than p-values alone. Confirm with top-k spread, citation rate, refusal rate and user feedback, since several signals moving together are far more reliable than one.

How often should you re-embed documents?

Re-embed changed documents nightly, re-embed everything whenever the embedding model changes, and do a full refresh after large corpus changes of roughly 20% or more. Static content never needs re-embedding with the same deterministic model. At $0.02 per 1M tokens for text-embedding-3-small (October 2026), a full re-embed of a few hundred thousand chunks costs a few dollars.

The call we'd make

Track the top-1 score distribution daily against your own baseline before anything else; it is the cheapest, earliest drift signal. Back it with refusal rate and user feedback, re-embed changed docs nightly, and shadow-test every model change. When the dashboard moves, look for a content gap before you blame the model: in our experience, that is where the fix usually is.

Sources

• BAAI bge-large-en-v1.5 model card (Hugging Face)
• OpenAI API pricing
• OpenAI: Embeddings guide
• SciPy: scipy.stats.ks_2samp