Pure vector search kept missing exact matches like error codes and CLI flags. Adding BM25 back and fusing the two lifted our retrieval recall by 11 points.
Hybrid Search for RAG: Combining BM25 and Vectors the Right Way
Our docs bot could answer conceptual questions beautifully and then fall on its face when someone pasted ERR_CONN_RESET_0x8004 and asked what it meant. The vector retriever returned three passages about connection handling in general and never the one page that contained that exact string. Embeddings are great at meaning and bad at rare literals. The fix wasn't a better embedding model, it was adding keyword search back and fusing the results.
Dense embeddings map text to a point in space where "reset the connection" and "connection was reset" land near each other. That's the strength. The weakness is that rare tokens (error codes, function names, version strings, --no-verify) get smeared into the surrounding semantic soup. The embedding for a chunk containing ERR_CONN_RESET_0x8004 barely moves because one weird token among 800 doesn't shift the vector much. So the exact match ranks below fuzzier but more "semantically central" passages.
BM25 has the opposite profile. It's a bag-of-words scorer that rewards exact term overlap and rare-term matches (via inverse document frequency). It nails ERR_CONN_RESET_0x8004 because that token is rare and appears verbatim. It's useless when the user's words don't overlap the document's words at all, which is where vectors shine. They fail on opposite inputs, which is exactly why fusing them works.
The clean way to combine them is Reciprocal Rank Fusion (RRF). It doesn't try to compare a cosine score against a BM25 score (those live on totally different scales, which is a trap). Instead it uses only the rank position from each retriever:
def rrf(rankings, k=60):
scores = {}
for ranking in rankings: # each is an ordered list of doc ids
for rank, doc_id in enumerate(ranking):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)
vec_ids = vector_search(query, top_k=50)
bm25_ids = bm25_search(query, top_k=50)
fused = rrf([vec_ids, bm25_ids])[:10]
The constant k=60 (from the original RRF paper) softens the influence of top ranks so a single retriever can't dominate. A doc that both retrievers rank highly floats to the top; a doc only one finds still gets a fair shot. On our eval set, RRF over BM25 and vectors moved Recall@10 from 0.78 (vectors alone) to 0.89. The error-code questions that used to fail now resolve because BM25 drags the exact-match page into the fused top 10.
Rolling your own fusion is fine, but if you're on Postgres with pgvector plus a tsvector column, or on OpenSearch, or Weaviate, the engine can do hybrid natively. In Postgres you keep both indexes and fuse in SQL:
WITH vec AS (
SELECT id, row_number() OVER (ORDER BY embedding <=> :qvec) AS r
FROM docs ORDER BY embedding <=> :qvec LIMIT 50
),
kw AS (
SELECT id, row_number() OVER (
ORDER BY ts_rank(tsv, plainto_tsquery(:q)) DESC) AS r
FROM docs WHERE tsv @@ plainto_tsquery(:q) LIMIT 50
)
SELECT COALESCE(vec.id, kw.id) AS id,
COALESCE(1.0/(60+vec.r),0) + COALESCE(1.0/(60+kw.r),0) AS score
FROM vec FULL OUTER JOIN kw USING (id)
ORDER BY score DESC LIMIT 10;
Same RRF math, one query, no application-side merge. <=> is pgvector's cosine distance operator; @@ is full-text match. Keeping it in the database meant one round trip instead of two and let us reuse existing GIN and HNSW indexes.
RRF treats both retrievers equally. Sometimes you want to tilt. For a codebase-heavy corpus we weighted BM25 slightly higher because exact identifiers mattered more than paraphrase:
scores[doc_id] += weight / (k + rank + 1) # weight 1.3 for bm25, 1.0 for vec
Be careful here. We spent two days tuning weights and gained half a recall point. The honest lesson: equal-weight RRF is 95% of the value, and per-retriever weighting is a rounding error unless your two retrievers have very different quality. Tune the retrievers first, the fusion weights last.
If your RAG has to handle exact literals (error codes, flags, symbol names, SKUs) alongside natural-language questions, run BM25 and vector search in parallel and fuse with RRF at k=60. It's a dozen lines, it needs no score normalization, and it recovers the exact-match cases pure embeddings quietly drop. Do it in your database if it supports both indexes. And don't burn a week on fusion weights before you've checked that each retriever is good on its own.
