Reranking in RAG: When a Cross-Encoder Earns Its Latency

Reranking in RAG: When a Cross-Encoder Earns Its Latency#

The complaint from support was specific: the bot would answer a question about refund windows and cite a paragraph about shipping delays that happened to mention the word "refund" once. The right paragraph was in our vector store. We checked. It was even in the top 20 results the retriever pulled back. It just wasn't in the top 4 we fed to the model, so the LLM never saw it.

That gap, between what a bi-encoder retrieves and what's actually most relevant, is the exact thing a reranker closes. It's also not free, and half the RAG systems I see add one without checking whether they need it.

Why the first-stage retriever misses#

Your vector search uses a bi-encoder. It embeds the query once, embeds every document once (offline), and compares them with cosine similarity. That's why it's fast: the document vectors are precomputed, so a query is one embedding plus an approximate nearest-neighbor lookup over millions of vectors in a few milliseconds.

The cost of that speed is that the query and document never actually meet. They're compressed into fixed vectors independently, and subtle relevance, "this paragraph is about the 30-day refund window specifically," gets averaged away. Two texts that share vocabulary can end up closer than a text that actually answers the question.

What a cross-encoder does differently#

A cross-encoder takes the query and one document together, as a single input, and runs them through a transformer that attends across both. It outputs a single relevance score. Because it sees the pair jointly, it catches the distinctions a bi-encoder blurs.

The catch is right there in the mechanism. You can't precompute anything. Every query-document pair is a fresh forward pass. So you never run a cross-encoder over your whole corpus. You use the fast retriever to get a candidate set, then rerank just those:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

# stage 1: bi-encoder pulls 40 candidates fast
candidates = vector_store.search(query, top_k=40)

# stage 2: cross-encoder scores each query-doc pair
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_docs = [c for c, _ in ranked[:4]]

Retrieve 40, rerank, keep 4. The LLM's context stays small and clean, but the 4 it gets are the genuinely best 4, not the best 4 the bi-encoder guessed.

The latency it costs#

On our setup, an ONNX-quantized bge-reranker-v2-m3 on a single L4 GPU scored 40 pairs in about 180ms at p50, 240ms at p95. That's added on top of the ~15ms vector lookup. For a chat bot where the LLM generation already takes 2-3 seconds, 180ms is invisible. For an autocomplete-style feature with a 100ms budget, it's a non-starter.

We measured the quality gain properly instead of trusting vibes. On a labeled set of 300 support questions, recall@4 (did the right passage make the final cut) went from 0.71 to 0.94. That 23-point jump is what killed the wrong-citation complaints.

Tuning the candidate count#

The one knob that matters most is how many candidates you rerank. Too few and the reranker can't fix a bad retrieval, because the right doc was never in the set. Too many and latency climbs linearly for shrinking returns. We swept it:

top_k=20  -> recall@4 0.88, rerank 95ms
top_k=40  -> recall@4 0.94, rerank 180ms
top_k=80  -> recall@4 0.95, rerank 360ms

Going from 40 to 80 doubled the latency for one point of recall. We stopped at 40. Your curve depends on how good your first-stage retrieval is; a weak retriever needs a wider net.

When to skip it#

If your recall@k is already high, meaning the right document reliably lands in your top few results, a reranker adds latency for no measurable gain. We have a second, smaller knowledge base (about 400 docs of internal API reference) where the bi-encoder alone hits 0.96 recall@4. We do not rerank that one. Adding a cross-encoder there would be pure cost.

The call we'd make#

Reach for a reranker when your evals show the answer is in your retrieved set but not in the final top-k you pass to the model. That's the specific failure it fixes, and it fixes it well. Don't add one reflexively because a blog post said RAG needs reranking. Measure recall@k first. If it's already high, spend your latency budget somewhere that moves the needle. And when you do add one, use a quantized ONNX build on a GPU, because the PyTorch fp32 version on CPU will turn 180ms into two full seconds and quietly wreck your p95.