Reranking in RAG: When a Cross-Encoder Earns Its Latency
Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixes relevance but adds a forward pass per candidate. Here's when that trade pays off.
Key takeaways
- Add a cross-encoder reranker when your evals show the right passage is retrieved but not in the final top-k sent to the model.
- Retrieve 20 to 100 candidates with the bi-encoder, rerank them, keep the best few.
- Skip it when recall at k is already high, because it adds a forward pass per candidate.
On this page
A cross-encoder reranker earns its latency when the right passage is in your retrieved candidates but not in the top few you pass to the model. Retrieve a wider set with the bi-encoder, score each query-passage pair with the cross-encoder, keep the best 4. If recall at k is already high, skip it: every candidate costs a forward pass.
Verified against
sentence-transformers 6.2.0 (PyPI), BAAI/bge-reranker-v2-m3 (0.6B params, Apache-2.0), against the sbert.net Retrieve & Re-Rank docs and the Hugging Face model card. Checked October 2026.
The complaint from support was specific: the bot would answer a question about refund windows and cite a paragraph about shipping delays that happened to mention the word "refund" once. The right paragraph was in our vector store. We checked. It was even in the top 20 results the retriever pulled back. It just wasn't in the top 4 we fed to the model, so the LLM never saw it.
That gap, between what a bi-encoder retrieves and what's actually most relevant, is the exact thing a reranker closes. It also costs something, and half the RAG systems I see add one without checking whether they need it.
Why the first-stage retriever misses
Your vector search uses a bi-encoder. It embeds the query once, embeds every document once (offline), and compares them with cosine similarity. That's why it's fast: the document vectors are precomputed, so a query is one embedding plus an approximate nearest-neighbor lookup over millions of vectors in a few milliseconds.
The cost of that speed is that the query and document never actually meet. They're compressed into fixed vectors independently, and subtle relevance, "this paragraph is about the 30-day refund window specifically," gets averaged away. Two texts that share vocabulary can end up closer than a text that actually answers the question. If you want the background on why, our beginner's guide to embeddings covers how those vectors are built.
What a cross-encoder does differently
A cross-encoder takes the query and one document together, as a single input, and runs them through a transformer that attends across both. It outputs a single relevance score. Because it sees the pair jointly, it catches the distinctions a bi-encoder blurs.
The catch is right there in the mechanism. You can't precompute anything. Every query-document pair is a fresh forward pass. So you never run a cross-encoder over your whole corpus. You use the fast retriever to get a candidate set (the sbert.net docs suggest something like 100), then rerank just those:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
# stage 1: bi-encoder pulls 40 candidates fast
candidates = vector_store.search(query, top_k=40)
# stage 2: cross-encoder scores each query-doc pair
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_docs = [c for c, _ in ranked[:4]]
Retrieve 40, rerank, keep 4. The LLM's context stays small and clean, but the 4 it gets are the genuinely best 4, not the best 4 the bi-encoder guessed.
The latency it costs
Expect a quantized bge-reranker-v2-m3 on a single modern GPU to add on the order of a couple of hundred milliseconds for a few dozen pairs, on top of the vector lookup. That is a rough order of magnitude, not a benchmark: it moves with hardware, batch size and passage length, so measure it on yours. For a chat bot where generation already takes seconds, that is close to invisible. For an autocomplete-style feature with a 100ms budget, it's a non-starter.
Measure the quality gain instead of trusting vibes. On a labeled set of real questions, compare recall@k (did the right passage make the final cut) with and without the reranker. If the jump is small, the reranker is not earning its latency. Building that labeled set is the part most teams skip; our offline retrieval evaluation harness is how we keep it honest.
Tuning the candidate count
The one knob that matters most is how many candidates you rerank. Too few and the reranker can't fix a bad retrieval, because the right doc was never in the set. Too many and latency climbs linearly for shrinking returns. Sweep it on your own data and record both numbers per setting. The shape usually looks like this (illustrative values, not a benchmark):
top_k=20 -> recall@4 lower, rerank time 1x
top_k=40 -> recall@4 much higher, rerank time 2x
top_k=80 -> recall@4 barely higher, rerank time 4x
When doubling k doubles latency for a point of recall, stop at the smaller k. Your curve depends on how good your first-stage retrieval is; a weak retriever needs a wider net, and adding keyword matching through hybrid BM25 plus embeddings search often helps more than widening k.
When to skip it
If your recall@k is already high, meaning the right document reliably lands in your top few results, a reranker adds latency for no measurable gain. Small, well-structured corpora, such as a few hundred pages of internal API reference, often reach this point with the bi-encoder alone. Adding a cross-encoder there is pure cost.
The other skip case is when the right document is not in the candidate set at all. A reranker only reorders what it is given. If recall@40 is low, fix chunking, embeddings, or filters first.
Questions people ask
Do I need a reranker for RAG?
Only if your evals show the right passage is retrieved but ranked below your final cutoff. Measure recall at your final k and at your candidate k. A big gap between the two (low recall at 4, but the passage reliably present in the top 40) is exactly what a reranker fixes. If both numbers are high, or both are low, a reranker will not move answer quality.
How many candidates should a cross-encoder rerank?
Start between 20 and 100 and sweep. The sentence-transformers Retrieve & Re-Rank docs use around 100 as an example. Cost grows linearly with candidates because each pair is a separate forward pass, while recall gains flatten. Stop at the knee of your curve; for our support corpus that was 40.
Is a cross-encoder slower than a bi-encoder?
Yes, per query. A bi-encoder compares one query embedding against precomputed document vectors, so search is a fast nearest-neighbor lookup. A cross-encoder runs the transformer on every query-document pair at query time, with nothing precomputed. That is why it is used only as a second stage over a small candidate set.
The call we'd make
Reach for a reranker when your evals show the answer is in your retrieved set but not in the final top-k you pass to the model. That's the specific failure it fixes, and it fixes it well. Don't add one reflexively because a blog post said RAG needs reranking. Measure recall@k first. If it's already high, spend your latency budget somewhere that moves the needle. And when you do add one, serve it on a GPU with a quantized or half-precision build, because a full-precision model on CPU can multiply that latency many times over and quietly wreck your p95.
Sources
Cite this article
Kiril Urbonas (2026). Reranking in RAG: When a Cross-Encoder Earns Its Latency. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/reranking-in-rag-when-cross-encoder-earns-latency
[Reranking in RAG: When a Cross-Encoder Earns Its Latency](https://www.devopsness.com/blog/reranking-in-rag-when-cross-encoder-earns-latency) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from AI
Explore more articles in this category
Mistral Large 4 'Le Chonk': Self-Host or Use the API?
A 1T-parameter MoE model with open weights doesn't make self-hosting the right call for most teams.
GPT-6.1 Sol's 80% Price Cut Rewrites Agentic Coding Math
OpenAI priced GPT-6.1 Sol at a fifth of GPT-6 Astra and called it nearly as good at agentic coding. That claim is now a routing decision, not a headline.
Claude Code Mods: Programmable, Unsandboxed, and Your Problem in CI
Mods make Claude Code a real runtime you can program. They also run with your full native permissions, no sandbox, so a bad one is a bad CI step.
You might have missed
Evergreen posts worth revisiting.