Skip to main content
Reranking in RAG: When a Cross-Encoder Earns Its Latency

Reranking in RAG: When a Cross-Encoder Earns Its Latency

KU
· Founder & Chief Editor
• 7 min read•Updated •

Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixes relevance but adds a forward pass per candidate. Here's when that trade pays off.

Key takeaways

  • Add a cross-encoder reranker when your evals show the right passage is retrieved but not in the final top-k sent to the model.
  • Retrieve 20 to 100 candidates with the bi-encoder, rerank them, keep the best few.
  • Skip it when recall at k is already high, because it adds a forward pass per candidate.

A cross-encoder reranker earns its latency when the right passage is in your retrieved candidates but not in the top few you pass to the model. Retrieve a wider set with the bi-encoder, score each query-passage pair with the cross-encoder, keep the best 4. If recall at k is already high, skip it: every candidate costs a forward pass.

Verified against

sentence-transformers 6.2.0 (PyPI), BAAI/bge-reranker-v2-m3 (0.6B params, Apache-2.0), against the sbert.net Retrieve & Re-Rank docs and the Hugging Face model card. Checked October 2026.

The complaint from support was specific: the bot would answer a question about refund windows and cite a paragraph about shipping delays that happened to mention the word "refund" once. The right paragraph was in our vector store. We checked. It was even in the top 20 results the retriever pulled back. It just wasn't in the top 4 we fed to the model, so the LLM never saw it.

That gap, between what a bi-encoder retrieves and what's actually most relevant, is the exact thing a reranker closes. It also costs something, and half the RAG systems I see add one without checking whether they need it.

Why the first-stage retriever misses

Your vector search uses a bi-encoder. It embeds the query once, embeds every document once (offline), and compares them with cosine similarity. That's why it's fast: the document vectors are precomputed, so a query is one embedding plus an approximate nearest-neighbor lookup over millions of vectors in a few milliseconds.

The cost of that speed is that the query and document never actually meet. They're compressed into fixed vectors independently, and subtle relevance, "this paragraph is about the 30-day refund window specifically," gets averaged away. Two texts that share vocabulary can end up closer than a text that actually answers the question. If you want the background on why, our beginner's guide to embeddings covers how those vectors are built.

What a cross-encoder does differently

A cross-encoder takes the query and one document together, as a single input, and runs them through a transformer that attends across both. It outputs a single relevance score. Because it sees the pair jointly, it catches the distinctions a bi-encoder blurs.

The catch is right there in the mechanism. You can't precompute anything. Every query-document pair is a fresh forward pass. So you never run a cross-encoder over your whole corpus. You use the fast retriever to get a candidate set (the sbert.net docs suggest something like 100), then rerank just those:

rerank.py
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

# stage 1: bi-encoder pulls 40 candidates fast
candidates = vector_store.search(query, top_k=40)

# stage 2: cross-encoder scores each query-doc pair
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_docs = [c for c, _ in ranked[:4]]

Retrieve 40, rerank, keep 4. The LLM's context stays small and clean, but the 4 it gets are the genuinely best 4, not the best 4 the bi-encoder guessed.

The latency it costs

Expect a quantized bge-reranker-v2-m3 on a single modern GPU to add on the order of a couple of hundred milliseconds for a few dozen pairs, on top of the vector lookup. That is a rough order of magnitude, not a benchmark: it moves with hardware, batch size and passage length, so measure it on yours. For a chat bot where generation already takes seconds, that is close to invisible. For an autocomplete-style feature with a 100ms budget, it's a non-starter.

Measure the quality gain instead of trusting vibes. On a labeled set of real questions, compare recall@k (did the right passage make the final cut) with and without the reranker. If the jump is small, the reranker is not earning its latency. Building that labeled set is the part most teams skip; our offline retrieval evaluation harness is how we keep it honest.

Tuning the candidate count

The one knob that matters most is how many candidates you rerank. Too few and the reranker can't fix a bad retrieval, because the right doc was never in the set. Too many and latency climbs linearly for shrinking returns. Sweep it on your own data and record both numbers per setting. The shape usually looks like this (illustrative values, not a benchmark):

code
top_k=20  -> recall@4 lower,      rerank time 1x
top_k=40  -> recall@4 much higher, rerank time 2x
top_k=80  -> recall@4 barely higher, rerank time 4x

When doubling k doubles latency for a point of recall, stop at the smaller k. Your curve depends on how good your first-stage retrieval is; a weak retriever needs a wider net, and adding keyword matching through hybrid BM25 plus embeddings search often helps more than widening k.

When to skip it

If your recall@k is already high, meaning the right document reliably lands in your top few results, a reranker adds latency for no measurable gain. Small, well-structured corpora, such as a few hundred pages of internal API reference, often reach this point with the bi-encoder alone. Adding a cross-encoder there is pure cost.

The other skip case is when the right document is not in the candidate set at all. A reranker only reorders what it is given. If recall@40 is low, fix chunking, embeddings, or filters first.

Questions people ask

Do I need a reranker for RAG?

Only if your evals show the right passage is retrieved but ranked below your final cutoff. Measure recall at your final k and at your candidate k. A big gap between the two (low recall at 4, but the passage reliably present in the top 40) is exactly what a reranker fixes. If both numbers are high, or both are low, a reranker will not move answer quality.

How many candidates should a cross-encoder rerank?

Start between 20 and 100 and sweep. The sentence-transformers Retrieve & Re-Rank docs use around 100 as an example. Cost grows linearly with candidates because each pair is a separate forward pass, while recall gains flatten. Stop at the knee of your curve; for our support corpus that was 40.

Is a cross-encoder slower than a bi-encoder?

Yes, per query. A bi-encoder compares one query embedding against precomputed document vectors, so search is a fast nearest-neighbor lookup. A cross-encoder runs the transformer on every query-document pair at query time, with nothing precomputed. That is why it is used only as a second stage over a small candidate set.

The call we'd make

Reach for a reranker when your evals show the answer is in your retrieved set but not in the final top-k you pass to the model. That's the specific failure it fixes, and it fixes it well. Don't add one reflexively because a blog post said RAG needs reranking. Measure recall@k first. If it's already high, spend your latency budget somewhere that moves the needle. And when you do add one, serve it on a GPU with a quantized or half-precision build, because a full-precision model on CPU can multiply that latency many times over and quietly wreck your p95.

Sources

Explore topics:LLMAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Reranking in RAG: When a Cross-Encoder Earns Its Latency. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/reranking-in-rag-when-cross-encoder-earns-latency

Markdown

[Reranking in RAG: When a Cross-Encoder Earns Its Latency](https://www.devopsness.com/blog/reranking-in-rag-when-cross-encoder-earns-latency) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

608 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.