Reranking fixes order, not retrieval. Pointed at the wrong problem it just buys you a few hundred milliseconds of latency and zero better answers, because a reranker cannot promote a document your retriever never returned. It is a second pass that re-sorts the documents your retriever already found, scoring each one against the query with a slower, more accurate model so the strongest match lands first. It does not search again. It takes the top 50 or 100 hits from a fast first-pass retrieval and reorders them.
First-pass retrieval is wide and cheap. Reranking is narrow and expensive. That one split tells you when to use it. You run the wide pass over millions of documents, then the expensive pass over a few dozen. Reranking earns its keep when your retriever pulls the right documents but stacks them in the wrong order.
TL;DR in 6 points: (1) Reranking re-sorts your top hits, it does not re-search. (2) First-pass retrieval uses a bi-encoder; reranking uses a slower cross-encoder. (3) Cross-encoders read the query and document together, so they judge relevance more precisely. (4) The cost is latency, typically in the low hundreds of milliseconds per call. (5) Rerank a small candidate set, usually the top 50 to 100, never your whole index. (6) Skip the reranker if your top result is already right most of the time.
The split that decides everything: first-pass retrieval is wide and cheap, reranking is narrow and expensive. Reach for the reranker only when your retriever finds the right documents but stacks them in the wrong order.
Bi-encoder retrieval vs cross-encoder reranking
First-pass retrieval uses a bi-encoder. Reranking uses a cross-encoder. The difference is when the query and the document meet. A bi-encoder turns the query into one vector and every document into its own vector, separately, ahead of time. At query time it just compares vectors with cosine similarity. That is why it scales to millions of documents: the document vectors are precomputed and stored.
A cross-encoder does the opposite. It feeds the query and a single document into the model together and lets every query token attend to every document token before producing one relevance score. That joint attention is what makes it accurate. It is also what makes it slow, because nothing can be precomputed. The work happens live, one pair at a time.
The Sentence Transformers documentation is blunt about the limit: scoring thousands or millions of query-document pairs with a cross-encoder would be far too slow. The fix is a split. Use the bi-encoder to retrieve a large candidate list, often around 100 hits, then use the cross-encoder to re-rank only those candidates. You pay the expensive model's price on 100 documents instead of a million.
Why first-pass retrieval gets the right docs in the wrong order
A bi-encoder compresses an entire document into a single vector before it ever sees your query. That compression is lossy. Two documents can land close together in vector space while only one actually answers the question, because the embedding captures broad topic similarity, not precise query-document fit. The retriever is great at the shortlist and weak at the running order: good at putting the right documents in the top 50, weaker at deciding which of those 50 belongs at position one.
Reranking fills exactly this gap. If your evaluation shows the correct answer is usually somewhere in your top 20 results but rarely at the top, you have an ordering problem, not a retrieval problem. Reranking fixes ordering. It cannot rescue an answer that never made it into the candidate set in the first place.
Recall is set by retrieval, top-k precision by reranking
Retrieval decides whether the right doc shows up. Reranking decides whether it shows up first. The first pass sets recall: did the right document make the shortlist at all? The rerank sets top-k precision: of the shortlist, which goes first? On the BEIR benchmark, a zero-shot evaluation across many retrieval datasets, re-ranking and late-interaction models achieved the best average performance, but the authors note this comes at high computational cost.
How a cross-encoder scores a query and a document together
A cross-encoder takes the query and one candidate document, concatenates them into a single input, and runs them through a transformer that produces one number: a relevance score, often normalized between 0 and 1. Because the two texts share the same forward pass, the model can notice that a specific phrase in the query is answered by a specific clause in the document.
You then sort your candidates by that score and keep the top few for your model's context window. The reranker does not generate or rewrite text; it only scores and reorders. Hosted rerank models, like Cohere's, also cap how much text each document contributes, truncating or chunking long documents so the combined query plus document stays within the model's context limit.
Reranking trade-off: quality versus latency and cost
Reranking buys you better ordering and costs you time, paid in the worst currency: the seconds between a question and its answer. Every candidate you rerank is one more forward pass through a heavier model, and that pass happens on the critical path. Latency varies by model and chunk size, but hosted rerankers typically add low hundreds of milliseconds to a request, and Cohere offers a dedicated fast variant aimed at low-latency, high-throughput use cases.
Cost scales with candidate count. Rerank 100 documents and you pay for 100 scoring operations. Rerank 1,000 and you pay ten times the latency for usually marginal gains. Cohere recommends against sending more than 1,000 documents in a single rerank request, which is a ceiling, not a target. Most pipelines rerank far fewer.
| Property | Bi-encoder (retrieval) | Cross-encoder (rerank) |
|---|---|---|
| When query meets document | Separately, document encoded ahead of time | Together, in one forward pass |
| Precompute possible | Yes, document vectors are stored | No, scoring happens live per query |
| Typical use | First pass over the full index | Second pass over the shortlist |
| Scale | Millions of documents | Tens to low hundreds of candidates |
| Relative accuracy | Good for recall | Higher for top-k precision |
| Latency added | Milliseconds, vector search | Typically low hundreds of ms per call |
How to know if you need a reranker
Run one check before adding a reranker: pull your top 20 retrieval hits for a set of real queries and record where the correct answer lands. If the correct document is usually present but often sits at position 5, 8, or 12, reranking will help. If the correct document frequently never appears in the top 20 at all, a reranker cannot save you. Fix retrieval first: better chunking, a stronger embedding model, or hybrid keyword plus vector search.
Measure recall@20 and top-1 accuracy separately. High recall@20 with low top-1 accuracy is the signal that reranking is worth it. Low recall@20 means the problem is upstream, and a reranker only adds latency without adding answers.
Where reranking helps most and where it is overkill
- Helps: large or noisy corpora where many documents look topically similar but only a few truly answer the query.
- Helps: question answering and RAG, where the model only reads the top 3 to 5 documents, so their order directly decides the answer.
- Helps: queries with subtle intent, where surface-level embedding similarity ranks a near-miss above the real match.
- Overkill: small collections where your retriever already returns the right document at position one most of the time.
- Overkill: latency-critical paths where an extra few hundred milliseconds breaks the experience and the ordering gain is small.
- Overkill: cases where you feed the model 20-plus documents anyway, so exact ordering inside that set matters less.
Picking a reranker: hosted API versus local model
Hosted rerank APIs, such as Cohere Rerank, give you a managed cross-encoder behind a single call: send a query and a list of documents, get back an ordered list with relevance scores. You skip model hosting and get a fast, regularly updated model, including low-latency variants. The trade-offs are per-request cost, network round-trip latency, and sending your documents to a third party.
Local open-source cross-encoders, available through the Sentence Transformers library, run on your own hardware. You keep data in house and pay no per-call fee, but you own the GPU, the throughput tuning, and the model updates. Smaller distilled cross-encoders, such as MiniLM-based rerankers, are popular precisely because they keep reranking latency manageable on modest hardware.
Common mistakes: reranking too many candidates, double-paying latency
The most expensive mistake is also the easiest to make: retrieve 500 hits and rerank all 500, paying a large latency tax for ordering gains that flatten out well before that. The candidate count is a knob, not a default. Start near the documented norm of about 100 candidates and tune down while watching your accuracy metric. Often 50 is plenty.
- Double-paying latency: running a reranker and then sending all reranked documents to the LLM regardless, so the rerank never reduces context size or cost.
- Reranking before deduplication: scoring five near-identical chunks wastes the candidate budget; deduplicate first.
- Ignoring the recall ceiling: a reranker cannot promote a document that retrieval never returned, so a low recall@k caps your gains.
- No evaluation: adding a reranker by vibe instead of measuring top-k accuracy before and after, then keeping it even when it does not help.
- Truncation blindness: long documents get truncated by hosted rerankers, so the deciding passage may be cut before it is scored.
How a memory layer reduces the candidates a reranker must sort
Reranking gets cheaper when the candidate set is already clean. If your first pass returns 30 tightly relevant documents instead of 300 loosely related ones, the cross-encoder has less to sort and the latency tax shrinks. Better upstream indexing, deduplication, and scoping by source or recency all shrink the pile before the expensive model ever runs.
This is where a memory layer fits. MemX (memx.app) is an external AI memory layer that captures your documents, screenshots, photos, voice notes, and forwarded messages, indexes them for retrieval, and lets you ask any model questions against that context. A well-scoped memory store returns a tighter, deduplicated candidate set, so in practice you can rerank 30 documents instead of 300, cutting the second-pass latency proportionally. MemX is a retrieval and memory layer, not a reranker itself, so it complements one rather than replacing it.
Key takeaway: Reranking is a slow, accurate second pass that fixes order, not retrieval. Reach for it when retrieval finds the right documents but orders them poorly, rerank a small candidate set, measure top-k accuracy before and after, and skip it entirely when your first result is already right.
01What is reranking in RAG?
Reranking in RAG is a second pass that re-sorts the documents a retriever already found, using a cross-encoder to score each one against the query and push the most relevant to the top. It does not search again; it only reorders the existing candidate set.
02Do I need a reranker for my RAG app?
Run one test: pull your top 20 retrieval hits and find where the correct answer lands. Usually present but rarely first means a reranker helps. Often missing entirely means fix retrieval first, because a reranker cannot rescue an answer that never made the shortlist.
03How many results should I rerank?
Start around the documented norm of 100 candidates and tune down while watching accuracy. Many pipelines do well with 50. Reranking 500 or more usually adds latency without meaningful gains. Cohere advises sending no more than 1,000 documents per request.
04What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder encodes query and documents separately, so document vectors are precomputed, making it fast and scalable for retrieval. A cross-encoder processes the query and one document together in a single pass, which is more accurate for top-k precision but too slow to run over a full index.
05How much latency does reranking add?
It depends on the model and candidate count, but hosted rerankers typically add low hundreds of milliseconds on the request path. Cohere offers a dedicated fast variant for low-latency, high-throughput cases. Latency scales with how many candidates you rerank.
