Reranking in RAG can't fix what your retriever never fetched

Automation Engineering · By Caleb Sakala · June 10, 2026

Reranking in RAG has a reputation as the quick win. Bolt a cross-encoder onto the pipeline, the story goes, and mediocre retrieval turns sharp. Half of that holds up. A reranker does sharpen results, but only the ones it can see. If your vector search hands it five chunks and the answer lives in the chunk sitting at rank thirty, no reranker on the market will rescue you. It reorders the five it was given, and the right one stays out in the cold.

That is the part the quick-win framing skips. A reranker inherits whatever recall ceiling your first stage set, and most teams never raise the ceiling before reaching for the second stage.

A reranker only reorders what it's handed

Production RAG runs in two stages, and Pinecone's walkthrough lays out the division of labor cleanly. The first stage is a fast bi-encoder vector search that pulls a candidate set out of a corpus that might run to billions of documents. The second stage is a slower cross-encoder, the reranker, that scores each query-and-chunk pair and reorders them by relevance.

Here is the constraint nobody puts on the slide. The reranker's input is fixed by how deep the first stage fetched. Set top_k=5 and the reranker sees five candidates, full stop. The relevant chunk that your bi-encoder ranked twenty-eighth is not 'low in the results.' It is absent. The reranker cannot promote a chunk it was never handed, any more than a judge can award the blue ribbon to a contestant who stayed home.

Pinecone's own example retrieves twenty-five candidates and reranks down to three. The twenty-five is doing the quiet work. Most people copy the reranking call and leave the retrieval depth at whatever the tutorial defaulted to.

Why the bi-encoder buries the right chunk in the first place

To understand why depth matters so much, look at what the first stage gives up for speed. A bi-encoder compresses the entire meaning of a chunk into a single vector, usually 1024 or 1536 dimensions, and it does this at indexing time, long before it has ever seen your query. Cosine similarity then compares pre-computed vectors. Fast, and lossy.

Because of that compression, a chunk that shares vocabulary with the query can outrank the chunk that actually answers it. The right passage does not vanish, it just lands at rank twenty or rank forty instead of rank three. A better embedding model shrinks that gap but rarely closes it, which is the whole reason the second stage exists.

The cross-encoder works differently. It reads the query and the chunk together in one transformer pass at query time, so it loses almost none of that information and scores relevance far more accurately. That accuracy is real. So is the cost, paid in time.

How deep to over-fetch before reranking in RAG

The numbers that matter here come from teams that published their experiments. When Anthropic measured contextual retrieval, the reranking configuration retrieved the top 150 chunks and then reranked down to 20. Stacked on contextual embeddings and contextual BM25, that pipeline cut the top-20 retrieval failure rate by 67%, from 5.7% down to 1.9%. Everyone quotes the 67%. The number that produced it is the 150.

Pinecone's tutorial fetches 25 and narrows to 3. Anthropic fetched 150 and narrows to 20. Pick the ratio that fits your corpus and latency budget, but the shape is consistent: cast wide on the first stage, then let the reranker squeeze hard on the second. A workable starting point for most knowledge bases is fetching 50 to 150 candidates and reranking down to the 3 to 20 you actually send the model.

This is where the common advice gets it backwards. Most reranking guides tell you to go shopping for a stronger reranker model. Wrong first move. A middling reranker over 100 candidates beats a top-of-the-leaderboard one over 5, because the first is allowed to find the answer and the second never sees it.

# The lever is top_k, not the reranker brand
candidates = vector_search(query, top_k=150)        # cast wide
ranked = reranker.rank(query, candidates, top_n=20)  # squeeze hard
context = ranked[:20]                                # send fewer

Picture a support automation answering tickets from a 50,000-document help center. It retrieves eight chunks, reranks those same eight, and ships replies that miss the mark about half the time. Rebuild it to over-fetch 100 candidates and rerank down to six, and the same workflow starts surfacing the doc that actually holds the answer. That retrieve-then-rerank sequence is exactly the kind of multi-step flow you can assemble in Chase Agents, where the retrieval step and the rerank step are separate, individually testable stages instead of one opaque call you have to trust on faith.

The latency bill arrives at query time

Cross-encoders are slow for a concrete reason: nothing is pre-computed. Every query-and-chunk pair is a fresh transformer inference. Pinecone runs the math using the Sentence-BERT numbers: scoring 40 million records with a small BERT cross-encoder on a V100 GPU would take more than 50 hours for a single query, while the first-stage vector search returns in under 100 milliseconds. That gap is why you never rerank the corpus. You rerank the shortlist.

Reranking 150 candidates is cheap. Reranking 5,000 is a self-inflicted outage.

And the model you pick matters less than teams assume. Agentset's 2026 reranker comparison puts Cohere Rerank 3.5 at 0.735 nDCG@10 with p95 latency around 210ms, bge-reranker-large-v2 at 0.715 nDCG@10 at 145ms, and the base variant at 0.699 at 92ms. Roughly four points of nDCG separate the heavy hosted model from the light open one. Anthropic's depth-and-stacking change moved its failure metric by far more than four points. Spend your first dollar on candidate count, your second on hybrid retrieval, and only then on the reranker brand.

The practical upshot is that depth and model choice trade off against each other through latency. A nightly batch automation that reranks 500 candidates per query can afford a heavy cross-encoder, while a live chat reply cannot. Because each automation in Chase Agents can override its own model and reach a vector store through an MCP server connection, the batch job and the chat reply can run different rerankers against the same index without forking your code.

When a reranker is the wrong tool

A reranker is not free insurance, and there are three cases where adding one is a mistake.

If your corpus is small, a few hundred chunks, and the bi-encoder already returns the right passage in its top results, a reranker buys you latency and nothing else. Measure first.

If recall is the actual problem, meaning the right chunk is not in your top 150 at all, the reranker is powerless. It reorders a set that does not contain the answer. The fix lives upstream, in hybrid search that pairs BM25 lexical matching with embeddings, or in contextual retrieval. Notice that Anthropic put contextual embeddings and contextual BM25 ahead of the reranker on purpose. The reranker delivered the last stretch of improvement, not the first.

And if you are cramming 20 reranked chunks into the prompt and answer quality drops anyway, you have walked into the 'Lost in the Middle' effect that Liu and colleagues documented in 2023: models retrieve less reliably from the middle of a long context than from its edges. The answer there is to rerank harder and send fewer chunks, not to widen the window and hope.

Open your retrieval config and read one number: how many candidates you fetch before the reranker runs. If it matches the number you send to the model, your reranker isn't ranking anything, it's rubber-stamping. Widen the first number to 50 or 100, narrow the second, and re-measure recall before you go shopping for a fancier model. The fix is almost never the reranker you bought. It's the candidates you never gave it.