RAG Reranking: Better Evidence Within a Latency Budget

RAG reranking is useful when search finds the right evidence but does not place it where your answer generator can use it. It is less useful when ingestion missed the document entirely. Before adding a reranker, inspect a few failed queries and determine which problem you have.

Suppose a search over deployment documentation returns the correct rollback instructions in position 23. Your prompt includes six passages. A reranker has an opportunity to help. If the rollback instructions are absent from the first hundred candidates, a reranker scoring those candidates cannot recover them.

Separate discovery from close reading

Sentence Transformers documents a two-stage design: retrieve a candidate set, then score query-document pairs with a cross-encoder. The second stage can examine the query and passage jointly, while the first stage makes searching a large collection tractable.

That architecture gives you two independent budgets. Candidate depth controls how much evidence the reranker can consider. Final context size controls how much the generator receives. Raising both at once makes it difficult to attribute a change in quality or cost.

Begin with logged examples where relevant evidence is present but poorly ordered. If those examples are rare, evaluate whether improving parsing, query formulation, or source freshness would do more for users than another inference step.

Define relevance for the actual question

A passage about “deployment rollback” may be topically relevant but wrong for the user's database version. Include version, platform, and environment constraints in the query or metadata that reaches the scoring stage.

For an illustrative question, “How do I reverse migration 7 on a self-hosted installation?”, a generic cloud rollback page should rank below the self-hosted migration procedure. Your evaluation labels should distinguish these cases. A broad topic label would reward the wrong result.

Cohere's reranking guide describes submitting a query and candidate documents to obtain reordered relevance results. Keep the service contract distinct from your product contract: a relevance score does not establish authorization, freshness, or factual authority.

Store those properties separately and enforce them in application code. An inaccessible page does not become acceptable because a model assigns it a high score.

Preserve the text the model actually scored

Long documents create a subtle debugging trap. Your application may log a full passage while the reranking path scores a shorter representation. If the relevant exception lies beyond the effective input, the output can look inexplicable.

Check the chosen model's documented input handling and test boundary cases. Record the exact input length, selected fields, and any truncation performed by your code. Avoid assuming that every provider truncates in the same way or that every reranker accepts the same context length.

A practical representation might include a title, section path, version, and bounded source text. It should be concise enough to score predictably while preserving the facts that distinguish otherwise similar candidates.

Do not add unsupported summaries merely to make a passage look relevant. If you introduce generated summaries, keep the original evidence available for final verification and citations.

Measure the candidate ceiling

Calculate the fraction of evaluation questions whose necessary evidence exists in the candidate set. This is an upper bound on what selecting from that set can accomplish, under your definition of necessary evidence.

Then measure how often the final selected context retains that evidence. The gap identifies room for ranking improvement. If candidate coverage is 70 percent in an illustrative dataset, a perfect selector still cannot supply missing evidence for the remaining 30 percent.

Also inspect multi-passage questions. A selector that chooses six near-duplicates may rank individually relevant text highly while failing to cover all required facts. Add a context assembly step that rewards complementary evidence rather than blindly taking the highest scores.

These percentages are teaching examples, not measured results from this blog or a model vendor.

Budget the complete request

A reranker adds processing, but a smaller final prompt may reduce work elsewhere. Measure end-to-end latency rather than assuming the added stage always makes the product slower or faster.

For an interactive assistant, record queue time, retrieval time, reranking time, time to first answer token, and total completion time. Compare typical and slow requests. A fast median can hide a timeout pattern that affects long documents or busy periods.

BEIR's evaluation reports a quality and computation tradeoff for reranking architectures in its tested settings. Use that as motivation to measure your own tradeoff, not as a promise of improvement from any particular model.

Choose a timeout policy before release. Falling back to the original ranking may be acceptable for low-risk search suggestions. For a policy assistant requiring a specific evidence standard, an explicit inability to answer may be preferable. Make fallback behavior visible in telemetry.

Introduce reranking as a controlled change

Replay the same queries against the original and proposed pipeline. Keep the embedding model, corpus, prompt, and answer settings fixed. Review regressions as carefully as wins, especially cases involving obsolete versions and exceptions.

The hybrid retrieval guide covers candidate discovery before reranking. Use the RAG evaluation breakdown to distinguish better evidence selection from improvements in the generated answer.

Adopt the reranker if it improves the evidence users receive within your operating budget. Keep the baseline configuration available for rollback, and preserve model and preprocessing versions in every comparison. Ranking improvements should be explainable through cases, not just through a more impressive architecture diagram.