Hybrid Search: Combine BM25 and Vectors with RRF

Hybrid search combines retrieval methods because user questions do not all look alike. “Why does my session end early?” is a semantic question. “AUTH-417” is an exact identifier. A support search system should handle both without forcing users to guess which vocabulary the index prefers.

A practical baseline is a lexical retriever, a vector retriever, and reciprocal rank fusion, or RRF. The important work is making their candidate sets comparable and permission-safe, then testing whether the combination improves the queries you actually receive.

Why two retrievers can help

Lexical retrieval ranks text using term matches and related statistics. Dense retrieval compares embeddings produced by a model. These are different signals, and neither should receive automatic authority over every query.

The BEIR research benchmark tested multiple retrieval architectures across diverse datasets and identified BM25 as a robust baseline in that evaluation. That does not settle which retriever wins on your corpus. It does justify keeping a straightforward lexical baseline when evaluating a more elaborate system.

For a hypothetical authentication knowledge base, label questions involving error codes, paraphrased symptoms, configuration names, and multi-step explanations. If vector search improves paraphrases but loses exact-code cases, your aggregate average should not hide the regression.

Also inspect your lexical analyzer. An error code can be tokenized in a way that changes matching behavior. A dedicated exact field for identifiers may be appropriate alongside normal text search. Hybrid retrieval cannot compensate for an identifier that ingestion discarded.

RRF combines ranks rather than raw scores

Elasticsearch's RRF documentation defines a document's fused score as the sum of reciprocal rank contributions: each list contributes 1 / (k + rank) when the document appears. The ranking constant controls how sharply the contribution declines with rank. This avoids directly adding incompatible score scales.

Consider two illustrative result lists:

Lexical: A, B, C
Vector:  C, D, A

With k = 60, A receives 1/61 + 1/63, C receives 1/63 + 1/61, B receives 1/62, and D receives 1/62. A and C tie above B and D. You need a deterministic tie-break rule if consistent ordering matters to your product.

This example demonstrates the arithmetic, not a recommended production constant. It also exposes a limitation: rank fusion knows where a document appeared, not whether the underlying evidence is correct. Two retrievers agreeing on an obsolete page can confidently promote the wrong version.

Candidate depth is part of the design

Fusion only sees candidates each retriever returns. If the correct article ranks just outside both retrieval windows, no fusion formula can bring it back. Log candidates before and after fusion while tuning the window sizes.

Start with a budget you can afford, then compare several depths on a fixed set of questions. A wider window may recover useful documents but adds work downstream, particularly if you rerank every candidate. Measure search latency separately from generation latency so you know which stage changed.

Do not confuse candidate depth with the number of passages sent to the model. You might retrieve dozens of candidates and deliver only a few complementary passages. Those are separate controls with separate failure modes.

Apply access rules consistently

Both retrieval branches should operate within the user's authorized document set. Filtering only after fusion can waste most of the window on inaccessible records, and exposing even a title or excerpt from a restricted source can violate the product's access expectations.

Bind tenant and permission constraints to trusted request context, rather than accepting them from generated model arguments. Recheck authorization when fetching full source passages. If permissions change between indexing and retrieval, stale index metadata must not become permission to disclose.

The specific filtering mechanics depend on the search engine. Verify its behavior for approximate vector search and your index configuration. A correct filtering design may still change recall, so include heavily filtered tenants in the test set.

Deduplicate without losing useful diversity

Two retrievers may return the same document under different chunk identifiers. Establish a canonical identity before fusion. Otherwise, repeated evidence can occupy several slots and appear more important than it is.

Document-level deduplication can be too aggressive when different sections supply different facts. A useful compromise is to fuse stable passage identities, then limit excessive overlap during context assembly. Preserve the source ranges so the generator can cite the actual evidence.

Anthropic's contextual retrieval description includes lexical and embedding retrieval followed by combination and deduplication. Treat that as an example pipeline; your final passage count should come from your own evidence and latency constraints.

Evaluate the union and the final answer

Compare lexical-only, vector-only, and fused retrieval on the same labeled cases. Inspect which branch supplied each relevant result. If almost every improvement comes from one branch, that is useful evidence about where to invest.

Then check delivered evidence coverage and answer correctness. Better retrieval metrics are valuable, but they do not establish that the final answer used the right version or included the required exception. Keep those checks separate.

Choose document boundaries with the RAG chunking guide, then use the retrieval evaluation workflow to check whether fusion actually recovers missing evidence. More retrieval components are useful only when they improve the queries that matter.

Use hybrid search when the complementary signals justify the extra operational work. If a small, identifier-heavy corpus is already served well by lexical search, keep that baseline. Complexity earns its place by fixing observed failures.