Hybrid retrieval, tuned: when BM25, when vectors, and when a reranker is a waste

1 · Where dense retrieval loses

An embedding is a position in a learned space, and a rare identifier has no meaningful position — it is close to whatever text talks about identifiers. Part numbers, error codes, policy numbers and freshly coined jargon are exactly the queries where the nearest neighbours are topically right and literally wrong. Raising top-k does not fix it: it returns more nearby paragraphs, not the one that contains the string. In our hybrid engine the vector side is a plain VectorIndexRetriever with a fixed top-k, so the failure is structural, not a tuning accident.

def build_baseline_engine(self, documents):
    print("Building baseline vector engine...")
    index = VectorStoreIndex.from_documents(documents)
    return index.as_query_engine(similarity_top_k=3)

2 · Where keyword retrieval loses

BM25 needs the query term to survive tokenisation and to appear in the document. Synonyms fail (refund vs money back), paraphrases fail, and so does any identifier whose punctuation the tokenizer splits. Ours is a regex over ASCII letters, digits and underscores after lowercasing, which has two consequences worth knowing before you blame BM25: E-1042 and E1042 tokenise differently, and a corpus that is not written in Latin script produces no keyword tokens at all.

# Tokenizer for the BM25 retriever: lowercase word tokens.
def word_tokenizer(text: str) -> List[str]:
    return re.findall(r"[A-Za-z0-9_]+", text.lower())

3 · Fuse on rank, not on score

BM25 scores are unbounded and corpus-dependent; cosine similarities live in a narrow band. Adding them requires normalising two different scales, and the normalisation is where most home-grown hybrids quietly break. Reciprocal rank fusion sidesteps the comparison: each retriever contributes candidates ranked, and the fused score is a function of position. Our engine asks for reciprocal rank explicitly and falls back to that mode when the installed LlamaIndex does not expose the enum, so an upgrade cannot silently switch the fusion strategy.

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)

vector_index = VectorStoreIndex(nodes)

bm25_retriever = BM25Retriever.from_defaults(
    nodes=nodes,
    similarity_top_k=2,
    tokenizer=word_tokenizer,
)
vector_retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=2)

try:
    sig = inspect.signature(QueryFusionRetriever.__init__)
    mode_enum = sig.parameters["mode"].annotation
    if hasattr(mode_enum, "RECIPROCAL_RANK"):
        target_mode = mode_enum.RECIPROCAL_RANK
    else:
        target_mode = "reciprocal_rank"
except Exception:
    target_mode = "reciprocal_rank"

fusion_retriever = QueryFusionRetriever(
    [vector_retriever, bm25_retriever],
    similarity_top_k=4,
    num_queries=1,
    mode=target_mode,
    use_async=True,
)

The engine

One chunking pass feeds both retrievers, so the two halves never disagree about what a passage is. Each side returns a small pool, fusion keeps the head of the merged list, and query expansion is switched off — one query in, one query retrieved, which keeps the comparison between strategies honest.

4 · Rerank: precision you pay for in latency

A cross-encoder reranker reads query and passage together; it is more accurate than either retriever and far more expensive per candidate. It pays off when the candidate pool is large and the final cut-off is tight — reranking 100 candidates down to 5 is a real change. It does not pay off when fusion already returns a handful of candidates: you pay a model call to reorder four items. Our own codebase ships no reranker at all; the fused retriever is the last ranking stage, and that is a deliberate position rather than an omission.

When not to add a reranker

• —Your fused pool is already small. Reranking four candidates is a model call that can only reorder four items; the latency is real, the upside is rounding.
• —The first token matters more than the fifth result. Reranking sits between retrieval and generation, so it delays every answer rather than some.
• —The query is already exact. Identifier lookups are decided by the keyword side; a cross-encoder can only agree with a match that already exists.
• —You have not measured what you are fixing. A reranker compensates for weak recall, but if the passage was never retrieved, reranking cannot recover it — fix the candidate pool first.