Track C · Document 15 · Serving at scale
Dense and sparse retrieval fail on different queries. Fusion decides which list wins; reranking decides what goes first — and they are not the same stage.
Everything here follows from one observation: a dense vector and a sparse vector fail on different queries. They are not two ways of doing the same thing at different quality levels. They are two different instruments, and a corpus full of identifiers, codes, names, error strings and acronyms needs the second one.
The library, with two librarians. The first has read every book and understands what each is about; ask for “something on getting money back after a bad purchase” and she walks straight to the refunds shelf. Ask her for the book with INV-20419 on the spine and she shrugs — she never memorised spine labels. The second has read nothing, but he has built a card index of every word and which books contain it. He hands you INV-20419 instantly and brings you nothing at all for “money back”. Hybrid is hiring both. Fusion is the rule for merging their two piles.
A second representation of every chunk, in a sparse column alongside the dense one — storage cost small. A second index, inverted, over that column — build time, write amplification, a few gigabytes. A merge rule in the query path — one extra step, microseconds.
And nothing else. That is why hybrid is the easiest large retrieval improvement to sell: it is purely additive, and no existing vector has to be recomputed.
Both are vectors. The difference is what a slot means.
A confusion worth killing early: there is no setting on an embedding model that switches it between dense and sparse output. They come from different pipelines that you run on the same text — an embedding model produces the dense vector, and BM25 counting (or a learned sparse model) produces the sparse one.
In pgvector this is explicit: a vector(1536) column your ingest code fills from
a model call, and a tsvector column Postgres computes from the text itself. Qdrant
and Weaviate hide the second pipeline — you declare a sparse vector field or a BM25
configuration and the server computes it on write — but it is still two representations
produced two different ways.
| Layer | Dense side | Sparse side |
|---|---|---|
| Representation — the shape | Short, all non-zero | Vocabulary-long, mostly zero |
| Producer — how it is filled | An embedding model | BM25 or TF-IDF counting, or a learned model such as SPLADE or ELSER |
| Index — how it is searched | HNSW, IVF, flat | An inverted index: word → posting list |
Say sparse for the representation, BM25 for the classic scoring that fills it, and learned sparse for SPLADE-style models. “Sparse is BM25” is loose — BM25 is one way to fill a sparse vector. That table is the answer to “what is the difference between sparse and BM25?”: they are different rows.
You will be asked to explain BM25 at some point. Here is the version that fits on a whiteboard, with the reference stack’s numbers already in it.
You are not expected to tune k1 and b in a RAG system, and saying so is the right answer. Lucene’s defaults of 1.2 and 0.75 are what Elasticsearch, Weaviate, Milvus and Qdrant all ship. k1 controls how fast repeated occurrences stop adding score; b controls how hard longer chunks are penalised. Move b down only if your chunk lengths are uniform anyway.
Stemming is per language. A Postgres tsvector built with the English
configuration turns “running” into “run” and drops “the” as a stop
word — and does nothing sensible at all to German or Japanese text. Index a mixed-language
corpus with one configuration and half of it degrades to near-exact string matching without a
single error being raised.
Three honest options. Detect the language at ingest and store a per-language sparse column.
Use the simple configuration, which does no stemming, and accept that you are
matching exact tokens only — which is fine if identifiers are the point. Or use a
multilingual learned sparse model. What is not an option is one English configuration over
a corpus that is not English, which is what most systems ship with.
The sparse side needs something to count. A 60-token chunk gives BM25 almost no term frequency signal, and the length normalisation in the formula treats it as maximally short, so tiny chunks all score alike. A dense-only system tuned to 100-token chunks is usually the wrong shape once the keyword half arrives.
The pattern that serves both: 200 to 400 tokens with the title prepended, or parent–child, where the child is embedded and the parent is what the sparse side counts. See document 01 for the chunking side of that argument.
Two ranked lists of N rows each; you need one list of k. The obvious idea is to add each row’s two scores, and it fails, because the two scales have nothing to do with each other.
Two judges, one scoring out of ten and the other out of a thousand. Add their marks and the second judge decides everything — not because he is better, but because his numbers are bigger. The fix is to stop asking for marks and ask each judge only who was first, second, third. Positions are comparable across judges; raw scores are not.
k is the whole dial: small k trusts the head of each list, large k trusts consensus. Sixty comes from Cormack, Clarke and Buettcher’s 2009 paper, where it was found empirically, and it has been the default in every engine since. Drag it to zero and watch first place steamroll everything — row A, first on one list and absent from the other, beats a row that both lists placed third.
The same five rows, fused three ways. Watch what each rule can and cannot see.
The rule that survives interviews: start with RRF; move to weighted fusion only once you have a labelled query set to tune α against, and re-tune it when the corpus changes. The one place weighted fusion wins from day one is when you already know a query type — “anything containing a ticket id” — should be keyword-led, and you set α per query type from a classifier rather than picking one α for everything.
| RRF — rank-based | Weighted — score-based | |
|---|---|---|
| Inputs | Positions only | Scores, rescaled per list |
| Parameters | k, default 60, rarely tuned | α or per-side weights — must be tuned |
| Sees a landslide | No | Yes |
| Stable as the corpus grows | Yes | Drifts — re-tune on a schedule |
| Needs an evaluation set | No | Yes, to choose α |
| Handles a missing list | Naturally — that list contributes zero | Naturally, but normalising a one-row list is degenerate |
| Default in | Elasticsearch rrf, Milvus RRFRanker, Qdrant
Fusion.RRF, the standard pgvector SQL pattern |
Weaviate since v1.24, Pinecone’s dense/sparse weighting |
No. Look at the RRF panel above: the identifier chunk, ranked 3 on dense and 1 on keyword, lost by 0.0003 to a general refund page ranked 1 and 2. RRF rewards both lists liking a row, and in that query both lists genuinely did like the general page.
What hybrid guarantees is that the exact-match chunk is in the fused list, which dense alone could not manage. Getting it to the top is a job for weighting, per-query-type routing, or a reranker — not for hoping RRF guesses your intent.
The single cheapest improvement in this entire document, and the one most often left at its default.
Every engine has the same coupling under a different name. hnsw.ef_search in pgvector, default 40. num_candidates in Elasticsearch, default 1.5×k. The limit on each Qdrant prefetch. The per-request limit in a Milvus AnnSearchRequest. And Elasticsearch’s rank_window_size defaults to the page size, so the fusion is only as deep as what you were going to display.
| Situation | Per-side depth for k = 10 | Why |
|---|---|---|
| Default | 50 | Rescues rows down to rank 50 on the weaker side at negligible cost |
| A reranker follows | 50–100, then rerank 30–50 | Reranker cost is per candidate; fusion depth is not |
| Filtered to a tiny tenant | whatever survives | A 138-row tenant cannot supply 100 — and brute force returns them all anyway |
Elasticsearch rrf | set 50–100 explicitly | rank_window_size defaults to the page size, so the fusion is only as deep
as what you were going to display |
Everything from document 14 does not stop at the dense index. If the query is scoped to tenant Z, that predicate must reach the keyword search too — and if it does not, the failure is not a recall bug.
Filter the dense arm and forget the sparse one, and rows from other tenants walk into the result set through the keyword list. Fusion can only reorder rows it was given; it cannot add scope. So filter both, then fuse.
In Postgres that means the same predicate in both CTEs. In Qdrant, the same filter object on
both prefetches. In Elasticsearch, the filter inside each retriever — and inside the
knn clause, so it stays a pre-filter. In Weaviate the where on a hybrid
call applies to both sides automatically; Milvus applies one expression across every
AnnSearchRequest.
Row-level security covers both arms for free, because it is the table that enforces it rather than the query — which is one more argument for the correctness ladder in document 14.
A filtered graph walk can strand: the neighbourhood contains no rows that pass the filter and the walk finds nothing. An inverted index has no such failure mode. The posting list for “refund” is intersected with the posting list for tenant Z, and the result is exactly the set of tenant-Z chunks containing “refund”, however small that is.
So for the 138-row tenant the keyword side is exact and instant, and it is the dense side that needs the brute-force fallback. In a tiered multi-tenant layout, hybrid improves the tail’s experience for free, because the half that never struggles with tiny selectivities is now in the loop.
Fusion can live in the engine or in your application, and the choice is really a question about how many stores you are running.
| In the engine | In the application | |
|---|---|---|
| When | One store holds both representations | Two stores — e.g. a dense-only vector database plus a separate search engine |
| Cost | One round trip, no candidate lists on the wire | Two round trips, two lists of ~100 ids and scores on the wire — a few kilobytes |
| Control | Whatever the engine exposes | Total — you can implement any fusion rule you like |
| The catch | — | Two systems, two consistency models, and a drift problem you now own |
| Engine | How hybrid is expressed |
|---|---|
| pgvector | A tsvector column with a GIN index next to the HNSW index. Two CTEs, a FULL
OUTER JOIN, and Σ 1/(60 + rank) written by hand. pg_search if
you want real BM25 rather than ts_rank |
| Qdrant | Named dense and sparse vectors; two prefetch clauses feeding a
FusionQuery of RRF or DBSF. An idf modifier on the sparse config
maintains corpus statistics inside the collection |
| Weaviate | hybrid(alpha=0.75), with relativeScoreFusion as the default
since v1.24 |
| Elasticsearch | An rrf retriever over standard, knn and
sparse_vector retrievers. rank_constant 60 — and raise
rank_window_size |
| Milvus | Dense plus SPARSE_FLOAT_VECTOR, a BM25 Function, and
hybrid_search with RRFRanker or WeightedRanker |
| Pinecone | A sparse-dense record scored by dot product, or a dense index and a sparse index fused in your application |
The number to hold in your head is a ratio: the sparse side is roughly one fifteenth of the dense side by storage. Hybrid does not double the problem.
The sparse side does not quantise. Everything in document 12 applies to the dense vectors and to nothing else — an inverted index is already a compressed integer structure. So as you compress the dense side the ratio moves against you: at scalar int8 the dense side is 29.2 GB per replica and the sparse side is still 7, which is a quarter rather than a fifteenth. Worth knowing before you quote “seven percent” in a design review for a quantised system.
BM25 fills the sparse vector by counting the words that are actually in the text. A learned sparse model — SPLADE, ELSER — runs a transformer over the text and fills the sparse vector with the terms it thinks should be there, weights included.
| BM25 | Learned sparse | |
|---|---|---|
| How the slots are filled | Counting, against corpus statistics | A model forward pass, per chunk and per query |
| Handles synonyms | No — “money back” and “refund” share nothing | Yes — it expands into related terms, which is the whole point |
| Cost at ingest | Effectively free | A second inference pass over the entire corpus |
| Cost at query | Effectively free | A model call on the query, in the hot path |
| Non-zero slots per chunk | ~40, the words present | Often 100–300 after expansion — a bigger index and longer posting lists |
| Drifts with the corpus | Yes — idf moves | Less so — the weights come from the model, not the corpus |
Reach for it when the per-slice evaluation shows a synonym gap: exact-token recall is fine, conceptual recall is fine, and the queries losing are the ones where the user’s vocabulary and the corpus’s vocabulary differ. That is precisely the gap BM25 cannot close and dense already covers, so check first that dense is not already covering it.
Do not reach for it as a default. It reintroduces exactly the costs hybrid was cheap for avoiding: a second model to version, a second inference bill at ingest, a model call in the query path, and a re-encode of the whole corpus when the model changes. BM25 has none of those and no version to pin.
Fusion merges two lists using positions or rescaled scores, and it never reads the text. A reranker reads both the query and the chunk, together, and scores relevance properly. They are different stages doing different jobs, and the engines that put them in one request have made this harder to see than it should be.
Why people confuse the two, and it is not their fault: engines put them in the same request and name them badly. Elasticsearch nests a text_similarity_reranker around an rrf retriever. Weaviate has a rerank clause on a hybrid query. Milvus calls its fusion classes “rankers” — RRFRanker, WeightedRanker — and now also ships model-based rerankers under the same name. Qdrant chains a prefetch fusion into a further query stage. The interview answer is to name the three stages and say which is which.
Cost is per candidate, and it is a model forward pass rather than a hash-map update. Thirty candidates at a few milliseconds each is 50 to 300 ms depending on the model and whether it is local or an API call. A hundred candidates is three times that.
Which sets the shape of the pipeline: fuse deep and rerank shallow. Fusion depth is nearly free, so take 50 or 100 per side; reranking depth is linear and expensive, so rerank the fused top 30 and no more. Every candidate you rerank beyond the point where the ordering stops changing is latency spent on nothing.
Resolving the fusion tie. The identifier chunk that RRF put second goes first, because a cross-encoder can see that the query is about that identifier.
Rescuing a weak first stage. If the retriever’s top 50 contains the answer but ranks it 30th, a reranker fixes the ordering — and if the answer is not in the 50, no reranker can help. Reranking cannot improve recall, only precision at the top.
Buying back the recall that quantisation and tight filters cost. A cheap, wide first stage plus a reranker is often better and faster than an expensive, precise first stage.
Everything above is really about one axis: when is the query allowed to meet the document? Answer that and the accuracy, the cost and the architectural position of every technique in this document fall out of it.
One spectrum, and the axis is when the query is allowed to meet the document. A bi-encoder never lets them meet — both sides are summarised alone, and the comparison is a dot product between two finished points. A cross-encoder lets them meet completely, and pays for it by having nothing it can precompute. Late interaction moves the meeting point in between: the document is encoded alone, but at token granularity, so the comparison at query time still has something to work with. Accuracy rises left to right; the amount you can precompute falls.
| Bi-encoder | Late interaction · ColBERT | Cross-encoder | |
|---|---|---|---|
| What is stored per chunk | One vector | One vector per token | Nothing — the chunk text |
| When the query meets it | Never — two finished points are compared | At scoring time, token against token | Inside the model, with full attention |
| Cost per candidate | A dot product | A MaxSim — arithmetic over stored vectors | A model forward pass |
| Can it be an index? | Yes — this is why ANN search exists | In principle; the storage arithmetic usually says no | Never — there is nothing to precompute |
| Where it belongs | First stage, over everything | A middle tier: rerank a shortlist of hundreds | Last stage: rerank a shortlist of tens |
Set the compression to float32 and the number stops being a design option. Three billion token vectors at 128 dimensions and four bytes each is over a terabyte — which is why ColBERTv2’s residual compression exists, and why the honest summary of late interaction is that it buys token-level matching at roughly the storage cost of the dense index all over again. Whether that is worth it is a measurement, not a principle: run it as a reranker on your own gold set and compare it against a cross-encoder on the same shortlist.
The full shape is: a bi-encoder plus BM25 retrieves thousands, late interaction reranks hundreds, and a cross-encoder reranks tens. Each stage is roughly an order of magnitude more expensive per candidate and roughly an order of magnitude narrower.
Most production systems run two tiers rather than three, because the middle one costs a second full index and the accuracy gain over “retrieve 50, cross-encode 30” is usually small. The case for the middle tier is a shortlist too large for a cross-encoder to afford — if you genuinely need to rank a thousand candidates and cannot spend a second doing it, late interaction is the technique that exists for that gap.
A single fusion setting is a compromise between query types that want different things. The step up is a cheap classifier in front of retrieval that picks the strategy per query — and a regex covers most of the value at zero latency.
| Query | Class | Strategy |
|---|---|---|
INV-20419 | Pure identifier | A metadata filter if the id is extractable at ingest; otherwise BM25 only |
| “refund INV-20419” | Mixed | RRF, depth 50, rerank the top 20 |
| “how do I get my money back” | Conceptual | Dense-led: α = 0.85, or dense-only plus a reranker |
| “ECONNRESET postgres 5432” | Exact-token | Keyword-led: α = 0.3, RRF k = 20 |
If the chosen side returns fewer than k results, fall through to the other side. A pure-identifier query whose id was mistyped should still get the semantically nearest chunks rather than an empty list.
RRF handles this for free, because a missing list simply contributes zero. A routed keyword-only path needs the fallback written explicitly, and it is exactly the sort of thing that gets left out and then found by a customer.
α does not have to be a constant. Every engine that exposes it does so per request
— Weaviate’s alpha, Milvus’s WeightedRanker weights,
Elasticsearch’s linear retriever weights, Qdrant’s prefetch weights — and they
are designed to be driven this way. Most teams set it once in a config file and never touch it
again, which is leaving the feature on the table.
One recall number hides exactly the thing hybrid is for. Split the evaluation queries by type and measure each retriever on each slice — the shape of this table is the answer to “how did you know hybrid was worth it?”
| recall@10 | conceptual n=400 | exact-token n=250 |
mixed n=350 | overall n=1,000 |
|---|---|---|---|---|
| dense only | 0.88 | 0.41 | 0.72 | 0.71 |
| sparse only, BM25 | 0.52 | 0.93 | 0.68 | 0.68 |
| RRF hybrid, depth 50 | 0.87 | 0.91 | 0.84 | 0.87 |
| RRF hybrid, depth 10 | 0.84 | 0.85 | 0.76 | 0.82 |
| weighted, α = 0.7 | 0.89 | 0.79 | 0.81 | 0.84 |
Illustrative numbers — the shape is what matters. Dense wins conceptual, sparse wins exact-token, hybrid is near the best of both on each slice and best overall. Two diagnostics fall straight out of it.
Hybrid worse than dense on conceptual queries means the depth is too shallow or α is too keyword-heavy.
Hybrid worse than sparse on exact-token queries means α is too dense-heavy, or RRF is letting a semantic favourite outvote the identifier match — the tie from section 5. The fix there is a reranker or a per-type α, not a different global constant.
| Slice | How to detect it from query logs | Typical share |
|---|---|---|
| Exact-token | Contains a token matching an id regex, an error code, a SKU, a proper name, or an all-caps acronym | 15–35% |
| Conceptual | No such token; question words; five or more content words | 40–60% |
| Mixed | Both | the rest |
Sparse retrieval is corpus-dependent in a way dense is not. The idf of every term is a function of how many chunks contain it, so a term that was rare and decisive becomes common and worthless as the corpus grows.
A product name in 12 of 2 million chunks has an idf of about 12.0. The same name in 40,000 of 10 million has an idf of about 5.5 — its weight more than halved, and a query for “ProjectAtlas billing” is now decided by “billing”. No error fires, and the dense half keeps returning plausible-looking results, so nobody notices.
Where the idf lives decides whether that drift is smooth or stepped. Postgres computes
ts_rank per query from the current table. Lucene computes BM25 per shard segment,
so idf differs across shards until merges settle — which is one more reason RRF works on
ranks. Qdrant computes it from a fixed model-side table unless you enable the idf modifier.
Milvus maintains corpus statistics per collection and updates them on write. Ask which one
your engine does.
| Practice | Cadence | What it catches |
|---|---|---|
| Re-run the per-slice evaluation | Weekly, and after any bulk load | Both kinds of drift — the only true signal |
| Track exact-token recall as its own SLO | Continuous, from a fixed query set | idf drift on identifiers specifically |
| Track the fraction of fused results that came from one side only | Continuous, from logs | One side silently returning junk, or nothing at all |
The embedding model is fixed until you change it, so a chunk’s dense vector never moves. What drifts on the dense side is the distribution: new documents cluster in a region the graph was not built around. Dense drift is a graph-quality problem; sparse drift is a scoring problem. One is fixed by a rebuild and the other by re-deriving statistics, and conflating them wastes a week.
Six questions, in order, and the first one can end the conversation.
| Exact-token share of queries | Recommendation |
|---|---|
| < 5% | Dense only. Revisit in six months |
| 5–15% | Dense plus BM25 with RRF at depth 50 — cheap insurance |
| 15–40% | Hybrid is load-bearing. Per-type α or a reranker |
| > 40% | The corpus is a lookup table. Consider keyword-first with dense as the fallback, or structured metadata filters instead of search |
A WHERE invoice_id = … beats any ranker, every time.
Do not build a ranker for something a filter answers exactly.
Under about 80 tokens, BM25 has nothing to count and the length normalisation flattens everything.
Re-chunk first, or count the parent.
Hybrid itself is cheap; a reranker is not. And hybrid without a reranker leaves the fusion tie unresolved.
Measure — the sparse side alone is usually inside budget.
A dense-only vector database plus a separate search engine means two systems, two consistency models and app-side fusion.
Pick one engine that does both, or split deliberately.
| Decision | Default | Move it when |
|---|---|---|
| Sparse producer | BM25 — tsvector or engine-native |
Evaluation shows a synonym gap → learned sparse |
| Fusion | RRF, k = 60 | You have labels and want landslides to count → weighted |
| Per-side depth | 50 for a final 10 | A reranker is present → 50–100, then rerank 30 |
| Chunk size | 200–400 tokens, title prepended | Dense-only was tuned to 100 → move up, or parent–child |
| Filters and tenant | The same predicate on both arms, fused after | Never |
| Fusion location | Inside the engine | Two stores, or you are experimenting with fusion rules |
| Evaluation | Per-slice recall@10 and MRR, weekly | Never less often. More often after a bulk load |
Storage: dense 95.5 GB per replica, sparse ~7 GB, so ~101 GB per replica and about 303 GB at replication factor 3. Seven percent more resident memory, one to ten milliseconds on p50, and zero re-embedding.
Query path: tenant bitmap → HNSW at efSearch 100, take 50 — and in parallel the inverted index intersected with the tenant list, take 50 → RRF k=60 over the union → optionally rerank the top 30 → return 10.
| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| A search for a ticket id returns ten unrelated tickets | The keyword arm is not running, or the id was tokenised away | Whether the sparse index exists; what the analyser did to the id; whether idf on that token is still high | Check tokenisation first — a hyphenated id split into two common words is the usual cause. Then depth, then a reranker |
| Hybrid is no better than dense on the overall number | The overall number is the wrong measurement | Per-slice recall, especially the exact-token slice | Report per slice. If exact-token really did not move, the sparse arm is not working |
| Hybrid is worse than dense on conceptual queries | Depth too shallow, or α too keyword-heavy | Per-side depth against the engine’s candidate parameter | Raise depth to 50 first — it is nearly free — then revisit α |
| Results include rows from another tenant | The filter was applied to one arm only | Both arms of the query, and whether RLS covers the sparse path | Filter both, fuse after. This is a leak, not a tuning problem |
| Identifier queries got worse over a year, nothing was deployed | idf drift — the term is no longer rare | The idf of the terms in the failing queries, then and now | Nothing to fix in the code. Track exact-token recall as an SLO so it is visible |
| Keyword results are good in English and useless elsewhere | One stemmer configuration over a mixed-language corpus | The analyser configuration against the actual language distribution | Per-language columns, the simple configuration, or a multilingual learned sparse model |
| The dense list is always exactly 40 rows | hnsw.ef_search is capping it below your LIMIT |
The engine’s candidate parameter against the depth you asked for | Raise it to at least the fusion depth |
| Adding a reranker did not improve recall | Working as designed — reranking cannot improve recall | Whether the answer was in the fused candidate list at all | If it was not, fix retrieval or depth. A reranker only fixes ordering |
| p99 latency tripled after adding the reranker | Reranking depth, or a cold model | How many candidates are reranked, and where the model runs | Rerank 30, not 100. And measure the ordering change per depth — it usually stops moving well before 50 |
| Weighted fusion worked at launch and drifted | Normalisation depends on the returned list, and the corpus moved | When α was last tuned, and against what | Re-tune on a schedule, or move back to RRF, which does not drift |
ArchitectWhat is hybrid retrieval and why would you use it?
Running a dense vector search and a keyword search over the same corpus and merging the two result lists. The reason is that they fail on different queries: dense retrieval is recall on meaning and sparse retrieval is precision on tokens. “Get my money back” shares no word with a chunk that says “refund”, so the keyword side returns nothing; “INV-20419” is one token the embedding model has barely seen, so the dense side returns forty plausible invoice chunks and no way to tell them apart.
The thing that makes it easy to adopt is that it is purely additive. The HNSW graph, its parameters, its filters and its tenant boundary are unchanged, and the answer to “what would we have to re-embed?” is nothing. On our stack it is about seven percent more resident memory and a few milliseconds.
And I would want to justify it with a number rather than a principle: count the share of real queries carrying a token that must match exactly. Under five percent, dense alone is the right answer and saying so is the mature call.
ArchitectExplain reciprocal rank fusion, and why it is the default.
RRF scores each document as the sum over lists of one over k plus its rank, with k conventionally 60. It uses positions and throws the scores away entirely.
It is the default because the scores are not comparable. Cosine similarity sits between about 0.70 and 0.92; BM25 is unbounded and routinely in the tens. Add them and the keyword side decides everything, not because it is better but because its numbers are bigger. Positions are the one thing both lists express on the same scale.
The k constant is the whole dial, and the flatness is deliberate: at k=60 first place is 0.0164 and tenth is 0.0143, a fifteen percent spread across ten positions. That is what stops one list’s favourite from steamrolling a document both lists ranked third. Small k trusts the head of each list; large k trusts consensus.
ArchitectWhen would you use weighted fusion instead?
When I have a labelled query set to tune α against, and when landslides matter. RRF cannot see that the keyword side’s first place scored 34.6 against a second-place 9.4 — it sees rank 1 and rank 2. Score fusion normalises each list and can see the gap.
The costs are real though. Normalisation depends on the min and max of the returned list, so five near-identical scores get stretched to zero-to-one exactly like a landslide does. And α drifts as the corpus and the query mix change, so it needs re-tuning on a schedule where RRF needs none.
Where weighted fusion wins from day one is per-query-type α: if I know that anything containing a ticket id should be keyword-led, a classifier setting α per query beats any single global value. Every engine exposes the weight per request precisely because it was meant to be driven that way.
ArchitectHow deep do you fetch on each side, and why does it matter?
Fifty per side for a final ten, and it matters more than almost any other setting because it is nearly free. Consider a document at keyword rank 3 and dense rank 40. At depth ten it is absent from the dense list, so it gets one vote and loses to something worse that appeared on both. At depth fifty it gets both votes and wins — and the user still sees ten.
The cost on the dense side is that the walk was going to return efSearch candidates anyway; a hundred is under five million multiply-adds, sub-millisecond. On the sparse side the posting-list scan already scored every matching row, so a deeper result is a bigger heap and not more work.
The trap is that every engine couples depth to a parameter that defaults too low.
In pgvector, if hnsw.ef_search stays at 40 and you LIMIT 100, you get 40 rows and a
silently shallow dense list. Elasticsearch’s rank_window_size defaults to the
page size, so the fusion is only as deep as what you were going to display.
ArchitectDoes hybrid guarantee the exact-match chunk comes first?
No, and I would want to be precise about what it does guarantee. In the worked example the identifier chunk was ranked 3 on dense and 1 on keyword, and it lost by 0.0003 to a general refund page ranked 1 and 2 — because RRF rewards both lists liking a row, and both lists genuinely did like the general page.
What hybrid guarantees is that the exact-match chunk is in the fused list, which dense alone could not do. Getting it to the top is a different job: a per-query-type weight, or a reranker.
That distinction is worth stating clearly because it maps onto the three stages. Fusion is what gets the chunk into the candidate set; reranking is what puts it on top.
ArchitectWhat is the difference between fusion and reranking?
Fusion merges two ranked lists using positions or rescaled scores, and it never reads the text — it is a hash-map update, microseconds. A reranker is a cross-encoder that reads the query together with each chunk, with full attention in both directions, and scores relevance properly — a model forward pass per candidate, fifty to three hundred milliseconds for thirty of them.
So they sit in a fixed order: retrieve fifty per side, fuse to at most a hundred, rerank the top thirty. Fusion depth is nearly free so you go wide; reranking is linear in candidates so you go narrow.
People conflate them because engines put them in the same request and name them badly — Milvus calls its fusion classes “rankers”, Elasticsearch nests a reranker retriever around an rrf retriever. Naming the three stages is usually enough to show you understand the distinction.
ArchitectWhat is a cross-encoder, and why can you not just index with one?
A cross-encoder concatenates the query and the chunk and runs one model over both, so every token of the query can attend to every token of the chunk. Nothing was summarised in advance, so nothing was lost in advance, and it is the most accurate relevance signal available.
And that is exactly why you cannot index with one: there is nothing to precompute. The score does not exist until the pair exists, so scoring ten million chunks means ten million forward passes per query. A bi-encoder can be indexed precisely because it gives up the interaction — each side is summarised alone and the comparison is a dot product between two finished points.
So the axis to reason on is when the query is allowed to meet the document. Never, for a bi-encoder. Completely, inside the model, for a cross-encoder. Accuracy rises along that axis and the amount you can precompute falls, which fixes each technique’s place in the pipeline.
ArchitectWhere does late interaction fit?
In between. ColBERT encodes each token of the chunk into its own small vector at index time, and at query time scores with MaxSim: every query token finds its best match anywhere in the chunk, and the score is the sum of those best matches. So an identifier can match one token deep inside a long chunk without being averaged away by the other 299 — which is exactly what a single pooled vector cannot do.
The reason it is usually a reranker rather than an index is arithmetic. Ten million chunks at three hundred tokens is three billion token vectors; at 128 dimensions and float32 that is over a terabyte. ColBERTv2’s residual compression brings it to roughly 32 bytes a vector, so about 96 GB — feasible, but that is the dense index again on top of the dense index.
So the honest positioning is a middle tier: cheaper per candidate than a cross-encoder, because a MaxSim is arithmetic over stored vectors rather than a forward pass, and far more expensive per chunk to store. Most teams run two tiers rather than three, and the case for the middle one is a shortlist too large for a cross-encoder to afford.
ArchitectHow do filters and multi-tenancy interact with hybrid?
The same predicate has to reach both arms, and if it does not, the failure is a leak rather than a recall problem — rows from another tenant walk in through the keyword list. Fusion can only reorder rows it was given; it cannot add scope. So filter both, then fuse.
There is a nice asymmetry worth raising. A filtered graph walk can strand, because the neighbourhood may contain no rows that pass the filter. An inverted index cannot: the posting list for the term is intersected with the posting list for the tenant, and the result is exactly right however small it is. So for a 138-row tenant the keyword side is exact and instant, and it is the dense side that needs a brute-force fallback.
Which means hybrid improves the long tail’s experience for free in a tiered multi-tenant layout — the half that never struggles with tiny selectivities is now in the loop.
ArchitectYour identifier searches worked at launch and are worse a year later. Nothing was deployed.
Almost certainly idf drift. The keyword side’s weights are a function of how many chunks contain each term, so a product name that was in twelve of two million chunks — idf around 12 — can be in forty thousand of ten million a year later, at an idf around 5.5. Its weight more than halved, and the query is now decided by whatever common word sits next to it.
Nothing errors, and the dense half keeps returning plausible-looking results, which is why nobody notices. The detector is a fixed exact-token query set tracked as its own SLO — the overall recall number will not move.
I would also check where the idf actually lives, because it decides whether the drift is smooth or stepped: Postgres computes it per query from the current table, Lucene per shard segment until merges settle, Qdrant from a fixed model-side table unless the idf modifier is on, and Milvus per collection updated on write.
Eng managerA team with a dense-only RAG wants to add hybrid next sprint. What do you ask for?
One number before anything else: the share of real queries in the logs that contain a token which has to match exactly. That takes an afternoon with a regex and it decides whether this is worth a sprint at all. Under five percent, the honest answer is no.
Then a per-slice evaluation set, before the implementation rather than after — conceptual, exact-token and mixed, with labels. Without it there is no way to tell whether hybrid helped, because the overall average is exactly the number that hides the improvement.
And I would set the expectation that the sprint is mostly plumbing rather than tuning: a second column, a second index, the same filters on both arms, depth 50 and RRF at 60. The interesting decisions — α, routing, a reranker — all come after there is a measurement to make them against.
Eng managerHow do you decide whether to spend the latency budget on a reranker?
By asking what problem it solves, because a reranker fixes ordering and cannot fix recall. If the answer is not in the fused candidate list, no reranker will find it — and that is a retrieval or depth problem which is far cheaper to fix.
So the sequence is: measure recall at the fusion depth first. If recall@50 is high and recall@10 is poor, the ordering is the problem and a reranker is exactly right. If recall@50 is also poor, spend the money on retrieval instead.
Then it is a straightforward budget conversation: fifty to three hundred milliseconds for thirty candidates, against whatever headroom the product has. I would also ask for the depth curve — how much the ordering actually changes between reranking 20, 30 and 50 — because it usually stops moving well before 50, and that is latency we can hand back.
Eng managerTwo engineers disagree: one wants learned sparse, the other wants a reranker. How do you settle it?
With the per-slice table, because they fix different failures. Learned sparse closes a synonym gap on the keyword side — the user says “money back” and the corpus says “refund”. A reranker fixes ordering in a candidate list that already contains the answer. If the evaluation does not show which of those is failing, neither proposal is ready.
I would also weigh what each one commits us to. A reranker is a model in the query path with a latency cost and no ingest cost. Learned sparse is a model in the ingest path with a version to pin, a corpus to re-encode when it changes, and a query-time call as well. The second is a much bigger standing commitment for a gap the dense side may already be closing.
And I would check the cheap options first, out loud, so the team sees the order: depth, then per-type routing, then a reranker, then learned sparse. Two of those four cost nothing.
Is sparse the same as BM25?
No — they are different layers. Sparse is the representation: a vocabulary-long vector that is mostly zeros. BM25 is one way to fill it, by counting terms against corpus statistics. Learned sparse — SPLADE, ELSER — is another way, using a model. Keeping those three words in their own rows is the answer to this question.
Can one model output both dense and sparse?
A few multi-output models do — BGE-M3 emits dense, sparse and multi-vector heads from one forward pass. It is still two representations, two columns and two indexes; what is shared is the model call. Convenient, and it does not change any of the architecture in this document.
If I already have a pgvector table with the chunk text, what do I need to add?
A tsvector column, a GIN index over it, and a query that runs two
CTEs and joins them with Σ 1/(60 + rank). Postgres computes the tsvector from
the text you already store, so there is no second pipeline to build and nothing to re-embed. If
you want real BM25 rather than ts_rank, that is what pg_search is
for.
Why is k in RRF 60 and not something tuned?
Because it came out of the 2009 paper empirically and has behaved well enough everywhere since that no engine has moved it. It is worth knowing what moving it does though: smaller k sharpens the ladder so a first place dominates, larger k flattens it so agreement dominates. If you deliberately want the keyword side’s top hit to win outright on identifier queries, a small k on that route is a legitimate knob.
Does hybrid help with the tiny-tenant recall problem?
Yes, for free, and it is a nice thing to raise unprompted. A filtered graph walk can strand on a 138-row tenant and needs a brute-force fallback; an inverted index just intersects two posting lists and returns exactly the right set. The keyword half never has the selectivity problem that the dense half does.
Where should the fusion actually run?
Inside the engine, if one engine holds both representations — one round trip, no candidate lists on the wire. In the application if you are running two stores, and then you own the consistency gap between them as well as the fusion. App-side fusion is about ten lines of code; the two-store consistency problem is not ten lines.
Can reranking replace hybrid?
No, and the reason is worth internalising: a reranker can only reorder what retrieval handed it. If the identifier chunk never entered the candidate list because the dense side ranked it 400th, the reranker never sees it. Hybrid fixes what gets retrieved; reranking fixes what gets ranked first. Skipping the first and buying the second is the most common way to spend money and get nothing.
Is a bigger reranker always better?
Not per millisecond. Reranking cost is linear in candidates and roughly linear in model size, and the quality curve flattens — most of the gain is in going from no reranker to a small one. The measurement to run is the ordering change as a function of depth and model size on your own gold set; teams routinely find that reranking 20 with a small model matches reranking 50 with a large one at a fraction of the latency.
Does quantisation interact with hybrid?
Only through the ratio. Everything in document 12 applies to the dense vectors and to nothing else — an inverted index is already a compressed integer structure and does not quantise. So the sparse side is one fifteenth of an unquantised dense side and about a quarter of an int8 one. Worth knowing before quoting “hybrid costs seven percent” for a quantised system.
How would you prove hybrid helped?
Per-slice recall@10 and MRR, measured on retrieval alone rather than end-to-end. A better retriever with the same LLM is the only way to attribute a change to the retriever rather than to the prompt. And report the slices separately — the overall average is precisely the number that hides the exact-token improvement hybrid was built for.
| Fact | Value |
|---|---|
| Sparse side against dense, unquantised | ~1/15 by storage |
| Sparse side against dense, at int8 | ~1/4 |
| Hybrid on the reference stack | +7% resident, +1–10 ms, 0 re-embedding |
| RRF constant | 60 |
| RRF spread, rank 1 to rank 10 at k=60 | 0.0164 to 0.0143, ~15% |
| Per-side fetch depth for k=10 | 50 |
| Rerank depth | 30 of the fused list |
| Exact-token share below which hybrid is not worth it | 5% |
| ColBERT token vectors on the reference stack | 3 billion · ~96 GB compressed |
| idf of a term in 1 chunk of 10M / in 50,000 | 15.7 / 5.3 |
“Dense and sparse retrieval fail on different queries, so hybrid runs both and merges the lists. Dense is recall on meaning; sparse is precision on tokens. It is purely additive — nothing gets re-embedded, the sparse side is about a fifteenth of the dense side by storage, and it costs a few milliseconds.
You cannot add the two scores, because cosine sits between 0.7 and 0.9 and BM25 is unbounded — the keyword side would decide everything for no good reason. So you fuse on positions: reciprocal rank fusion, one over sixty plus rank, summed across lists. The flatness of that ladder is deliberate; it makes a document both lists ranked third beat one that is first on one list and missing from the other.
Two things I would raise unprompted. Fetch depth: fuse fifty per side, not ten, because a document at dense rank 40 and keyword rank 3 never enters the fusion otherwise — and depth is nearly free while every engine defaults it too low. And filters apply to both arms: forget the sparse one and rows from another tenant walk in through the keyword list, which is a leak rather than a tuning problem.
Then reranking is a third stage, not fusion. Fusion never reads the text; a cross-encoder reads the query and the chunk together with full attention, which is why it is the most accurate signal available and why it can never be an index — there is nothing to precompute. So fusion gets the identifier chunk into the list and reranking puts it on top. Late interaction sits between the two: token-level vectors stored at index time and MaxSim at query time, normally deployed as a middle reranking tier because storing one vector per token is the dense index all over again.”
| Thread from this document | Resolved in |
|---|---|
| Chunk size, titles and parent–child, which the sparse side changes | 01 · Chunking foundations |
| What an embedding model is doing on the dense side | 05 · Embedding models |
| efSearch and k, which fetch depth is really spending | 11 · Parameters and tuning |
| Why the sparse side does not quantise, and what that does to the ratio | 12 · Quantisation and capacity |
| Why per-shard fetch depth and fusion depth are the same conversation | 13 · Sharding and replication |
| Filters, tenants, and the leak that fusion cannot repair | 14 · Filtered search and multi-tenancy |
| Per-slice evaluation, gold sets and the CI gate | 16 · Evaluation and observability |