TL;DR — There are two families of retriever and they fail on different queries, which is the whole reason hybrid search exists. Lexical (BM25) matches words: unbeatable on identifiers, blind to paraphrase — it scored exactly 0.0 on both a reworded question and a synonym. Dense matches meaning: handles paraphrase, and cannot separate
ERR-4012fromERR-4021, whose vectors are near-identical. Each scored 2/3 below; fusing them scored 3/3. But hybrid is not free: naive Reciprocal Rank Fusion also scored 2/3, because a retriever with no signal at all still returns a full ranked list, and its arbitrary first entry casts a rank-1 vote that overrode the retriever that was right. Dropping zero-score results before fusing fixes it. It stops being about the retriever when relevance isn't similarity at all — freshness and authority are metadata problems no ranker solves.
1. Simple explanation
Once documents are chunked and indexed, something has to decide which chunks a question needs. There are two fundamentally different ways to do it.
Match the words. If the question says "refund" and a chunk says "refund", that's evidence. This is keyword search, and its modern form is BM25 — a scoring function that weights rare words more heavily and stops long documents from winning just by being long. It's decades old, extremely fast, and completely literal.
Match the meaning. Turn the question and the chunks into vectors positioned so that similar meanings sit close together, then rank by distance. This handles "can I send this back?" finding a passage about returns with no shared words at all.
Neither is better. They are good at different things, and — critically — they are bad at different things, which is what makes combining them worth the trouble.
Analogy — searching a library by index card versus by shelf. The card catalogue finds an exact title instantly and is useless if you can't remember the wording. Wandering to the topic shelf finds books about what you want and can't distinguish two nearly identical titles. A librarian uses both, and the useful part of the analogy is why: not because two methods are better than one in general, but because these two specifically fail on non-overlapping cases.
2. Diagram
TWO FAMILIES, TWO BLIND SPOTS
LEXICAL (BM25) DENSE (embeddings)
────────────────────────── ──────────────────────────
matches WORDS matches MEANING
+ exact identifiers, codes + paraphrase, synonyms
+ rare terms, names + cross-lingual
+ fast, no model needed + no shared words needed
- paraphrase -> score 0.0 - 'ERR-4012' vs 'ERR-4021'
- synonyms -> score 0.0 near-identical vectors
MEASURED — 3 queries, 4 documents
query want BM25 dense RRF RRF+drop0
'can I give something back' d1 d1 d1 d1 d1
'ERR-4012' d3 d3 d2 x d3 d3
'how long is shipping' d4 d1 x d4 d1 x d4
BM25 2/3
dense 2/3
RRF naive 2/3 <- combining them naively bought NOTHING
RRF drop-zero 3/3 <- ...until you drop the no-signal ranking
THE TRAP
BM25 for 'how long is shipping':
d1: 0.00 d2: 0.00 d3: 0.00 d4: 0.00
^
every score is ZERO -- no term matched anything.
But it still returns a full ORDERED list, so naive RRF
counts d1 as a rank-1 vote and outranks the dense
retriever that actually knew the answer.
-> filter out zero-score results BEFORE fusing.
RECIPROCAL RANK FUSION — why fuse by rank, not score
BM25 score 2.21 unbounded, corpus-dependent
cosine 0.94 always 0..1
^
not comparable. RRF uses only the ORDER:
score(d) = sum of 1 / (k + rank_in_that_list)
3. How it works
3.1 BM25, briefly
BM25 scores a document against a query by summing, over the query's terms, three ideas:
- Term frequency — more occurrences is more evidence, with diminishing returns (the
k1parameter controls how fast it saturates). - Inverse document frequency — a term appearing in few documents is more informative than one appearing everywhere.
- Length normalisation — divide out document length so long documents don't win by containing more of everything (
bcontrols how aggressively).
It needs no model, no training, and no GPU, and it is very hard to beat on queries where the user's words match the document's words. Search Systems covers it as a systems problem.
3.2 Where each one fails, precisely
Lexical fails on vocabulary mismatch. In §4, "can I give something back" shares no content word with the returns document, and "how long is shipping" fails because the document says Delivery. Both score exactly 0.0 — not a low score, no signal whatsoever.
Dense fails on near-identical strings with different meanings. ERR-4012 and ERR-4021 are, to an embedding model, almost the same token sequence in almost the same context. Their vectors are nearly identical, so the retriever cannot tell them apart — while operationally they are unrelated errors. Measured: dense ranked the wrong error code first, at cosine 1.000.
This is the important asymmetry. Dense retrieval's failure comes with a high score, so a relevance threshold gives you no protection at all — see Vector Search for Retrieval §3.6.
3.3 Hybrid search, and why you fuse by rank
Run both retrievers and combine. The obvious approach — add the scores — doesn't work, because BM25 scores are unbounded and corpus-dependent while cosine sits in a fixed range. Any weighting you pick is arbitrary and shifts when the corpus changes.
Reciprocal Rank Fusion sidesteps this by throwing the scores away and using only position:
rrf_score(d) = sum over rankings of 1 / (k + rank(d)) k ~= 60
A document ranked first anywhere gets a big contribution; one ranked twentieth everywhere gets little. No scale reconciliation is needed, and k damps the difference between the top few positions so a single confident retriever can't dominate entirely.
3.4 The RRF trap: no signal is not the same as low rank
Here is what the measurement caught, and it is not obvious.
For "how long is shipping", BM25 matched nothing — every document scored 0.00. But sorted() still returns a complete ordered list, so d1 sat in position 1 and RRF counted it as a rank-1 vote. That vote outranked the dense retriever, which had correctly found d4.
The result: naive RRF scored 2/3, exactly the same as either retriever alone. All the machinery of hybrid search, and no improvement whatsoever.
The fix is one line — drop zero-score entries before fusing — and it takes the same setup to 3/3. Generalising: a retriever that has no signal must abstain, not rank. Any fusion that treats "I have no idea, here's an arbitrary order" as equivalent to "here is my confident ordering" will be corrupted by it.
3.5 The rest of the toolbox
Beyond the two families, three techniques worth knowing:
| technique | what it does | when |
|---|---|---|
| Reranking | a slower, more accurate model re-scores a shortlist | you retrieved generously and need precision |
| Query expansion | add synonyms or a generated paraphrase before retrieving | vocabulary mismatch is your main failure |
| MMR | trade some relevance for diversity among results | top-k keeps returning near-duplicate chunks |
The standard production shape is: retrieve wide with hybrid, then rerank to a few. You get recall from the cheap wide pass and precision from the expensive narrow one — see Reranking: The Second Pass That Decides What the Model Sees.
3.6 What no retriever can fix
Similarity — lexical or semantic — has no notion of currency or authority. A superseded document scores exactly as well as its replacement, and will be returned confidently. That is a metadata problem: date fields, version filters, and deleting obsolete content.
Nor can retrieval answer questions that require aggregation ("how many contracts expire this quarter?") — that's a database query wearing a sentence. And if the answering passage was destroyed at chunk time, no retriever recovers it; see Chunking Strategies.
4. The math
4.1 BM25
score(D, Q) = sum over terms t in Q of
IDF(t) * ( tf(t,D) * (k1 + 1) )
---------------------------------------------
tf(t,D) + k1 * (1 - b + b * |D| / avgdl)
IDF(t) = log( 1 + (N - df(t) + 0.5) / (df(t) + 0.5) )
k1 ~= 1.2-2.0 how fast term frequency saturates
b ~= 0.75 how strongly to normalise for length
4.2 Reciprocal Rank Fusion
rrf(d) = sum over rankings r of 1 / (k + rank_r(d)) k ~= 60
uses ORDER only -> no need to reconcile incompatible score scales
4.3 Worked example
Four documents: a returns policy, two nearly identical error-code entries, and a delivery note. Three queries, each designed to expose one retriever's blind spot.
query want BM25 dense RRF RRF+drop0
'can I give something back' d1 d1 d1 d1 d1
'ERR-4012' d3 d3 d2 x d3 d3
'how long is shipping' d4 d1 x d4 d1 x d4
method correct
BM25 2/3
dense 2/3
RRF naive 2/3
RRF drop-zero 3/3
Each retriever alone gets 2 of 3, and they miss different ones — which is exactly the precondition for fusion to help.
The failures, in detail:
BM25 on a paraphrase: [('d1', 0.0), ('d2', 0.0), ('d3', 0.0)]
no content word is shared with d1. Lexical has literally nothing.
dense on an identifier: [('d2', 1.0), ('d3', 1.0), ('d4', 0.152)]
ERR-4012 and ERR-4021 are near-identical vectors. One character
flips the meaning and the embedding cannot see it.
Note dense's scores: 1.000 and 1.000. It isn't uncertain between the two error codes — it is maximally confident about both, and picks the wrong one. No threshold catches that.
4.4 The trap, measured
BM25 for 'how long is shipping': [('d1', 0.0), ('d2', 0.0), ('d3', 0.0), ('d4', 0.0)]
Every score zero, and yet a complete ordered list. Naive RRF gave d1 a rank-1 vote and it beat d4, which dense had ranked first correctly.
So naive RRF scored 2/3 — no better than either retriever alone. Hybrid search delivered nothing until the no-signal ranking was excluded, at which point it delivered 3/3. The lesson is that hybrid is a real technique with a real gotcha, not a free upgrade.
5. Real code
"""BM25, dense, and hybrid: each wins on different queries -- and RRF has a trap."""
import math
from collections import Counter
DOCS = {
"d1": "Returns policy: unwanted purchases may be sent to us within 30 days "
"for a full refund.",
"d2": "Error ERR-4021 means the payment gateway timed out. Retry the transaction.",
"d3": "Error ERR-4012 means the card was declined by the issuing bank.",
"d4": "Delivery normally takes three to five working days across the country.",
}
# Toy 'embeddings': topic weights, standing in for a real encoder.
# [returns, errors, delivery]
VEC = {
"d1": [0.95, 0.05, 0.15],
"d2": [0.05, 0.94, 0.10],
"d3": [0.05, 0.93, 0.08],
"d4": [0.10, 0.05, 0.96],
}
QUERIES = {
# paraphrase: shares NO content word with d1 -> lexical has nothing to match
"can I give something back": ("d1", [0.92, 0.06, 0.14]),
# exact identifier: d2 and d3 are semantically identical -> dense cannot separate
"ERR-4012": ("d3", [0.05, 0.93, 0.09]),
# 'shipping' never appears in d4, which says 'Delivery' -> lexical blind again
"how long is shipping": ("d4", [0.08, 0.05, 0.93]),
}
STOP = {"can", "i", "this", "is", "the", "how", "a", "to", "us", "for", "be", "may"}
def toks(s):
return [w.strip(".,:").lower() for w in s.split()
if w.strip(".,:").lower() not in STOP]
# ---- BM25 (lexical) ------------------------------------------------------
N = len(DOCS)
DF = Counter()
for d in DOCS.values():
for w in set(toks(d)):
DF[w] += 1
AVGDL = sum(len(toks(d)) for d in DOCS.values()) / N
def bm25(query, k1=1.5, b=0.75):
scores = {}
for did, text in DOCS.items():
tf = Counter(toks(text))
dl = len(toks(text))
s = 0.0
for w in toks(query):
if w not in tf:
continue
idf = math.log(1 + (N - DF[w] + 0.5) / (DF[w] + 0.5))
s += idf * (tf[w] * (k1 + 1)) / (tf[w] + k1 * (1 - b + b * dl / AVGDL))
scores[did] = s
return sorted(scores.items(), key=lambda kv: -kv[1])
# ---- dense (semantic) ----------------------------------------------------
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
na = math.sqrt(sum(x * x for x in a))
nb = math.sqrt(sum(x * x for x in b))
return dot / (na * nb)
def dense(qvec):
return sorted(((d, cosine(qvec, v)) for d, v in VEC.items()), key=lambda kv: -kv[1])
# ---- hybrid: reciprocal rank fusion -------------------------------------
def rrf(*rankings, k=60, drop_zero=False):
"""Fuse by RANK, not score -- BM25 and cosine are not on the same scale.
drop_zero matters: a retriever with NO signal still returns a full ranking,
and its arbitrary order otherwise casts real votes."""
fused = Counter()
for ranking in rankings:
rows = [(d, s) for d, s in ranking if s > 0] if drop_zero else list(ranking)
for rank, (did, _s) in enumerate(rows, start=1):
fused[did] += 1 / (k + rank)
return sorted(fused.items(), key=lambda kv: -kv[1])
print(f"{'query':<28} {'want':>5} {'BM25':>7} {'dense':>7} {'RRF':>7} {'RRF+drop0':>11}")
hits = {"BM25": 0, "dense": 0, "RRF naive": 0, "RRF drop-zero": 0}
for q, (want, qv) in QUERIES.items():
b, d = bm25(q), dense(qv)
got = {"BM25": b[0][0], "dense": d[0][0],
"RRF naive": rrf(b, d)[0][0],
"RRF drop-zero": rrf(b, d, drop_zero=True)[0][0]}
for m, g in got.items():
hits[m] += (g == want)
mk = lambda g: f"{g}{'' if g == want else ' x'}"
print(f"{q!r:<28} {want:>5} {mk(got['BM25']):>7} {mk(got['dense']):>7} "
f"{mk(got['RRF naive']):>7} {mk(got['RRF drop-zero']):>11}")
print(f"\n{'method':<16} {'correct':>8}")
for m, n in hits.items():
print(f"{m:<16} {f'{n}/{len(QUERIES)}':>8}")
print("\nWHY EACH ONE FAILS WHERE IT DOES")
b = bm25("can I give something back")
print(f" BM25 on a paraphrase: {[(x, round(s,2)) for x, s in b[:3]]}")
print(" no content word is shared with d1. Lexical has literally nothing.")
d = dense(QUERIES['ERR-4012'][1])
print(f" dense on an identifier: {[(x, round(s,3)) for x, s in d[:3]]}")
print(" ERR-4012 and ERR-4021 are near-identical vectors. One character")
print(" flips the meaning and the embedding cannot see it.")
print("\nTHE RRF TRAP")
b2 = bm25("how long is shipping")
print(f" BM25 for 'how long is shipping': {[(x, round(s,2)) for x, s in b2]}")
print(" every score is 0 -- no term matched anything. But the ranking is still")
print(" a full ordered list, so naive RRF counts its arbitrary first entry as a")
print(" rank-1 vote and can override the retriever that DID have signal.")
print(" -> drop zero-score results before fusing.")
assert hits["RRF drop-zero"] == len(QUERIES)
assert hits["RRF naive"] < len(QUERIES) # the trap is real
assert hits["BM25"] < len(QUERIES)
assert hits["dense"] < len(QUERIES)
# Dense cannot separate two near-identical identifiers; BM25 can.
assert dense(QUERIES["ERR-4012"][1])[0][0] != "d3"
assert bm25("ERR-4012")[0][0] == "d3"
# Lexical is blind to a pure paraphrase and to a synonym ('shipping' vs 'Delivery').
assert bm25("can I give something back")[0][1] == 0.0
assert bm25("how long is shipping")[0][1] == 0.0
print("\nall assertions passed")
# Output:
# query want BM25 dense RRF RRF+drop0
# 'can I give something back' d1 d1 d1 d1 d1
# 'ERR-4012' d3 d3 d2 x d3 d3
# 'how long is shipping' d4 d1 x d4 d1 x d4
#
# method correct
# BM25 2/3
# dense 2/3
# RRF naive 2/3
# RRF drop-zero 3/3
#
# WHY EACH ONE FAILS WHERE IT DOES
# BM25 on a paraphrase: [('d1', 0.0), ('d2', 0.0), ('d3', 0.0)]
# no content word is shared with d1. Lexical has literally nothing.
# dense on an identifier: [('d2', 1.0), ('d3', 1.0), ('d4', 0.152)]
# ERR-4012 and ERR-4021 are near-identical vectors. One character
# flips the meaning and the embedding cannot see it.
#
# THE RRF TRAP
# BM25 for 'how long is shipping': [('d1', 0.0), ('d2', 0.0), ('d3', 0.0), ('d4', 0.0)]
# every score is 0 -- no term matched anything. But the ranking is still
# a full ordered list, so naive RRF counts its arbitrary first entry as a
# rank-1 vote and can override the retriever that DID have signal.
# -> drop zero-score results before fusing.
#
# all assertions passed
The dense side uses three interpretable topic weights instead of a real encoder, so the ranking can be checked by eye. BM25 is the genuine formula — k1, b, IDF and length normalisation all as specified.
6. Real-world example
A team ran dense-only retrieval over a support knowledge base. Recall looked good in testing, because their test questions were all natural language written by the team.
In production, a complaint pattern emerged around error codes. A customer quoting ERR-4021 would get an answer about a different error — plausible, wrong remedy, occasionally advising a destructive action.
The cause is §3.2. To an embedding model, ERR-4021 and ERR-4012 are almost the same string in almost the same context, so their vectors are nearly identical. Semantic similarity is the wrong measure for an identifier, where a single character changes everything.
Two things had hidden it. No test question contained an identifier, so the failure class was never exercised. And the similarity scores were high — around 0.95, because the passages genuinely were about the same topic — so the relevance threshold they had added offered no protection at all.
They added BM25 alongside and fused the results. Then they hit the §3.4 trap: on paraphrased queries BM25 returned all-zero scores, and naive fusion let its arbitrary top result outrank the dense retriever that had the answer. Hybrid search briefly made things worse on the queries dense had been handling fine.
Filtering zero-score results before fusing fixed it, and identifier queries started resolving correctly.
Two lessons. A test set only exercises the failure modes you thought of — theirs contained only the kind of query the technique was already good at. And combining retrievers is a technique with its own failure modes, not a free upgrade: measure the hybrid the same way you measured the parts.
7. Interview questions companies actually ask
Q1. When does keyword search beat vector search? Whenever the exact string matters: identifiers, error codes, SKUs, product names, rare technical terms. BM25 also needs no model, no GPU, and no embedding step, so it is cheap and fast. Its blind spot is vocabulary mismatch — in the worked example it scored exactly 0.0 on a paraphrase and on a synonym, meaning no signal at all rather than a weak one.
Q2. Explain BM25 in one breath. Sum over query terms of: how often the term appears in the document, with diminishing returns; weighted by how rare the term is across the corpus; normalised by document length so long documents do not win by containing more of everything. k1 controls the saturation, b controls the length normalisation.
Q3. Why can't you just add BM25 and cosine scores together? They are not on the same scale — BM25 is unbounded and corpus-dependent, cosine sits in a fixed range. Any weighting is arbitrary and shifts when the corpus changes. Reciprocal Rank Fusion avoids the problem by discarding the scores and using only rank position, so nothing needs reconciling.
Q4. What's the trap in hybrid retrieval? A retriever with no signal still returns a full ordered list. In our measurement BM25 scored 0.00 on every document for one query, and naive RRF still counted its arbitrary first result as a rank-1 vote — which outranked the dense retriever that was right. Naive fusion scored 2/3, the same as either retriever alone. Dropping zero-score entries before fusing took it to 3/3.
Q5. Your retriever returns high scores for the wrong passage. What's happening? Probably confusable content rather than irrelevant content — passages genuinely about the same topic that differ in a detail the embedding compresses away, like a code or a number. It is the dangerous case because a score threshold gives no protection when the score is 0.95. The fix is lexical matching alongside, not a higher threshold.
Q6. When do you add a reranker? When you are retrieving generously for recall and paying for it in precision. The standard shape is retrieve wide with hybrid, then rerank the shortlist with a slower cross-encoder and keep the best few — recall from the cheap pass, precision from the expensive one, and you only run the expensive model on a shortlist.
Q7. What can no retriever fix? Currency and authority — similarity has no notion of which document is current, so a superseded page scores as well as its replacement. That is metadata: dates, version filters, deletion. Also aggregation questions, which are database queries wearing a sentence. And anything destroyed at chunk time, since retrieval can only return a chunk that exists.
8. When to use / tradeoffs
Use BM25 alone when:
- Queries are keyword-like — codes, names, exact phrases
- You have no embedding infrastructure and want something strong today
- Latency and cost budgets are very tight
Use dense alone when:
- Users phrase things very differently from your documents
- Paraphrase and synonym matching is the main requirement
- Identifiers essentially never appear in queries
Use hybrid when:
- Query traffic is mixed — most real products
- You have measured that each retriever misses different queries
- You can also handle the zero-signal case correctly
| Situation | Why it breaks | Do this instead |
|---|---|---|
| Dense only, identifier queries | Near-identical vectors, high scores | Add BM25; fuse |
| BM25 only, paraphrased queries | Score exactly 0.0 — no signal | Add dense; fuse |
| Adding BM25 and cosine scores | Incompatible scales | RRF, which uses rank only |
| Naive RRF | No-signal rankings still vote | Drop zero-score results first |
| Threshold to catch wrong-but-similar | Score is 0.95; threshold is blind | Lexical matching, or a reranker |
| Test set of natural-language questions only | Never exercises identifier failures | Include the query types you fear |
| Superseded documents retrieved | Similarity has no notion of currency | Date/version metadata filters |
| "How many X?" | Needs aggregation, not retrieval | Database query |
Honest limits. The dense side of §5 uses three hand-set topic weights rather than a real encoder, so the "embedding" is exactly as good as I designed it to be — a real model's failures are messier and less predictable than a clean 1.000/1.000 tie. Four documents and three queries is a demonstration, not a benchmark: the 2/3 and 3/3 counts show the structure of the argument and carry no information about how hybrid performs on your corpus. RRF's k=60 is the conventional default and the results are not very sensitive to it here, which they can be at scale. The zero-score fix also assumes a retriever that reports genuinely zero scores; a dense retriever never does, so "no signal" for it means "everything below threshold" — a judgement call rather than a clean test. And the whole article assumes a fixed query, where query rewriting before retrieval often matters more than which retriever you chose.
9. Summary + related articles
- Two families, two blind spots. Lexical matches words and scores exactly 0.0 on paraphrase and synonyms; dense matches meaning and cannot separate
ERR-4012fromERR-4021. - Dense's failure comes with a high score (1.000 in the example), so a relevance threshold offers no protection.
- Each scored 2/3 alone, missing different queries — the precondition for fusion to help.
- Fuse by rank, not score. BM25 is unbounded and corpus-dependent; cosine is 0–1. RRF uses position only.
- The trap: naive RRF also scored 2/3. A retriever with no signal still returns a full ranked list, and its arbitrary top entry cast a rank-1 vote that beat the retriever that was right.
- Dropping zero-score results before fusing took it to 3/3. A no-signal retriever must abstain, not rank.
- Standard production shape: retrieve wide with hybrid, then rerank to a few.
- Query expansion helps vocabulary mismatch; MMR helps near-duplicate results.
- No retriever fixes currency or authority — that's metadata. Nor aggregation. Nor anything destroyed at chunk time.
- A test set only exercises the failures you thought of.
Related:
- Chunking Strategies — what these retrievers are searching over, and why it caps them
- Vector Search for Retrieval — the dense side in detail, including thresholds
- Anatomy of a RAG Pipeline — where retrieval sits, and the four failure modes
- Reranking: The Second Pass That Decides What the Model Sees — the precision stage after a wide hybrid retrieve
- Query Transformation: Fixing the Question Before You Retrieve — rewriting the question before you retrieve
- Multi-Index RAG: Merging Several Retrievers Into One Answer — hybrid and multiple indexes at scale
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — where hybrid is implemented in practice
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring retrieval separately from the answer
- Search Systems — BM25 and ranking as a systems problem
- Embeddings and Cosine Similarity — what the dense vectors are
Resources
- Robertson & Zaragoza (2009) — The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval 3(4) — the definitive treatment of §4.1, by its authors.
- Cormack, Clarke & Buettcher (2009) — Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR — the paper RRF and
k=60come from: https://dl.acm.org/doi/10.1145/1571941.1572114 - Karpukhin et al. (2020) — Dense Passage Retrieval for Open-Domain Question Answering, arXiv:2004.04906 — dense retrieval measured against BM25 baselines, with the cases each wins: https://arxiv.org/abs/2004.04906
- Thakur et al. (2021) — BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, arXiv:2104.08663 — evidence that BM25 remains a very strong baseline across domains: https://arxiv.org/abs/2104.08663
- Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 6 and 11 — vector space models and probabilistic retrieval, free online: https://nlp.stanford.edu/IR-book/
- Carbonell & Goldstein (1998) — The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries, SIGIR — the diversity technique in §3.5: https://dl.acm.org/doi/10.1145/290941.291025