TL;DR — Real systems retrieve from more than one place: a vector index, a keyword search, an external catalogue API, sometimes the live web. Each returns overlapping documents with incomparable scores, so two problems appear that a single-index system never has. Deduplication: the same work arrives under different titles from different sources, and exact-string matching catches almost none of it — measured below, exact matching merges 10% of duplicates while an identifier-first cascade with normalised fuzzy fallback merges 100% with zero wrong merges. Fusion: you cannot add a BM25 score to a cosine similarity to a citation count. Summing raw scores lets whichever source has the widest numeric range silently decide the ranking — nDCG@5 of 0.607 versus 0.816 after min-max normalisation. Blending metadata priors (popularity, recency) on top of relevance helps, but only in a narrow band: the best mix here scores 0.831 at 70/20/10, while pushing popularity to half the weight collapses it to 0.649, and ranking by popularity alone gives 0.289. Multi-index RAG is mostly plumbing, and the plumbing is where the quality goes.
1. Simple explanation
One index is easy. You ask, it returns ten things with ten scores, you sort by score, done.
Now add a second source — say a keyword index alongside your vector index — and three things break at once.
The same document comes back twice, once from each source, under slightly different titles, and if you do not notice, it occupies two of your five context slots with identical text. The scores are on different scales: one source returns cosine similarities between 0 and 1, another returns BM25 scores between 0 and 40, a third returns a raw citation count in the thousands. And some sources carry metadata the others do not — a publication date, a popularity count, a quality tier — which you would like to use in ranking but which exists for only part of the pool.
None of this is intellectually deep. All of it decides whether the system works.
Analogy — merging three shortlists into one hiring decision. Three interviewers each hand you a ranked list. One scores out of 10, one gives letter grades, one just says "yes / maybe / no." Two of them interviewed the same candidate without realising, and spelled their name differently. Before you can pick the top five you must figure out who is the same person and put everyone on one scale. Get either wrong and the shortlist is arbitrary — not because anyone judged badly, but because you merged badly.
2. Diagram
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ VECTOR INDEX │ │ KEYWORD │ │ CATALOGUE │ │ WEB SEARCH │
│ cosine 0..1 │ │ BM25 0..40 │ │ cites 0..9k │ │ rank only │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │
└────────┬────────┴────────┬────────┴────────┬────────┘
│ run in PARALLEL; slowest source
│ sets the latency, and one that is
▼ down must not fail the query
┌─────────────────────┐
│ ① MERGE + DEDUPE │ strong id -> normalised title -> fuzzy
│ │ measured: exact 10%, cascade 100%
└──────────┬──────────┘
▼
┌─────────────────────┐
│ ② NORMALISE SCORES │ min-max or z-score, PER SOURCE, per query
│ │ raw sum 0.607 -> min-max 0.816
└──────────┬──────────┘
▼
┌─────────────────────┐
│ ③ BLEND PRIORS │ 0.70*relevance + 0.20*popularity
│ │ + 0.10*recency -> 0.831
└──────────┬──────────┘ (0.30/0.50/0.20 -> 0.649: overweighted)
▼
top 5
WHY RAW SUMMING FAILS
─────────────────────
doc A cosine 0.91 BM25 2.0 raw sum ≈ 502.9 ← BM25 source decides
doc B cosine 0.42 BM25 38.0 raw sum ≈ 540.4 everything, because
its numbers are big
after per-source min-max, both sources get an equal vote.
3. How it works
3.1 Why more than one index
Different retrievers fail differently, and that is the entire argument. A dense vector index handles paraphrase and synonymy but misses exact identifiers — part numbers, error codes, surnames. A lexical index nails exact tokens and is helpless at paraphrase. An external catalogue has authoritative metadata your local index lacks. Live web search covers what you have not ingested.
Fusing helps only to the extent the sources make independent mistakes. Two vector indexes over the same corpus with the same embedding model will agree with each other and add nothing but cost. This is the same correlation trap as decomposing one query into overlapping sub-queries — see Query Transformation: Fixing the Question Before You Retrieve.
Run the sources in parallel. The slowest sets your latency, so give each a timeout and treat a missing source as degraded, not fatal.
3.2 Deduplication: identity is a cascade, not a key
The same work arrives as "Attention Is All You Need", "Attention is all you need.", "ATTENTION IS ALL YOU NEED", and "Attention Is All You Need (v5)". There is no single field that resolves this, so use a cascade, cheapest and most reliable first:
1. STRONG IDENTIFIER an id both sources agree on (DOI, ISBN, SKU, URL
after canonicalisation). Trust it absolutely.
Only some records carry one.
2. NORMALISED TITLE lowercase, strip accents/punctuation/version
suffixes, collapse whitespace. Exact match on THAT.
Catches most of the rest.
3. FUZZY MATCH similarity above a threshold on the normalised
title. Catches subtitle drift and truncation.
The only step that can merge WRONG things.
Two rules keep this safe. A strong identifier is decisive in both directions — if two records carry different strong ids they are different works, no matter how similar the titles, and fuzzy matching must not override that. This is what stops "Deep Residual Learning for Image Recognition" merging with "Deep Residual Learning for Speech Recognition". And keep the merged record's union of metadata: if one source supplied a date and another a popularity count, the cluster should carry both. Dedup is a merge, not a delete.
3.3 Normalising incomparable scores
Once deduplicated, every document has one score per source that found it, on that source's scale. Three options:
MIN-MAX x' = (x - min) / (max - min) -> [0,1] per source
simple; sensitive to a single outlier stretching the range
Z-SCORE x' = (x - mean) / stdev -> mean 0, sd 1
robust to range, assumes roughly symmetric scores
RANK-BASED discard scores, use positions (RRF) -> no calibration at all
immune to any scale problem; throws away score MAGNITUDE,
so a runaway-best document looks the same as a marginal winner
Normalise per source and per query, not globally. Score distributions shift query to query — an easy query has several documents near the top of the range, a hard one has none — and a global normalisation baked in at index time cannot see that.
Then there is the missing-source problem. If a document was found by two sources out of three, what is its third score? Treating "not returned" as zero punishes it for a source's top-k cutoff rather than for irrelevance. Options: impute the minimum observed for that source, average only over sources that returned it, or use rank-based fusion which sidesteps the question. All three are defensible; silently defaulting to zero is not, and it is the most common bug in this code.
3.4 Blending in metadata priors
Relevance is what the user asked for. Metadata is what you know independently: how popular a document is, how recent, how authoritative the venue. A weighted blend combines them:
final = w_rel * relevance + w_pop * popularity + w_rec * recency
(each term normalised to [0,1] first, or the weights are lies)
Priors help because relevance scoring is noisy, and an independent signal partially corrects that noise. Measured below, a 70/20/10 split lifts nDCG@5 from 0.816 to 0.831.
They hurt fast when overweighted, for a specific and important reason: popularity is a lagging indicator and correlates with age, not just quality. An old, mediocre document accumulates more citations, clicks, and inbound links than a new, excellent one. Weight popularity heavily and you systematically bury the best recent document under a popular older one — and because the result looks authoritative, nobody notices. Ranking by popularity alone scored 0.289 here against 0.816 for relevance alone.
Recency partly counteracts popularity's age bias, which is why the two are usually tuned together rather than separately.
3.5 Where this stops working
If your sources are highly correlated, fusion adds cost and no quality. If one source is much better than the others, blending it with weaker ones drags it down — the same sign-flip that makes a weak reranker harmful, covered in Reranking: The Second Pass That Decides What the Model Sees. Check that each source earns its place by measuring the fused result against each source alone; a source that does not improve the fusion should be removed, not down-weighted.
4. The math
4.1 Why raw summing hands the decision to one source
Suppose source s returns scores with mean m_s and standard deviation sd_s. Summing raw scores gives:
total(d) = sum_s score_s(d)
The constant offsets m_s shift every document equally and cannot change the ranking. What decides the ranking is the variation, and a source contributes variation in proportion to its sd_s. So the effective weight of each source is:
weight_s ∝ sd_s NOT 1/number_of_sources
With the three sources in §5 — scales of roughly 1, 40, and 0.01 — the middle source has about 40× the standard deviation of the first and 4000× the third. It decides the ranking essentially alone, and the other two are decoration. That is the measured drop from 0.816 to 0.607: not a subtle degradation, but two of three sources being silently discarded.
Normalising to a common scale is what makes "sum the sources" mean what you intended: an equal vote each.
4.2 The dedup confusion matrix
Deduplication is a classification problem over pairs of records, so it has two error modes that trade off:
truly same truly different
merged ✓ good ✗ OVER-MERGE (two works become one;
you lose a document entirely)
kept separate ✗ UNDER-MERGE ✓ good
(duplicate eats
a context slot)
They are not symmetric. An under-merge wastes a context slot — annoying, bounded. An over-merge destroys a document: it vanishes from the results and cannot be recovered downstream, and you will never see it in a log. Tune the fuzzy threshold conservatively, and let strong identifiers veto fuzzy matches.
Measured on 17 records describing 7 distinct works:
strategy clusters duplicates merged wrong merges
exact 16 10% 0
normalized 8 90% 0
id_then_fuzzy 7 100% 0
ideal 7 100% 0
Exact string matching catches one duplicate in ten. Normalisation alone gets to 90% — the remaining miss is a genuine subtitle difference that no amount of case-folding resolves. The full cascade reaches the ideal clustering with no over-merges, because the two look-alike distinct works carry different strong identifiers.
4.3 The prior-weight curve
w_rel w_pop w_rec nDCG@5
1.00 0.00 0.00 0.816 relevance only
0.85 0.10 0.05 0.828
0.70 0.20 0.10 0.831 <- best
0.50 0.35 0.15 0.796 already worse than no priors
0.30 0.50 0.20 0.649
0.00 1.00 0.00 0.289 popularity only
An interior optimum with a sharp right-hand fall. Going from the best mix to a popularity-dominant one costs 0.831 - 0.649 = 0.182, six times the 0.015 the priors gained in the first place. The asymmetry is the practical lesson: the downside of over-weighting a prior is much larger than its upside. When unsure, under-weight.
5. Real code
Standard library only, deterministic. Experiment A uses real paper titles as they actually appear across sources; B and C simulate three retrievers with independent errors and deliberately incomparable score scales.
import math
import random
import re
import unicodedata
from difflib import SequenceMatcher
SEED = 7
# ----------------------------------------------------------------- A. dedup
# The same underlying work, as each source actually stores it.
DUP_GROUPS = [
["Attention Is All You Need",
"Attention is all you need.",
"ATTENTION IS ALL YOU NEED",
"Attention Is All You Need (v5)"],
["Deep Residual Learning for Image Recognition",
"Deep residual learning for image recognition",
"Deep Residual Learning for Image Recognition"],
["BERT: Pre-training of Deep Bidirectional Transformers",
"BERT - Pre-training of Deep Bidirectional Transformers",
"Bert: pre-training of deep bidirectional transformers for language understanding"],
["A Survey of Large Language Models",
"A Survey of Large Language Models",
"A survey of large language models"],
["Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks",
"Retrieval Augmented Generation for Knowledge Intensive NLP Tasks"],
]
# Distinct works whose titles look similar -- dedup must NOT merge these.
NEAR_MISSES = [
"Deep Residual Learning for Speech Recognition",
"A Survey of Small Language Models",
]
def make_records():
"""(record_id, title, strong_id) -- strong_id is None when the source
did not supply one, which is the realistic case."""
rng = random.Random(SEED)
recs = []
for gi, group in enumerate(DUP_GROUPS):
for vi, title in enumerate(group):
# only ~half of records carry a usable strong identifier
sid = f"ID-{gi}" if rng.random() < 0.5 else None
recs.append((f"r{gi}-{vi}", title, sid))
for ni, title in enumerate(NEAR_MISSES):
recs.append((f"n{ni}", title, f"ID-N{ni}"))
return recs
def norm_title(t):
t = unicodedata.normalize("NFKD", t).lower()
t = re.sub(r"\(v\d+\)", " ", t)
t = re.sub(r"[^a-z0-9]+", " ", t)
return " ".join(t.split())
def dedup(records, strategy):
"""Return clusters: [strong_id_or_None, canonical_title, [record_ids]]."""
clusters = []
for rid, title, sid in records:
nt = norm_title(title)
placed = False
for c in clusters:
if strategy == "exact" and c[1] == title:
placed = True
elif strategy == "normalized" and c[1] == nt:
placed = True
elif strategy == "id_then_fuzzy":
if sid and c[0] and sid == c[0]:
placed = True
elif sid and c[0] and sid != c[0]:
# different strong ids => different works, never merge
continue
elif SequenceMatcher(None, c[1], nt).ratio() >= 0.90:
placed = True
if placed:
c[2].append(rid)
if strategy == "id_then_fuzzy" and sid and not c[0]:
c[0] = sid
break
if not placed:
key = sid if strategy == "id_then_fuzzy" else None
clusters.append([key, title if strategy == "exact" else nt, [rid]])
return clusters
def truth_group(rid):
return rid.split("-")[0]
def dedup_scores(clusters):
"""(fraction of true duplicates merged, number of wrong merges)"""
merged = wrong = 0
for _, _, ids in clusters:
if len({truth_group(i) for i in ids}) > 1:
wrong += 1
merged += len(ids) - 1
return merged / sum(len(g) - 1 for g in DUP_GROUPS), wrong
# ---------------------------------------------------------------- B. fusion
N_QUERIES = 300
POOL = 40
FINAL_N = 5
def make_world():
rng = random.Random(SEED + 1)
world = []
for _ in range(N_QUERIES):
docs = []
for _ in range(POOL):
r = rng.random()
grade = 3 if r < 0.03 else 2 if r < 0.10 else 1 if r < 0.22 else 0
# popularity correlates with relevance but is dominated by age:
# old mediocre documents accumulate more citations than new good ones
age = rng.uniform(0, 20)
pop = max(0.0, 0.6 * grade + 0.30 * age + rng.gauss(0, 1.5))
recency = max(0.0, 1 - age / 20)
# each source sees the document through its OWN error: the sources
# carry independent information, which is why fusing them helps
docs.append({"grade": grade, "pop": pop, "recency": recency,
"noise": [rng.gauss(0, 1.4) for _ in range(3)]})
world.append(docs)
return world
WORLD = make_world()
# Three sources with deliberately incomparable score scales.
SOURCES = [{"scale": 1.0, "offset": 0.0},
{"scale": 40.0, "offset": 500.0},
{"scale": 0.01, "offset": -2.0}]
def source_scores(docs, si):
s = SOURCES[si]
return [s["offset"] + s["scale"] * (d["grade"] + d["noise"][si]) for d in docs]
def minmax(xs):
lo, hi = min(xs), max(xs)
return [0.5] * len(xs) if hi == lo else [(x - lo) / (hi - lo) for x in xs]
def zscore(xs):
m = sum(xs) / len(xs)
sd = (sum((x - m) ** 2 for x in xs) / len(xs)) ** 0.5
return [0.0] * len(xs) if sd == 0 else [(x - m) / sd for x in xs]
def dcg(gs):
return sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(gs))
def ndcg(order, docs):
den = dcg(sorted((d["grade"] for d in docs), reverse=True)[:FINAL_N])
return dcg([docs[i]["grade"] for i in order[:FINAL_N]]) / den if den else 0.0
def fuse(method, w_rel=1.0, w_pop=0.0, w_rec=0.0):
tot = 0.0
for docs in WORLD:
per = [source_scores(docs, si) for si in range(len(SOURCES))]
if method == "raw":
comb = [sum(p[i] for p in per) for i in range(POOL)]
elif method == "minmax":
n = [minmax(p) for p in per]
comb = [sum(x[i] for x in n) for i in range(POOL)]
elif method == "zscore":
n = [zscore(p) for p in per]
comb = [sum(x[i] for x in n) for i in range(POOL)]
elif method == "rrf":
comb = [0.0] * POOL
for p in per:
for rank, i in enumerate(sorted(range(POOL), key=lambda j: -p[j]), 1):
comb[i] += 1.0 / (60 + rank)
elif method == "weighted":
n = [minmax(p) for p in per]
rel = minmax([sum(x[i] for x in n) for i in range(POOL)])
pop = minmax([d["pop"] for d in docs])
rec = [d["recency"] for d in docs]
comb = [w_rel * rel[i] + w_pop * pop[i] + w_rec * rec[i]
for i in range(POOL)]
order = sorted(range(POOL), key=lambda i: -comb[i])
tot += ndcg(order, docs)
return tot / N_QUERIES
print("A. DEDUPLICATION -- one work, many spellings")
print(f"{'strategy':>16} {'clusters':>9} {'dups merged':>12} {'wrong merges':>13}")
print("-" * 54)
recs = make_records()
ideal = len(DUP_GROUPS) + len(NEAR_MISSES)
for st in ("exact", "normalized", "id_then_fuzzy"):
rec, wrong = dedup_scores(dedup(recs, st))
print(f"{st:>16} {len(dedup(recs, st)):9} {rec:11.0%} {wrong:13}")
print(f"{'(ideal)':>16} {ideal:9} {'100%':>11} {0:13}")
print(f"\n{len(recs)} records describing {ideal} distinct works")
print("\n\nB. FUSING INCOMPARABLE SCORES")
print(" three sources, score scales ~1, ~40 (+500 offset), ~0.01")
print(f"{'method':>10} {'nDCG@5':>8}")
print("-" * 20)
res = {}
for m in ("raw", "minmax", "zscore", "rrf"):
res[m] = fuse(m)
print(f"{m:>10} {res[m]:8.3f}")
print("\n\nC. BLENDING IN METADATA PRIORS (popularity + recency)")
print(f"{'w_rel':>6} {'w_pop':>6} {'w_rec':>6} {'nDCG@5':>8}")
print("-" * 30)
best = None
for wr, wp, wc in [(1.00, 0.00, 0.00), (0.85, 0.10, 0.05), (0.70, 0.20, 0.10),
(0.50, 0.35, 0.15), (0.30, 0.50, 0.20), (0.00, 1.00, 0.00)]:
v = fuse("weighted", wr, wp, wc)
if best is None or v > best[1]:
best = ((wr, wp, wc), v)
print(f"{wr:6.2f} {wp:6.2f} {wc:6.2f} {v:8.3f}")
pure = fuse("weighted", 1.0, 0.0, 0.0)
print(f"\npure relevance {pure:.3f}; best blend {best[1]:.3f} at {best[0]}")
print(f"popularity-only {fuse('weighted', 0.0, 1.0, 0.0):.3f}")
# claims made in the prose
assert res["raw"] < res["minmax"], "unnormalised summing must lose"
assert res["raw"] < res["rrf"] < res["minmax"]
assert abs(res["zscore"] - res["minmax"]) < 0.02
assert dedup_scores(dedup(recs, "exact"))[0] < 0.3
assert dedup_scores(dedup(recs, "id_then_fuzzy"))[0] == 1.0
assert dedup_scores(dedup(recs, "id_then_fuzzy"))[1] == 0
assert best[0] != (1.00, 0.00, 0.00), "a modest prior blend should help"
assert fuse("weighted", 0.30, 0.50, 0.20) < pure, "over-weighting must hurt"
assert fuse("weighted", 0.0, 1.0, 0.0) < 0.5, "popularity alone is a poor ranker"
print("\nasserts passed")
# Output:
# A. DEDUPLICATION -- one work, many spellings
# strategy clusters dups merged wrong merges
# ------------------------------------------------------
# exact 16 10% 0
# normalized 8 90% 0
# id_then_fuzzy 7 100% 0
# (ideal) 7 100% 0
#
# 17 records describing 7 distinct works
#
#
# B. FUSING INCOMPARABLE SCORES
# three sources, score scales ~1, ~40 (+500 offset), ~0.01
# method nDCG@5
# --------------------
# raw 0.607
# minmax 0.816
# zscore 0.820
# rrf 0.797
#
#
# C. BLENDING IN METADATA PRIORS (popularity + recency)
# w_rel w_pop w_rec nDCG@5
# ------------------------------
# 1.00 0.00 0.00 0.816
# 0.85 0.10 0.05 0.828
# 0.70 0.20 0.10 0.831
# 0.50 0.35 0.15 0.796
# 0.30 0.50 0.20 0.649
# 0.00 1.00 0.00 0.289
#
# pure relevance 0.816; best blend 0.831 at (0.7, 0.2, 0.1)
# popularity-only 0.289
#
# asserts passed
Note that RRF (0.797) slightly trails min-max (0.816) here. That is expected and worth understanding: rank-based fusion throws away score magnitude, so it cannot tell a document that won its source by a mile from one that won by a hair. When your scores are meaningfully calibrated, normalising beats ranking. When they are not — different model families, a source that returns only positions — RRF is the safer choice precisely because it ignores the numbers.
6. Real-world example
A product search merged an in-house catalogue index with a supplier feed. Both were good. Fused, results got noticeably worse, and it took three weeks to find out why.
The supplier feed returned a "match confidence" between 0 and 1000. The catalogue returned cosine similarity between 0 and 1. The fusion code summed them. Every ranking decision was therefore made by the supplier feed alone, with the catalogue contributing at most 1 point of tie-breaking — the failure in §4.1, in production, invisible because results still looked plausible and no error was ever logged.
Dedup made it worse rather than better. Matching on normalised product title merged two genuinely different items — a cable and its 3-metre variant differed only by a suffix the normaliser stripped. The longer variant disappeared from search entirely. Nobody noticed for a month, because an over-merged document produces no error, no empty result, and no log line. It surfaced through a sales report showing one SKU had stopped selling.
Both fixes were small: normalise per source per query before summing, and make the supplier's SKU a decisive identifier that fuzzy title matching cannot override. The lesson the team took away was about observability, not ranking — they added a metric for cluster-size distribution, so an unexpected jump in merges now pages someone instead of quietly deleting inventory.
7. Interview questions companies actually ask
Q1 [easy] "Why can't you just add the scores from two retrievers together?"
A Because the ranking is decided by each source's VARIATION, not its mean --
effective weight is proportional to standard deviation. With scales of ~1 and
~40, the second source has 40x the spread and decides the order essentially
alone. Measured: raw summing gave nDCG@5 0.607 vs 0.816 after per-source
min-max. You silently discarded two of three sources.
Q2 [easy] "How do you deduplicate documents from different sources?"
A A cascade, cheapest and most reliable first: strong identifier (DOI/ISBN/SKU/
canonical URL), then exact match on a normalised title (lowercased, punctuation
and version suffixes stripped), then fuzzy similarity above a threshold. Exact
string matching alone caught 10% of duplicates in the measurement above; the
full cascade caught 100%.
Q3 [medium] "Which dedup error is worse -- over-merging or under-merging?"
A Over-merging, by a lot. An under-merge wastes a context slot: annoying, bounded,
visible. An over-merge DELETES a document from the result set with no error, no
empty result, and no log line -- you find out from a business metric weeks later.
So: conservative fuzzy threshold, and let differing strong ids veto a fuzzy match.
Q4 [medium] "Min-max, z-score, or RRF?"
A Min-max is simple but one outlier stretches the range. Z-score is robust to range
and assumes roughly symmetric scores. RRF ignores scores entirely and uses ranks,
so it's immune to calibration problems but blind to MAGNITUDE -- it can't tell a
document that won by a mile from one that won by a hair. Measured here RRF
(0.797) trailed min-max (0.816) because the scores were meaningful. Use RRF when
they aren't.
Q5 [medium] "A document was returned by two of your three sources. What's its third
score?"
A Not zero -- that punishes it for a top-k cutoff rather than for irrelevance, and
it's the most common bug in fusion code. Impute the minimum observed for that
source, average only over sources that returned it, or switch to rank-based
fusion which sidesteps the question entirely. Any of those; just make it explicit.
Q6 [hard] "You blend popularity into ranking. What goes wrong?"
A Popularity is a lagging indicator correlated with AGE, not just quality. Old
mediocre documents out-cite new excellent ones, so weighting popularity heavily
systematically buries the best recent result -- and the output looks authoritative,
so nobody reports it. Measured: 70/20/10 relevance/popularity/recency gave the best
nDCG@5 at 0.831, but 30/50/20 collapsed to 0.649 and popularity alone gave 0.289.
The downside of over-weighting is ~6x the upside. Pair recency with popularity to
counteract the age bias.
Q7 [hard] "When does adding a source make the system worse?"
A When it's correlated with an existing source (cost, no new information) or when
it's substantially weaker than the others (it drags the fusion down -- same sign
flip as a weak reranker). Test by measuring the fused result against each source
alone. A source that doesn't improve the fusion should be removed, not
down-weighted.
Q8 [hard] "One of your four sources is timing out. What should happen?"
A Degrade, don't fail. Sources run in parallel with per-source timeouts; a missing
source returns nothing and the query proceeds on the rest, with the event logged
and surfaced. Two follow-ons: per-query normalisation must handle a variable
source count, and you should track fused quality with each source absent so you
know what a given outage actually costs.
8. When to use / tradeoffs
REACH FOR MULTI-INDEX RAG WHEN:
+ your sources fail DIFFERENTLY (dense misses exact ids, lexical misses
paraphrase, external catalogue has metadata you lack)
+ coverage matters more than latency
+ some documents exist only in one source
DON'T WHEN:
- the sources are correlated (two dense indexes, same embedding model)
- one source clearly dominates -- fusing drags it down
- your latency budget can't absorb the slowest source
| Situation | Why it breaks | Use instead |
|---|---|---|
| Raw score summing | Widest-variance source decides alone — 0.607 vs 0.816 | Per-source, per-query normalisation |
| Missing score treated as 0 | Punishes a top-k cutoff as if it were irrelevance | Impute source minimum, or use RRF |
| Fuzzy dedup with no id veto | Over-merges look-alikes; a document silently vanishes | Strong id decides both ways |
| Global (index-time) normalisation | Score distributions shift per query | Normalise per query |
| Heavy popularity weight | Age bias buries the best recent document | ≤0.2 weight, paired with recency |
| Uncalibrated cross-model scores | Normalisation assumes comparable distributions | RRF — ranks need no calibration |
| One source down | Whole query fails | Per-source timeout, degrade and log |
Honest limits. Experiments B and C are simulations: three sources whose errors are independent Gaussians, which is the friendliest possible case for fusion. Real sources share biases — they often index the same underlying documents and inherit the same popularity signals — so real fusion gains are smaller than shown, and the correlation caveat in §3.1 matters more than the numbers here suggest. The optimal prior weights (70/20/10) are an artefact of how strongly I coupled popularity to age (pop = 0.6*grade + 0.30*age + noise); a corpus where popularity tracks quality more tightly would support a heavier popularity weight, and one where it tracks age more tightly would support less. Do not copy those weights — copy the method of sweeping them and the observation that the curve falls off much faster than it rises. Experiment A uses real titles but only 17 records; the 10%/90%/100% figures show the ordering of the strategies robustly and their exact values not at all. Finally, nothing here measures latency, which in practice is what usually kills a four-source design.
9. Summary + related articles
- Multi-index RAG pays only when sources fail differently. Correlated sources add cost, not quality.
- Dedup is a cascade: strong identifier → normalised title → conservative fuzzy. Exact string matching caught 10% of duplicates; the cascade caught 100% with no wrong merges.
- Over-merging is the dangerous error — it deletes a document silently, with no error and no log line. Let differing strong ids veto fuzzy matches.
- Never sum raw scores. Effective source weight is proportional to standard deviation, so the widest-range source decides alone: 0.607 raw vs 0.816 normalised.
- Normalise per source, per query. A missing score is not zero.
- Metadata priors have an interior optimum: 0.831 at 70/20/10, 0.649 at 30/50/20, 0.289 at popularity-only. The fall-off is ~6× the gain — when unsure, under-weight.
- Boundary: all of this assumes the sources are worth merging. Measure the fused result against each source alone before believing the fusion helps.
Related:
- Reranking: The Second Pass That Decides What the Model Sees — the step after fusion, and the same sign-flip risk: a component weaker than what it corrects makes things worse
- Query Transformation: Fixing the Question Before You Retrieve — §4.2 measures the same correlation trap for sub-queries that §3.1 describes for sources; §4.3 derives RRF
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — what a "document" is before any of this merging starts
- Weighted Averages & Aggregation — the general form of the blend in §3.4, and why terms must be normalised before weights mean anything
- Search Systems — multi-source retrieval as it appears in general web search architecture
Resources
- Cormack, Clarke & Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods", SIGIR 2009 — RRF, and the argument for rank-based fusion over score normalisation.
- Fox & Shaw, "Combination of Multiple Searches", TREC-2 proceedings, 1994 — the original CombSUM / CombMNZ score-fusion family that min-max summing descends from.
- Christen, Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, Springer 2012 — chapters 5–6 cover blocking and similarity thresholds; the standard reference for §3.2.
- Fellegi & Sunter, "A Theory for Record Linkage", Journal of the American Statistical Association 64(328), 1969 — the probabilistic foundation of the merge/no-merge decision.
- Järvelin & Kekäläinen, "Cumulated Gain-Based Evaluation of IR Techniques", ACM TOIS 20(4), 2002 — the nDCG definition used throughout.