TL;DR — Users write questions in their vocabulary; documents are written in the author's. Retrieval that matches the raw question against the corpus fails on that gap alone — in the harness below, plain BM25 recovers 43.3% of relevant documents and returns nothing useful at all for two of five questions. Query transformation rewrites the question before it reaches the index: expansion adds the corpus's vocabulary, decomposition splits a multi-part question into sub-queries that are fused afterwards, HyDE embeds a hypothetical answer instead of the question, and step-back prompting asks a more general question first. Flat expansion lifts recall@5 to 83.3% for one extra LLM call. The result that matters: decomposing into typed slots reaches exactly the same 83.3% while issuing 3.2× more searches — because the sub-queries retrieve 39.8% overlapping documents, so the union is far smaller than an independence assumption predicts. Transformation pays for itself; decomposition only pays when the branches are genuinely disjoint, or when you need the slots for something other than recall.
1. Simple explanation
Someone types "my outdoor sensor keeps dropping off wifi in winter after the last update." The document that answers it says "outdoor sensor disassociates from the access point when ambient temperature falls below freezing."
Those two sentences share almost no words. "Dropping off" versus "disassociates". "Winter" versus "below freezing". "The last update" versus "firmware 3.2.1". A keyword index sees two unrelated strings. Even an embedding model, which handles synonyms far better, is being asked to bridge a gap it never had to bridge at training time.
This is the vocabulary problem, and it is old — Furnas and colleagues measured it in 1987 and found that two people spontaneously choose the same term for the same thing less than 20% of the time. Retrieval quality is capped by it before your reranker, your chunking strategy, or your generation prompt ever gets a turn.
Query transformation is the step that closes the gap. You spend one cheap LLM call turning the user's question into something that looks like the documents you are searching, and then you retrieve.
Analogy — asking a librarian versus asking the catalogue. Walk up to a librarian and say "I need the book about the boat and the big fish." They will ask what you actually mean, translate it into the catalogue's language — Melville, Moby-Dick, 813.3 — and walk you to the shelf. Type the same sentence into the catalogue search box and you get nothing. The catalogue is not stupid; it simply indexes the author's words and you gave it yours. The librarian is the transformation step. Every technique in this article is a different way of automating that translation, and each one buys a different kind of accuracy for a different price.
2. Diagram
WITHOUT TRANSFORMATION WITH TRANSFORMATION
───────────────────────── ────────────────────────────────────
"sensor drops off wifi "sensor drops off wifi in winter"
in winter" │
│ ▼
│ ┌──────────────────────┐
│ │ TRANSFORM (1 call) │
│ ├──────────────────────┤
│ │ expand → add corpus│
│ │ vocabulary│
│ │ decompose→ sub-query │
│ │ per facet │
│ │ HyDE → draft a │
│ │ fake answer
│ │ step-back→ ask the │
│ │ general Q │
│ └──────────┬───────────┘
│ │
│ "disassociation dropout link loss
│ freezing sub-zero firmware update"
▼ ▼
┌───────────┐ ┌───────────┐
│ INDEX │ │ INDEX │
└─────┬─────┘ └─────┬─────┘
│ │
▼ ▼
d08 d01 d21 d14 d02 d01 d02 d03 d15 d07
▲ ▲ ▲
└─ 1 of 4 relevant, and └──┴─ 2 relevant, both at the top
the top hit is wrong MRR 0.50 → 1.00
MEASURED OVER 5 QUERIES (harness in §5, 24-document corpus):
strategy recall@5 MRR searches/query LLM calls
────────────────────────────────────────────────────────
raw 0.433 0.500 1.0 0
expand 0.833 1.000 1.0 1
slots 0.833 1.000 3.2 1
▲ ▲
└─ identical └─ 3.2x the work, same recall
3. How it works
3.1 The gap you are closing
Write the retrieval score as score(q, d) — high when query q and document d match. The trouble is that the user's q_user and the ideal query q* (the one phrased in the corpus's own language) are different strings. What you want is score(q*, d); what you compute is score(q_user, d).
Transformation is a function T that moves the query toward q*:
retrieved = search( T(q_user) ) instead of search( q_user )
Everything below is a different T. They are not exclusive — production systems stack two or three.
3.2 Expansion: add the corpus's words
The cheapest transform. Take the question and append terms the corpus is likely to use: synonyms, domain jargon, the formal name for the informal thing.
in : "why does my battery die so fast when it is cold outside"
out: "battery lithium alkaline capacity voltage cold freezing sub-zero outdoor"
Classical IR did this with a thesaurus or with pseudo-relevance feedback — retrieve once, harvest frequent terms from the top few documents, search again with those added. An LLM does it in one call and handles jargon no thesaurus contains. The risk is query drift: added terms that pull toward a neighbouring topic. Keep the original terms so the anchor holds.
3.3 Decomposition: one question, several searches
Some questions are really several questions welded together. "Does the new firmware fix the cold-weather dropout on battery sensors?" has three facets — firmware version, temperature, power source — and a document covering all three may not exist. Documents covering one each certainly do.
Decomposition issues one search per facet and fuses the rankings. Reciprocal rank fusion is the standard fuser because it needs no score calibration — it only reads positions:
RRF(d) = sum over branches b of 1 / (k + rank_b(d)) , k = 60 by convention
A document ranked 1st by one branch and 9th by another beats a document ranked 3rd by one branch alone. Documents that several facets agree on float up. That is the whole idea.
3.4 Typed slots: decomposition with a schema
Free-form decomposition gives you a bag of sub-queries. Slot-based structuring gives you a bag of labelled sub-queries, by extracting the question into a fixed schema:
"my outdoor sensor keeps dropping off wifi in winter after the last update"
device -> outdoor sensor, battery powered node
symptom -> disassociation, link loss, dropout
condition -> sub-zero, freezing, low temperature
trigger -> firmware update, over-the-air, regression
The template is domain-specific and that is the point. Evidence-based medicine has used PICO — Population, Intervention, Comparison, Outcome — since the mid-1990s to turn a clinical question into a searchable one; e-commerce uses product/attribute/constraint; incident response uses service/symptom/blast-radius. If your users ask structurally similar questions, a schema exists, and writing it down is usually a one-afternoon job.
Two properties make slots worth more than the raw recall numbers suggest:
- A missing slot is a detectable gap. If
conditioncomes back empty, the system knows the user has not said when the problem happens, and can ask instead of guessing. - Slots route. The
deviceslot picks the index; thesymptomslot picks which specialist retriever or tool to invoke. A flat expanded string cannot do that — it has thrown the structure away. This is the strongest argument for slots, and it is not a retrieval-quality argument.
3.5 HyDE: retrieve with a fake answer
Hypothetical Document Embeddings inverts the problem. Instead of making the question look like a document, generate a document: ask the LLM to answer the question, wrong facts and all, then embed that answer and search with its vector.
It works because a fabricated answer is written in answer-shaped language — the same register, length, and jargon as the corpus — so it sits much closer to real answers in embedding space than the question ever did. The factual errors mostly do not matter: the draft is a search probe, never shown to the user.
Where it bites: if the hallucinated answer invents a specific, wrong entity — a product name, a drug, an API that does not exist — that entity dominates the embedding and drags retrieval into the wrong neighbourhood. HyDE is strongest on broad conceptual questions and weakest on narrow entity lookups, which is the opposite of most people's intuition.
3.6 Step-back prompting: ask the general question first
For questions needing a principle before a particular, generate a more abstract question, retrieve that, and use both result sets. "Why did the sensor at 22 Elm Street stop reporting on 3 January?" steps back to "what causes battery sensors to stop reporting?" The specific query finds the incident record; the general one finds the mechanism that explains it.
3.7 Where this stops working
Transformation assumes the question is underspecified, not wrong. If the answer is genuinely absent from the corpus, every technique here confidently retrieves the nearest wrong thing — expansion widens the net over an empty sea. Detecting an unanswerable question needs a grounding check after retrieval; see Hallucination Detection & Grounding.
Transformation also costs a call on the latency-critical path, before retrieval can start. Where authors already write the way users speak — support tickets, chat logs, forum posts — the gap is small and the call is waste.
4. The math
4.1 What decomposition should buy
Suppose a relevant document g is retrieved by branch i with probability p_i. If the branches were independent, the probability that at least one finds it is:
P(union) = 1 - prod_i (1 - p_i)
With four branches at p_i = 0.5 each:
P(union) = 1 - (0.5)^4 = 1 - 0.0625 = 0.9375
A jump from 50% to 93.75% for four searches. That arithmetic is why decomposition looks so attractive on a whiteboard.
4.2 Why it does not deliver that
The branches are not independent. They are sub-queries of the same question, run against the same index, so they retrieve heavily overlapping sets. Measured on the harness in §5, across 5 queries and 16 branches:
results drawn across all branches : 93
distinct documents recovered : 56
overlap : 1 - 56/93 = 39.8%
Nearly two in five retrieved slots are a document another branch already found. The effective number of independent branches is far below the nominal count, so the union barely exceeds what one well-chosen query returns — which is exactly the measured tie at recall@5 = 0.833.
The correct mental model: decomposition buys coverage only to the extent the facets are disjoint. Facets of one sentence rarely are.
4.3 Worked RRF example
Take query q1 and its four slot branches (real rankings from the run in §5):
device d02 d21 d07 d01 d03 d16 d06 d15 d18
symptom d02 d15 d07 d24 d21 d04 d11 d01
condition d01 d11 d03 d15 d18 d24
trigger d08 d02 d18 d14 d04 d10 d05 d17 d01 d09
Score three documents with k = 60:
d01: 1/(60+4) + 1/(60+8) + 1/(60+1) + 1/(60+9) = 0.06122 <- 4 branches
d02: 1/(60+1) + 1/(60+1) + 1/(60+2) = 0.04892 <- 3 branches
d18: 1/(60+9) + 1/(60+5) + 1/(60+3) = 0.04575 <- 3 branches
d01 wins despite never ranking 1st in any branch, because all four facets agree it is relevant, while d02 tops two branches but is invisible to condition. That is RRF's central behaviour: broad agreement beats a single strong opinion.
Note the constant k = 60 flattens rank differences hard — 1st place contributes 1/61 = 0.0164 and 9th contributes 1/69 = 0.0145, only 12% less. RRF is deliberately nearly rank-blind; it counts votes far more than positions. Lower k if you trust your branches' internal ordering.
4.4 The cost ledger
strategy LLM calls searches recall@5
raw 0 1.0 0.433
expand 1 1.0 0.833
slots 1 3.2 0.833
Per-query latency for slots is not 3.2× — the branches run in parallel — but the cost is, and the tail latency is set by the slowest branch. Pay it for routing, not for recall.
5. Real code
Standard library only, no network, no model, fully deterministic. It builds a BM25 index over 24 support documents whose vocabulary deliberately differs from the questions, then compares the three strategies.
import math
import re
from collections import Counter
# Support notes for a smart-home product line. The vocabulary is deliberately
# the WRITER's ("disassociates", "sub-zero"), not the user's ("drops off",
# "winter") -- that gap is the thing being measured.
CORPUS = {
"d01": "Outdoor sensor disassociates from the access point when ambient temperature falls below freezing. Lithium cells lose terminal voltage in sub-zero conditions and the radio browns out during transmit.",
"d02": "Firmware 3.2.1 release notes. Fixes a regression in the low-power radio duty cycle introduced in 3.2.0 that caused intermittent link loss on battery powered outdoor units.",
"d03": "Battery replacement procedure for outdoor sensors. Use lithium iron disulfide cells rather than alkaline for installations exposed to cold. Alkaline capacity collapses near freezing.",
"d04": "Access point channel planning. Overlapping 2.4 GHz channels cause retransmission storms and apparent device dropout in dense apartment buildings.",
"d05": "Indoor thermostat pairing walkthrough. Hold the pair button for five seconds until the ring pulses amber, then select the unit in the mobile application.",
"d06": "Mesh repeater placement guidance. Each hop adds latency; place repeaters within line of sight of the hub for reliable outdoor coverage.",
"d07": "Understanding link quality indicators. RSSI below negative eighty five decibel milliwatts predicts frequent disassociation events on battery powered nodes.",
"d08": "Rolling back firmware. Downgrade from 3.2.0 to 3.1.4 using the recovery partition if a unit becomes unstable after an over the air update.",
"d09": "Thermostat schedule editor. Configure setback periods and vacation holds from the schedule tab.",
"d10": "Water leak detector installation. Mount the probe flat against the floor in the lowest point of the utility room.",
"d11": "Cold weather deployment checklist. Below freezing, prefer lithium chemistry, shorten reporting interval, and expect reduced radio range from condensation on the antenna housing.",
"d12": "Hub factory reset. Press and hold recessed reset for fifteen seconds. All paired devices must be re-provisioned afterwards.",
"d13": "Mobile application push notification settings. Configure per device alert thresholds and quiet hours.",
"d14": "Over the air update mechanism. Units download the payload in chunks during their scheduled wake window and apply it on the next boot.",
"d15": "Diagnosing intermittent dropout. Collect the event log, correlate disassociation timestamps against ambient temperature and battery voltage telemetry.",
"d16": "Door and window contact sensor magnet alignment. Gap must not exceed twelve millimetres or the reed switch will not close.",
"d17": "Energy monitoring clamp calibration. Set the current transformer ratio to match the installed clamp before trusting reported wattage.",
"d18": "Known issue: units on firmware 3.2.0 in cold climates report spurious low battery warnings because the voltage curve is sampled during transmit.",
"d19": "Camera night vision troubleshooting. Infrared reflection from nearby surfaces washes out the image; reposition away from walls.",
"d20": "Provisioning devices onto a hidden SSID. Enter the network name manually during setup; broadcast suppression prevents automatic discovery.",
"d21": "Zigbee versus wifi tradeoffs for battery powered endpoints. Wifi association costs significantly more energy per wake cycle.",
"d22": "Warranty and return policy for hardware purchased through authorised resellers.",
"d23": "Smart plug scheduling and away mode behaviour during a network outage.",
"d24": "Interpreting the event log. Codes beginning with ASSOC relate to radio association; codes beginning with PWR relate to supply voltage.",
}
# gold = documents a support engineer judged genuinely useful for the question.
QUERIES = [
{"id": "q1",
"raw": "my outdoor sensor keeps dropping off wifi in winter after the last update",
"gold": {"d01", "d02", "d11", "d18"},
"slots": {"device": "outdoor sensor battery powered node",
"symptom": "disassociation link loss dropout radio association",
"condition": "sub-zero freezing cold temperature lithium voltage",
"trigger": "firmware 3.2.0 over the air update regression"},
"expand": "outdoor sensor disassociation dropout link loss freezing sub-zero cold firmware update"},
{"id": "q2",
"raw": "why does my battery die so fast when it is cold outside",
"gold": {"d01", "d03", "d11", "d18"},
"slots": {"device": "battery powered outdoor sensor cells",
"symptom": "low battery warning capacity collapse terminal voltage",
"condition": "cold freezing sub-zero ambient temperature"},
"expand": "battery lithium alkaline capacity voltage cold freezing sub-zero outdoor"},
{"id": "q3",
"raw": "device wont show up when i try to add it to my network",
"gold": {"d20", "d12", "d05"},
"slots": {"device": "device unit hub thermostat",
"symptom": "not discovered provisioning pairing setup",
"condition": "hidden SSID broadcast suppression re-provisioned"},
"expand": "provisioning pairing setup hidden SSID discovery reset re-provisioned"},
{"id": "q4",
"raw": "signal seems weak in the garden",
"gold": {"d06", "d07", "d04"},
"slots": {"device": "outdoor coverage repeater access point",
"symptom": "weak signal RSSI link quality decibel",
"condition": "range placement line of sight hop"},
"expand": "RSSI link quality signal strength repeater placement coverage range outdoor"},
{"id": "q5",
"raw": "should i roll back the firmware",
"gold": {"d08", "d02", "d14"},
"slots": {"device": "unit firmware partition",
"symptom": "unstable downgrade rollback recovery",
"trigger": "over the air update 3.2.0 3.1.4 release notes"},
"expand": "downgrade rollback recovery partition firmware release notes update unstable"},
]
def tokenize(text):
return re.findall(r"[a-z0-9]+", text.lower())
class BM25:
"""Textbook BM25 (Robertson/Sparck Jones), k1=1.5, b=0.75."""
def __init__(self, docs, k1=1.5, b=0.75):
self.k1, self.b = k1, b
self.ids = list(docs)
self.tf = {i: Counter(tokenize(docs[i])) for i in self.ids}
self.len = {i: sum(self.tf[i].values()) for i in self.ids}
self.avgdl = sum(self.len.values()) / len(self.ids)
df = Counter()
for i in self.ids:
df.update(self.tf[i].keys())
n = len(self.ids)
self.idf = {t: math.log(1 + (n - c + 0.5) / (c + 0.5)) for t, c in df.items()}
def score(self, query, doc_id):
s = 0.0
tf, dl = self.tf[doc_id], self.len[doc_id]
for t in tokenize(query):
if t not in tf:
continue
num = tf[t] * (self.k1 + 1)
den = tf[t] + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
s += self.idf.get(t, 0.0) * num / den
return s
def search(self, query, k=5):
scored = [(self.score(query, i), i) for i in self.ids]
scored.sort(key=lambda x: (-x[0], x[1]))
return [i for s, i in scored[:k] if s > 0]
def rrf(rankings, k=60):
"""Reciprocal rank fusion: sum 1/(k+rank) across per-branch rankings."""
pooled = Counter()
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
pooled[doc_id] += 1.0 / (k + rank)
return [d for d, _ in pooled.most_common()]
def recall_at_k(hits, gold, k):
return len(set(hits[:k]) & gold) / len(gold)
def mrr(hits, gold):
for rank, d in enumerate(hits, start=1):
if d in gold:
return 1.0 / rank
return 0.0
bm25 = BM25(CORPUS)
K = 5
totals = {s: {"recall": 0.0, "mrr": 0.0} for s in ("raw", "expand", "slots")}
print(f"{'query':6} {'strategy':9} {'recall@5':>9} {'MRR':>6} top-5")
print("-" * 78)
for q in QUERIES:
runs = {
"raw": bm25.search(q["raw"], k=K),
"expand": bm25.search(q["expand"], k=K),
# Each branch retrieves DEEPER than K: fusion needs candidates below
# the cutoff to work with, or it just re-ranks a short list.
"slots": rrf([bm25.search(v, k=2 * K) for v in q["slots"].values()])[:K],
}
for name, hits in runs.items():
r, m = recall_at_k(hits, q["gold"], K), mrr(hits, q["gold"])
totals[name]["recall"] += r
totals[name]["mrr"] += m
print(f"{q['id']:6} {name:9} {r:9.2f} {m:6.2f} {' '.join(hits)}")
print()
n = len(QUERIES)
print("=" * 78)
print(f"{'MEAN':16} {'recall@5':>9} {'MRR':>6}")
for name in ("raw", "expand", "slots"):
print(f"{name:16} {totals[name]['recall']/n:9.3f} {totals[name]['mrr']/n:6.3f}")
mean = {s: {m: totals[s][m] / n for m in totals[s]} for s in totals}
assert abs(mean["raw"]["recall"] - 0.433) < 0.01
assert abs(mean["expand"]["recall"] - 0.833) < 0.01
# transforming the query nearly doubles recall ...
assert mean["expand"]["recall"] > 1.9 * mean["raw"]["recall"]
# ... but decomposing into slots does NOT beat flat expansion here: it ties
assert abs(mean["slots"]["recall"] - mean["expand"]["recall"]) < 1e-9
assert mean["raw"]["mrr"] == 0.5 and mean["expand"]["mrr"] == 1.0
print("\nasserts passed")
calls = {"raw": 0, "expand": 1, "slots": 1}
searches = {"raw": 1, "expand": 1, "slots": sum(len(q["slots"]) for q in QUERIES) / n}
print(f"\n{'':16} {'LLM calls':>10} {'searches/query':>15}")
for name in ("raw", "expand", "slots"):
print(f"{name:16} {calls[name]:10} {searches[name]:15.1f}")
# Output:
# query strategy recall@5 MRR top-5
# ------------------------------------------------------------------------------
# q1 raw 0.50 0.50 d08 d01 d21 d14 d02
# q1 expand 0.50 1.00 d01 d02 d03 d15 d07
# q1 slots 0.75 1.00 d01 d02 d15 d18 d07
#
# q2 raw 1.00 1.00 d18 d03 d14 d01 d11
# q2 expand 1.00 1.00 d03 d01 d11 d18 d15
# q2 slots 0.75 1.00 d01 d03 d15 d18 d02
#
# q3 raw 0.00 0.00 d24 d17 d03 d08 d23
# q3 expand 1.00 1.00 d20 d12 d05
# q3 slots 1.00 1.00 d20 d05 d12 d09 d16
#
# q4 raw 0.00 0.00 d10 d05 d14 d02 d01
# q4 expand 0.67 1.00 d06 d07 d02 d11 d03
# q4 slots 0.67 1.00 d06 d10 d02 d07 d01
#
# q5 raw 0.67 1.00 d08 d18 d02 d10 d05
# q5 expand 1.00 1.00 d08 d02 d14 d18
# q5 slots 1.00 1.00 d08 d02 d18 d05 d14
#
# ==============================================================================
# MEAN recall@5 MRR
# raw 0.433 0.500
# expand 0.833 1.000
# slots 0.833 1.000
#
# asserts passed
#
# LLM calls searches/query
# raw 0 1.0
# expand 1 1.0
# slots 1 3.2
Two per-query results deserve attention, because the means hide them. On q1 — the genuinely multi-facet question — slots beat expansion, 0.75 to 0.50. On q2 — a narrow question about one thing — slots lost, 0.75 to 1.00, because splitting a focused question into three branches diluted it. They cancel. That is the tie, and it is the article's real lesson.
A caveat stated plainly: this corpus is constructed, and the size of the raw-to-expanded gain depends on how far apart the two vocabularies are. It demonstrates the mechanism and the cost ratio; it is not a forecast of the gain on your corpus. Measure yours.
6. Real-world example
A team building search over an internal engineering wiki shipped HyDE after it beat plain retrieval on their evaluation set. Conceptual questions improved markedly — "how does our rate limiter handle bursts?" now returned the design doc rather than a changelog.
Then the on-call complaints started. Questions naming a specific internal service returned documents about different services. The cause: asked "why is billing-reconciler timing out?", the model — which had never heard of billing-reconciler — wrote a fluent hypothetical answer about a plausible-sounding payment-reconciliation-service, complete with invented config keys. That fabricated name dominated the embedding, and retrieval landed confidently in the wrong part of the wiki. The answer was coherent, well-cited, and about the wrong system.
The failure was invisible in aggregate metrics because entity-lookup questions were a minority of the eval set and every other category improved. It surfaced only as a scattering of "this is wrong" reports.
The fix was routing, not tuning: detect whether the question contains a known entity, send entity questions to plain lexical retrieval, and reserve HyDE for conceptual ones. Which transform to apply is itself a decision that needs making per query — and the honest reading of the numbers in §5 is that a system doing this well spends its complexity budget on choosing the transform, not on making any single transform more elaborate.
7. Interview questions companies actually ask
Q1 [easy] "Why transform the query at all? Why not just search what the user typed?"
A Because users write in their vocabulary and documents in the author's, and the
two agree less than you'd guess -- Furnas 1987 measured under 20% spontaneous
agreement on term choice. In the harness above, raw BM25 gets recall@5 = 0.433
and returns nothing useful for 2 of 5 questions. One LLM call of expansion takes
it to 0.833. Retrieval quality is capped by the vocabulary gap before reranking
or chunking matters at all.
Q2 [easy] "What is HyDE and why does generating a wrong answer help?"
A Generate a hypothetical answer to the question, embed THAT, and search with its
vector. It works because a fake answer is written in answer-shaped language --
same register, length, jargon as the corpus -- so it lands nearer real answers
in embedding space than the question does. The facts being wrong is usually fine
because the draft is a search probe, never shown to the user.
Q3 [medium] "When does HyDE actively hurt?"
A Narrow entity lookups. If the model doesn't know the entity it invents a
plausible neighbour, that invented name dominates the embedding, and retrieval
lands confidently in the wrong neighbourhood. HyDE is strongest on broad
conceptual questions and weakest on specific ones -- the opposite of most
people's intuition. Route entity questions to lexical retrieval instead.
Q4 [medium] "You decompose a query into 4 sub-queries. What recall gain should you
expect?"
A Much less than the independence calculation suggests. 1 - (1-p)^4 with p=0.5
predicts 0.9375, but sub-queries of the same question against the same index
return heavily overlapping sets -- 39.8% overlap measured above -- so the
effective branch count is far below 4. Decomposition buys coverage only to the
extent the facets are genuinely disjoint.
Q5 [medium] "Why RRF rather than summing the relevance scores?"
A RRF reads only ranks, so it needs no score calibration across branches -- BM25
scores and cosine similarities aren't on a common scale and summing them lets
whichever branch has the widest range dominate. RRF(d) = sum 1/(k+rank_b(d)),
k=60. The large k makes it nearly rank-blind: 1st contributes 1/61, 9th 1/69,
only 12% less. It counts votes, not positions.
Q6 [hard] "Structured slots gave you the same recall as flat expansion at 3.2x the
searches. Why ship slots?"
A Not for recall -- for what the structure enables downstream. A missing slot is
a detectable gap, so the system can ask a clarifying question instead of
guessing. And slots ROUTE: device picks the index, symptom picks the specialist
retriever or tool. A flat expanded string has thrown that structure away. Ship
slots when you need routing or clarification; ship expansion when you only need
recall.
Q7 [hard] "How do you evaluate a transformation step in isolation?"
A Freeze everything downstream -- same index, same k, same reranker -- and vary
only T(q), so the delta is attributable. Report recall@k AND a rank-sensitive
metric (MRR/NDCG): in the harness, raw and expand differ on MRR 0.50 vs 1.00 at
q1 even where recall@5 is identical, because expansion moved a relevant doc to
position 1. Report per-query, not just the mean -- the means hid that slots won
q1 and lost q2.
8. When to use / tradeoffs
REACH FOR QUERY TRANSFORMATION WHEN:
+ users and authors use different vocabulary (jargon, formal vs colloquial)
+ questions arrive underspecified or as multi-part sentences
+ you measured raw retrieval and recall is the bottleneck, not ranking
+ one extra LLM call fits the latency budget
PICK THE TRANSFORM BY QUESTION SHAPE:
expansion cheapest, near-universal win; the default. Keep original terms.
decomposition genuinely multi-facet questions, disjoint facets
typed slots when you need ROUTING or gap-detection, not just recall
HyDE broad conceptual questions over a corpus of prose answers
step-back "why did X happen" questions needing a principle and a particular
| Situation | Why it breaks | Use instead |
|---|---|---|
| Corpus written in user vocabulary | No gap to close; the call is pure latency | Retrieve raw; spend on reranking |
| Question names a specific entity | HyDE invents a neighbouring entity and anchors retrieval to it | Lexical/hybrid retrieval on the entity |
| Facets overlap heavily | Union barely exceeds one query; ~40% duplicate results | Flat expansion, one search |
| Answer absent from corpus | Widens the net over an empty sea; retrieves confident nonsense | Post-retrieval grounding check |
| Hard latency SLA | Transform is sequential before retrieval, on every query | Cache transforms; skip on short queries |
| Narrow, focused question | Decomposition dilutes it — measured 0.75 vs 1.00 on q2 | Do not decompose single-facet questions |
Honest limits. The numbers here come from a 24-document corpus with 5 queries and a vocabulary gap I constructed deliberately; treat the ratios and the mechanism as transferable and the magnitudes as not. The gold labels are a judgement call, and recall@5 against a 3–4 document gold set moves in coarse steps — one document is worth 0.25. There is no statistical significance to speak of at n=5. Crucially, everything measured here is lexical BM25; a strong embedding model closes part of the vocabulary gap on its own, so the raw baseline would start higher and the headroom for expansion would be smaller — how much smaller is exactly the thing to measure on your own corpus rather than inherit from an article. Finally, the comparison holds transform quality constant by hand-writing the expansions; a real LLM transform introduces variance and occasional drift that this harness cannot show.
9. Summary + related articles
- Retrieval is capped by the vocabulary gap between how users ask and how authors write. Measured here: recall@5 of 0.433 raw, with two of five questions returning nothing useful.
- Expansion is the cheapest fix and usually the largest single win — 0.833 for one LLM call and no extra searches. Keep the original terms to prevent drift.
- Decomposition is not free recall. Branches of one question overlap (39.8% measured), so the independence calculation badly overpromises. Here it tied expansion at 3.2× the searches.
- Typed slots earn their cost through routing and gap-detection, not retrieval quality. Ship them for what the structure enables downstream.
- HyDE inverts the problem and wins on conceptual questions; it fails hardest on entity lookups, where it invents a plausible wrong entity and anchors to it.
- Choosing which transform to apply per query matters more than perfecting any one of them.
- Boundary: transformation assumes the question is underspecified, not unanswerable. It cannot tell you the answer is absent from the corpus — that needs a grounding check afterwards.
Related:
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — the other half of retrieval quality: transformation fixes the query, chunking fixes what it can match against
- Hallucination Detection & Grounding — the post-retrieval check that catches what transformation cannot: an answer the corpus does not support
- Search Systems — where BM25, hybrid retrieval, and recall@k sit in a full search architecture
- Study Designs & the Evidence Hierarchy — PICO's home discipline, and why structured questions took hold there first
- Backtesting, Baselines & Sensitivity Analysis — how to hold everything else fixed so a measured delta is attributable to the transform
- Agent Orchestration — what typed slots enable once you have them: routing a request to the right specialist
Resources
- Furnas, Landauer, Gomez & Dumais, "The Vocabulary Problem in Human-System Communication", Communications of the ACM 30(11), 1987 — the original measurement of term-choice disagreement.
- Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond", Foundations and Trends in Information Retrieval 3(4), 2009 — derives the BM25 formula used above.
- Cormack, Clarke & Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods", SIGIR 2009 — the source of RRF and of the
k = 60default. - Gao, Ma, Lin & Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels", arXiv:2212.10496 (ACL 2023) — the HyDE paper.
- Zheng et al., "Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models", arXiv:2310.06117 (ICLR 2024) — step-back prompting.
- Richardson, Wilson, Nishikawa & Hayward, "The Well-Built Clinical Question: A Key to Evidence-Based Decisions", ACP Journal Club, 1995 — the origin of PICO as a query schema. Volume/issue not independently verified.
- Pinecone, "Query Transformation" — https://www.pinecone.io/learn/query-transformation/