TL;DR — Retrieval hands you a ranked list; generation has to turn it into an answer someone can check. Three things decide whether that works, and none of them are the prompt's wording. Order matters: models attend unevenly across a long context, so placing the best chunk first rather than at random moves the probability it gets used from 75.8% to 95.0% — a 19.2-point swing from ordering alone, same chunks, same model. "Cited" is not "supported": a cheap resolve-check (does the cited id actually exist in the context?) catches 15.3% of bad claims for zero model calls, but 12.5% remain cited, resolvable, and wrong — 45% of all bad claims survive the cheap check and need an entailment judge. Failure needs a ladder: with a 6% no-results rate, 2% generation failure and 5% parse failure, a pipeline with no fallback returns something unusable 12.86% of the time; retry plus a regex fallback plus an explicit no-evidence reply takes that to 0.04% — and the floor is not zero. The last rung matters most: when retrieval found nothing, the usable answer is "no evidence", never a guess.
1. Simple explanation
You have ten chunks and a question. Now write the answer.
It is tempting to treat this as prompt wording — instructions, tone, a few examples. But the wording is rarely what fails. What fails is the plumbing around it: which chunks you include and in what order, whether the answer's citations point at anything real, and what happens on the queries where something goes wrong.
That last one is the most neglected. A RAG pipeline has several ways to produce nothing useful — retrieval returns no candidate above threshold, generation errors out, the model returns prose but omits the structured block your UI parses. Each is individually rare. Together, on a system with no fallbacks, they are common enough to be the thing users notice.
Analogy — a barrister preparing a closing argument. They have a box of evidence and must produce an argument a judge can verify. Three things decide whether it lands, and none is eloquence. Order: they lead with the strongest exhibit, because attention is finite and the jury remembers the opening. Attribution: every claim must point at an exhibit number that exists in the bundle — and pointing at a real exhibit that does not actually say what you claimed is worse than not pointing at all, because it survives a superficial check. What to do when the evidence isn't there: the professional answer is "the evidence does not establish this," not an eloquent guess. Systems that cannot say that are the ones that fabricate.
2. Diagram
ranked chunks ┌──────────────────────────────┐
from retrieval ───────────────► │ ① ASSEMBLE │
│ order, dedupe, truncate │
│ to a token budget │
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ ② GENERATE │
│ prose + inline citations │
│ + a machine-readable block│
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ ③ VERIFY │
│ do cited ids resolve? │ cheap
│ do sources support them? │ costly
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ ④ FALL BACK │
│ retry / regex / say │
│ "no evidence" │
└──────────────────────────────┘
① ORDER IS NOT COSMETIC (§5) ③ CITED ≠ SUPPORTED (§5)
best chunk first 95.0% cited + supported 72.3%
best chunk last 86.0% no citation 10.2% ┐ cheap
random order 75.8% id not in context 5.0% ┘ check
▲ 19.2 points, same chunks cited but WRONG 12.5% ← judge
needed
④ THE LADDER (§5)
no fallback 12.86% unusable
+ retry generation once 10.88%
+ regex-extract the block 6.27%
+ explicit "no evidence" reply 0.04% ← floor, not zero
3. How it works
3.1 Assembly: order, dedupe, budget
Three decisions before a token is generated.
Order. Models do not attend uniformly across a long context. The well-replicated finding is a U-shape: material at the beginning and end is used more reliably than material in the middle. Retrieval hands you a ranked list, so the default of "paste them in rank order" happens to be right — but only if you preserve it. Pipelines that shuffle, group by source, or sort by date are throwing away a free 19 points.
Dedupe. Near-duplicate chunks waste slots and bias the model toward whatever is repeated, which reads as corroboration when it is just one document indexed twice. Deduplication belongs upstream at merge time — see Multi-Index RAG: Merging Several Retrievers Into One Answer — but verify it happened before assembly.
Budget. More context is not better. Every chunk past the ones that answer the question is a distraction, and the marginal chunk is by construction the least relevant one you have. Fitting the budget by truncating the lowest-ranked chunks is right; truncating the last N characters of the assembled prompt is a bug that silently cuts your instructions.
Label each chunk with a stable identifier at assembly time. That identifier is what the citation contract in §3.2 depends on.
3.2 The citation contract, and why "cited" is not "supported"
Ask for citations and you will get them. That is the problem: citation presence is easy to produce and easy to check, and it is not the property you care about.
Three failure modes, in increasing difficulty:
NO CITATION the claim has no marker at all
-> trivially detectable, cheap to reject
UNRESOLVABLE ID cites [7] when only 5 chunks were supplied, or an
id that never existed
-> detectable with a set membership test, no model call
RESOLVABLE BUT cites a real chunk that does not actually support
UNSUPPORTED the claim
-> requires reading both and judging entailment
Measured: the first two together are 15.3% of claims and cost nothing to catch. The third is 12.5% — and 45% of all bad claims fall in it. A pipeline that only checks that citations resolve will report itself healthy while nearly half its bad claims pass through.
The cheap checks are still worth doing first, because they are free and they catch the majority. But do not mistake them for verification. The entailment check — NLI, an LLM judge, or span-overlap heuristics — is the only thing that touches the residue, and its methods and failure modes are covered in Hallucination Detection & Grounding.
Two design choices that make verification much easier:
- Require a quoted span, not just an id. "As stated in [3]" is hard to verify.
[3] "the effect persisted at 12 months"can be checked by exact substring against chunk 3 before any model is involved — and a model that must produce a verbatim quote fabricates less. - Make the id space small and explicit. Number the chunks
[1]–[10]in the prompt. Unresolvable ids become a set-membership test instead of a parsing problem.
3.3 Structured blocks inside prose
Applications usually need more than prose: a confidence value, a list of sources, a flag for whether the sources agreed. The common pattern is asking the model to emit prose plus a machine-readable block:
...answer text with inline citations [1] [4]...
<summary>{"confidence": "moderate", "sources": [1, 4], "conflict": false}</summary>
This works and it fails a few percent of the time — the block is missing, the JSON is malformed, the model wraps it in a code fence, or it emits two of them. Treat parse failure as an expected outcome with a defined path, not an exception.
The general problem of getting reliable structured output — schemas, constrained decoding, validation — is covered in Tool Use. What is specific here: never let a parse failure destroy a good answer. The prose is usually fine even when the block is malformed. Extract what you can, default the rest, and return the answer.
And keep the prompt itself out of your application code — versioned prompt files make the assembly step reviewable and diffable, which matters because prompts change more often than code. That is Prompt Management Architecture: Prompts as Files, Not Strings.
3.4 The fallback ladder
Enumerate the ways the step can produce nothing usable, then give each a rung:
FAILURE RUNG
generation errors / empty retry once (different seed or temperature)
structured block malformed regex-extract; failing that, default the
fields and keep the prose
retrieval returned nothing an explicit "no supporting evidence found"
-- this IS a usable answer
everything failed a plain error, surfaced honestly
Measured on a pipeline with 6% no-results, 2% generation failure and 5% parse failure:
no fallback 12.86% unusable
retry generation once 10.88%
retry + regex-extract the block 6.27%
retry + regex + explicit no-evidence reply 0.04%
Two things to take from that table. The rungs are cheap — a retry, a regex, a canned response — and together they remove almost all of a 12.86% failure rate. And the floor is not zero: 0.04% is generation failing twice in a row, which no amount of laddering removes. Systems that promise a valid answer 100% of the time are either lying or fabricating.
3.5 The rung that must not become fabrication
The last rung is the one to get right, and the tempting version is wrong.
When retrieval returns nothing above threshold, a "best-effort answer from whatever we found" is exactly the behaviour that produces confident, well-formatted nonsense — the failure mode that makes people distrust RAG systems generally. The correct fallback is an explicit statement that the corpus does not support an answer.
That is why the ladder above counts "explicit no-evidence reply" as a usable outcome. It is a correct answer to the question "what does our corpus say about this?" A guess is not.
This connects to the threshold problem: a threshold strict enough to keep bad context out will produce empty result sets on hard queries (Reranking: The Second Pass That Decides What the Model Sees). The pair only works if the empty case has an honest path.
3.6 Where this stops mattering
If your answers are short extractions from a single chunk, most of this is overhead — order does not matter with one chunk, and a citation contract is trivial. The machinery earns its cost when answers synthesise several sources, when someone downstream must verify a claim, or when being wrong is expensive. For a low-stakes internal search box, a retry and an honest empty state are enough.
4. The math
4.1 Ordering
With a positional use-probability profile u[i] and the best chunk placed at position p, the probability it is used is just u[p]:
u = [0.95, 0.90, 0.82, 0.72, 0.63, 0.60, 0.63, 0.70, 0.78, 0.86]
best chunk first -> u[0] = 95.0%
best chunk last -> u[9] = 86.0%
random position -> mean(u) = 75.8%
The profile is an assumption, taken from published "lost in the middle" results; the arithmetic on top of it is exact. What transfers is the ordering — first beats last beats random — and the rough magnitude, not the specific numbers.
Note that best-last (86.0%) beats random (75.8%), because the U-shape recovers at the end. If your assembly cannot preserve rank order, putting the top chunk at the very end is better than shuffling.
4.2 Where bad claims survive
Let c be the probability a claim carries a citation, r that the id resolves, and s that the source supports it:
P(correct) = c * r * s = 0.90 * 0.94 * 0.85 = 72.3%
P(no citation) = 1 - c = 10.2% (measured)
P(unresolvable) = c * (1 - r) = 5.0%
P(resolvable, wrong) = c * r * (1 - s) = 12.5%
The cheap check covers the middle two. The share of bad claims it misses:
12.5 / (10.2 + 5.0 + 12.5) = 45%
The lesson generalises past citations: the cheap check catches the failures that are cheap to make. Fabricating a plausible id is a low-probability slip; misreading a real source is the model's characteristic error. Verification effort should be aimed at the characteristic error, not the convenient one.
4.3 The ladder is multiplicative
Independent failure modes compose as survival probabilities:
P(usable) = P(retrieval ok or honest empty)
* P(generation ok after retries)
* P(block parsed or recovered)
Retrying an independent failure squares it (0.02 → 0.0004), which is why one retry buys most of the benefit and a second buys almost nothing. The measured 0.04% floor is exactly that squared term, and it is irreducible by more rungs of the same kind — you would need a different mechanism, not another retry.
5. Real code
Standard library only, deterministic, runs in about two seconds.
import random
SEED = 31
N = 20000
# Relative probability the model USES a chunk, by position in a 10-chunk
# context. Taken from the published U-shape: strong at the start, weaker in
# the middle, partial recovery at the end.
USE_BY_POSITION = [0.95, 0.90, 0.82, 0.72, 0.63,
0.60, 0.63, 0.70, 0.78, 0.86]
def expected_use(order):
"""order[i] = rank of the chunk placed at position i (0 = best chunk)."""
# only the BEST chunk actually answers the question here
return USE_BY_POSITION[order.index(0)]
def orderings():
k = len(USE_BY_POSITION)
return {
"relevance descending (best first)": list(range(k)),
"relevance ascending (best last)": list(reversed(range(k))),
"random": None,
}
def positional():
print("A. WHERE THE BEST CHUNK SITS IN THE PROMPT")
print(f"{'ordering':>38} {'P(best chunk used)':>20}")
print("-" * 60)
rng = random.Random(SEED)
res = {}
for name, order in orderings().items():
if order is None:
tot = 0.0
for _ in range(N):
o = list(range(len(USE_BY_POSITION)))
rng.shuffle(o)
tot += expected_use(o)
v = tot / N
else:
v = expected_use(order)
res[name] = v
print(f"{name:>38} {v:19.1%}")
best = max(res, key=res.get)
worst = min(res, key=res.get)
print(f"\n best ordering : {best} ({res[best]:.1%})")
print(f" worst ordering : {worst} ({res[worst]:.1%})")
print(f" spread from ORDERING ALONE: {res[best]-res[worst]:.1%} -- same "
f"chunks, same model, same prompt")
return res
def citations():
print("\n\nB. CITATION CONTRACTS")
print(" 'cited' and 'supported' are different properties")
rng = random.Random(SEED + 1)
# model behaviour per claim
P_CITES = 0.90 # emits a citation at all
P_ID_REAL = 0.94 # the cited id actually exists in the context
P_SUPPORTS = 0.85 # the cited chunk actually supports the claim
uncited = fake_id = unsupported = clean = 0
for _ in range(N):
if rng.random() > P_CITES:
uncited += 1
elif rng.random() > P_ID_REAL:
fake_id += 1
elif rng.random() > P_SUPPORTS:
unsupported += 1
else:
clean += 1
print(f"\n{'claim outcome':>26} {'share':>8} {'caught by a resolve-check?':>28}")
print("-" * 64)
for label, n, caught in (
("cited + supported", clean, "n/a (correct)"),
("no citation at all", uncited, "YES - cheap"),
("citation id not in context", fake_id, "YES - cheap"),
("cited but unsupported", unsupported, "NO - needs a judge")):
print(f"{label:>26} {n/N:8.1%} {caught:>28}")
cheap = (uncited + fake_id) / N
hard = unsupported / N
print(f"\n a resolve-check (does the id exist?) catches {cheap:.1%} of bad")
print(f" claims for zero model calls. The remaining {hard:.1%} are cited,")
print(f" resolvable, and WRONG -- only an entailment check finds those.")
print(f" {hard/(cheap+hard):.0%} of all bad claims survive the cheap check.")
return cheap, hard
def fallback():
print("\n\nC. THE FALLBACK LADDER")
rng = random.Random(SEED + 2)
P_NO_RESULTS = 0.06 # retrieval returned nothing above threshold
P_GEN_FAIL = 0.02 # generation errored or produced empty output
P_PARSE_FAIL = 0.05 # structured block missing/malformed
print(f"{'policy':>34} {'unusable answers':>18}")
print("-" * 56)
policies = {}
for name in ("no fallback",
"retry generation once",
"retry + regex-extract the block",
"retry + regex + explicit no-evidence reply"):
bad = 0
for _ in range(N):
if rng.random() < P_NO_RESULTS:
# nothing retrieved: only an explicit no-evidence reply
# counts as a usable answer here
if "no-evidence" not in name:
bad += 1
continue
gen_bad = rng.random() < P_GEN_FAIL
if gen_bad and "retry" in name:
gen_bad = rng.random() < P_GEN_FAIL
if gen_bad:
bad += 1
continue
if rng.random() < P_PARSE_FAIL:
if "regex" not in name:
bad += 1
policies[name] = bad / N
print(f"{name:>34} {bad / N:17.2%}")
final = policies['retry + regex + explicit no-evidence reply']
print(f"\n each rung is cheap; the ladder takes "
f"{policies['no fallback']:.2%} -> {final:.2%}")
print(f" the floor is NOT zero: {final:.2%} is generation failing twice.")
print(" and note what the last rung is -- when retrieval found nothing the")
print(" usable answer is an explicit 'no evidence', never a guess.")
return policies
pos = positional()
cheap, hard = citations()
pol = fallback()
# claims made in the prose
best = max(pos.values())
worst = min(pos.values())
assert pos["relevance descending (best first)"] == best, \
"best-first must be the strongest ordering"
assert best - worst > 0.15, "ordering alone must move the number a lot"
assert pos["random"] < best, "random ordering must lose to best-first"
assert hard > 0.10, "unsupported-but-cited must be a large residue"
assert cheap > hard, "the cheap check must still catch the majority"
assert pol["no fallback"] > 4 * pol["retry + regex + explicit no-evidence reply"]
assert pol["retry + regex + explicit no-evidence reply"] > 0, (
"the ladder must not claim a zero floor")
print("\nasserts passed")
# Output:
# A. WHERE THE BEST CHUNK SITS IN THE PROMPT
# ordering P(best chunk used)
# ------------------------------------------------------------
# relevance descending (best first) 95.0%
# relevance ascending (best last) 86.0%
# random 75.8%
#
# best ordering : relevance descending (best first) (95.0%)
# worst ordering : random (75.8%)
# spread from ORDERING ALONE: 19.2% -- same chunks, same model, same prompt
#
#
# B. CITATION CONTRACTS
# 'cited' and 'supported' are different properties
#
# claim outcome share caught by a resolve-check?
# ----------------------------------------------------------------
# cited + supported 72.3% n/a (correct)
# no citation at all 10.2% YES - cheap
# citation id not in context 5.0% YES - cheap
# cited but unsupported 12.5% NO - needs a judge
#
# a resolve-check (does the id exist?) catches 15.3% of bad
# claims for zero model calls. The remaining 12.5% are cited,
# resolvable, and WRONG -- only an entailment check finds those.
# 45% of all bad claims survive the cheap check.
#
#
# C. THE FALLBACK LADDER
# policy unusable answers
# --------------------------------------------------------
# no fallback 12.86%
# retry generation once 10.88%
# retry + regex-extract the block 6.27%
# retry + regex + explicit no-evidence reply 0.04%
#
# each rung is cheap; the ladder takes 12.86% -> 0.04%
# the floor is NOT zero: 0.04% is generation failing twice.
# and note what the last rung is -- when retrieval found nothing the
# usable answer is an explicit 'no evidence', never a guess.
#
# asserts passed
The three probabilities in part B — P_CITES, P_ID_REAL, P_SUPPORTS — are the ones to replace with your own. They are measurable: sample 100 answered queries, and for each claim check whether it carries a citation, whether the id resolves, and whether the source actually supports it. The third takes real effort to label, which is exactly why most teams measure only the first two and conclude their citations are fine.
6. Real-world example
A team shipped a RAG assistant with a strict citation requirement: every sentence had to carry a source marker, and a validator rejected any answer with an unresolvable id. The validator passed 96% of answers. They treated that as a quality metric and moved on.
Six weeks later a customer disputed an answer. The claim was cited. The id resolved. The cited document said something related but materially different — a threshold applied to a different product tier.
Auditing 200 answers by hand found the pattern: the id-resolution check had been catching a small, easy class of error while the model's characteristic mistake — citing a real, adjacent, non-supporting chunk — passed every automated check they had. Their "96% valid citations" number had never measured support.
Two changes fixed it, and the cheap one mattered more. They required a verbatim quoted span alongside each id, verified by exact substring match against the chunk. That is still free — no model call — and it collapsed the adjacent-chunk error, because a model forced to quote cannot cite a chunk that does not contain the words. Then they added an entailment check on the small residue.
The lesson generalises: an automated check that passes 96% of the time is telling you about the check, not the system. Ask which failure mode it cannot see, and estimate how big that class is before trusting the number.
7. Interview questions companies actually ask
Q1 [easy] "Does the order of chunks in the prompt matter?"
A Yes, substantially. Models attend unevenly across long context -- a U-shape,
strong at the start and end, weaker in the middle. Measured on a published
profile: best chunk first gives 95.0% use, last 86.0%, random 75.8%. That's
19.2 points from ordering alone, with identical chunks and prompt. Retrieval
already ranks them; just don't destroy the order.
Q2 [easy] "Your answers all carry citations. Are they trustworthy?"
A Unknown from that alone. Citation PRESENCE, id RESOLUTION, and actual SUPPORT
are three different properties. Measured: 15.3% of claims fail the first two
(free to catch) but 12.5% cite a real, resolvable chunk that doesn't support
the claim -- 45% of all bad claims survive the cheap check.
Q3 [medium] "How do you make citations cheap to verify?"
A Require a verbatim quoted span alongside the id, and verify by exact substring
against that chunk -- no model call. It also suppresses the error itself: a
model forced to quote can't cite a chunk that doesn't contain the words. Keep
the id space small and explicit ([1]-[10]) so resolution is a set-membership
test.
Q4 [medium] "The model sometimes omits the JSON block your UI parses. What do you
do?"
A Treat it as an expected outcome, not an exception. Retry once, then
regex-extract, then default the fields and keep the prose -- a malformed block
should never destroy a good answer. Measured, regex recovery alone took
unusable answers from 10.88% to 6.27%.
Q5 [medium] "Retrieval returns nothing above threshold. What should the system say?"
A That it found no supporting evidence. That IS the correct answer to 'what does
our corpus say about this'. A 'best-effort answer from whatever we found' is
precisely the behaviour that produces confident nonsense. Strict thresholds and
honest empty states are a package -- you can't ship one without the other.
Q6 [medium] "Why does one retry help but a second barely does?"
A Independent failures compose multiplicatively, so retrying squares the failure
probability: 2% becomes 0.04%. The first retry removes 98% of that mode; a
second removes 98% of what's left, which is already negligible. Getting below
the floor needs a DIFFERENT mechanism, not another retry of the same one.
Q7 [hard] "Your citation validator passes 96% of answers. What's wrong with using
that as a quality metric?"
A It measures the check, not the system. Ask which failure mode the check cannot
see and how big that class is. An id-resolution check misses the model's
characteristic error -- citing a real but non-supporting chunk -- which is the
larger class. Cheap checks catch the failures that are cheap to make.
Q8 [hard] "Is more context always better for generation?"
A No. The marginal chunk is by construction your least relevant one, and it
competes for attention with the chunks that answer the question. Past the point
where the answer is covered, added context dilutes -- the same effect that makes
wide agentic fan-out lose to a narrower one. Fit the budget by dropping the
lowest-ranked chunks, never by truncating the assembled prompt, which silently
cuts your instructions.
8. When to use / tradeoffs
ASSEMBLY CHECKLIST
+ preserve retrieval rank order (or put the best chunk LAST, not random)
+ dedupe before assembly; repeated chunks read as corroboration
+ label chunks with small explicit ids the citation contract can use
+ fit the budget by dropping lowest-ranked chunks, not truncating the prompt
VERIFICATION LADDER (cheapest first)
1. citation present? free
2. id resolves? free -- catches 15.3% of bad claims
3. quoted span matches? free -- collapses the adjacent-chunk error
4. source entails the claim? costs a call -- the remaining 45%
| Situation | Why it breaks | Use instead |
|---|---|---|
| Shuffling or re-grouping chunks | Loses 19.2 points of use probability | Preserve rank order |
| Truncating the assembled prompt | Silently cuts instructions | Drop lowest-ranked chunks |
| Checking only that citations resolve | Misses 45% of bad claims | Add quoted spans, then entailment |
| Treating parse failure as an exception | A malformed block destroys a good answer | Regex fallback; default fields |
| "Best-effort answer" on no results | The exact path to confident fabrication | Explicit "no evidence found" |
| Two or three retries | Failure squares; second retry buys ~nothing | One retry, then a different mechanism |
| Promising a 100% valid-answer rate | The floor is 0.04%, not 0 | Surface the residue honestly |
| Prompt as a string literal in code | Not reviewable, diverges across call sites | Versioned prompt files |
Honest limits. The positional profile in part A is assumed, not measured — it encodes the published U-shape, and the exact values differ by model, context length, and task; long-context models have improved on the middle-of-context weakness since that work, so the 19.2-point spread should be read as "large enough to care about", not as a forecast. Parts B and C are probability models with parameters I chose (P_SUPPORTS = 0.85, P_PARSE_FAIL = 0.05, and so on); the structure — three distinct citation failure modes, multiplicative ladder rungs, a non-zero floor — is what transfers, and every rate is measurable on your own system in an afternoon. The model also assumes the three citation failures are independent, which they are not: an answer that omits one citation tends to omit several, so real per-answer outcomes are more clustered than per-claim rates suggest. Nothing here measures answer quality in any richer sense than "supported and non-empty" — fluency, completeness, and usefulness are separate axes this harness is silent on.
9. Summary + related articles
- Order is not cosmetic. Best chunk first vs random moved use probability 75.8% → 95.0%. Retrieval already ranks; preserve it. If you cannot, put the best chunk last (86.0%) rather than shuffling.
- "Cited" is not "supported." The free resolve-check catches 15.3% of bad claims, but 12.5% are cited, resolvable and wrong — 45% of all bad claims survive it.
- Require a verbatim quoted span. Still free to verify, and it suppresses the model's characteristic error rather than just detecting it.
- Build a fallback ladder: retry → regex-extract → explicit no-evidence. Took unusable answers from 12.86% → 0.04%.
- The floor is not zero. 0.04% is generation failing twice; more rungs of the same kind will not remove it.
- When retrieval finds nothing, say so. A best-effort guess is the fabrication path, and strict thresholds only work when the empty case has an honest exit.
- Boundary: for short single-chunk extractions this machinery is overhead. It earns its cost when answers synthesise sources and someone must verify a claim.
Related:
- Hallucination Detection & Grounding — §3.3 covers the entailment check that handles the 45% residue the cheap check misses
- Prompt Management Architecture: Prompts as Files, Not Strings — keeping the assembly prompt out of application code, versioned and reviewable
- Tool Use — reliable structured output in general: schemas, constrained decoding, validation
- Reranking: The Second Pass That Decides What the Model Sees — §3.4 on score thresholds, the other half of the empty-result problem
- Multi-Index RAG: Merging Several Retrievers Into One Answer — §3.2 dedupe, which must happen before assembly or repeats read as corroboration
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — separating a generation failure from a retrieval failure, so you know which of these levers to pull
Resources
- Liu, Lin, Hewitt et al., "Lost in the Middle: How Language Models Use Long Contexts", TACL 2024 (arXiv:2307.03172) — the positional U-shape assumed in §4.1.
- Gao, Yen, Yu & Chen, "Enabling Large Language Models to Generate Text with Citations", EMNLP 2023 (arXiv:2305.14627) — the ALCE benchmark; citation precision and recall as distinct measures.
- Menick, Trebacz, Mikulik et al., "Teaching Language Models to Support Answers with Verified Quotes", arXiv:2203.11147, 2022 — the verbatim-quote contract in §3.2.
- Rashkin, Nikolaev, Lamm et al., "Measuring Attribution in Natural Language Generation Models", Computational Linguistics 49(4), 2023 — the AIS framework; a precise definition of "supported by a source".
- Es, James, Espinosa-Anke & Schockaert, "RAGAS: Automated Evaluation of Retrieval Augmented Generation", EACL 2024 (arXiv:2309.15217) — faithfulness scoring for the entailment rung.