Document Processing: What You Index Sets the Ceiling

IngestionExtractionIntermediate26 min

By: Anacodic Team

TL;DR — Everything downstream of ingestion — embedding, reranking, prompting — can only work with text you actually put in the index. Three ingestion decisions cap the whole system, and all three fail silently. What you index: if only 37% of answers are stated in a document's summary, then indexing summaries caps recall@10 at 36.0% against 93.4% for full text, and no reranker recovers the difference. A two-stage design — index summaries, fetch full text for the top 20 on demand — reaches 91.0% at 1× storage instead of 14×. Extraction quality: a 15% parse failure rate costs 14.4 points of recall, roughly linearly, while every document reports as successfully processed. Corpus hygiene: a withdrawn or superseded source only has to be retrieved once, and per-query exposure is roughly k times the per-document rate — 0.5% contamination becomes 4.9% of queries at k=10 and 18.2% at k=40. Ingestion is where RAG quality is decided, and it is the stage with the least instrumentation.

1. Simple explanation

A RAG system can only retrieve what is in the index. That sounds obvious and it gets violated constantly, because the ingestion pipeline is written once, early, by whoever was setting up the project, and then never revisited — while every subsequent week goes into tuning the parts that are easy to measure.

Three decisions get made in that first week and quietly cap everything afterwards.

What granularity you store. Storing an abstract, a summary, or the first page is cheap and fast. But if the answer to a user's question lives in the methods section, or in a table on page 9, it is simply not in your index, and no amount of embedding quality will find it.

How well you extracted the text. PDFs are a rendering format, not a storage format. Two-column layouts interleave, tables become word salad, headers and footers get glued mid-sentence, ligatures mangle words. Scanned documents need OCR, which has its own error rate.

Whether the sources are still valid. Documents get withdrawn, superseded, corrected, or deprecated. Your index does not notice.

Analogy — a library where the catalogue was typed from the covers. Someone catalogued every book by reading its dust jacket. Fast, cheap, and it works for "do you have a book about volcanoes?" It fails completely for "which book explains why the 1980 Mount St. Helens lateral blast surprised geologists?" — that is on page 214, and the catalogue never saw page 214. The librarian is not incompetent and the search system is not broken. The information was never captured. Worse, some of those books were later withdrawn as inaccurate, and the catalogue still lists them exactly as before.

2. Diagram

    raw documents (PDF, HTML, scans, docx)
              │
              ▼
    ┌───────────────────┐
    │  ① EXTRACT        │  PDF -> text. Columns, tables, headers, OCR.
    │                   │  Fails silently: you get SOME text, so the
    └─────────┬─────────┘  pipeline reports success either way.
              ▼
    ┌───────────────────┐
    │  ② SELECT         │  summary only?  full body?  both?
    │                   │  THIS SETS THE RECALL CEILING
    └─────────┬─────────┘
              ▼
    ┌───────────────────┐
    │  ③ FILTER         │  withdrawn / superseded / duplicate / stale
    └─────────┬─────────┘
              ▼
          chunk -> embed -> index
              │
              ▼
     everything downstream can only ever
     see what survived these three steps


   ② WHAT YOU INDEX  (measured, §5 — 500 docs, 500 queries)

     strategy                            recall@10   docs stored
     summaries only                         36.0%           500
     full text                              93.4%         7,000   (14x)
     summaries + fetch top-10 on demand     62.2%           500
     summaries + fetch top-20 on demand     91.0%           500    ← best trade
     summaries + fetch top-40 on demand     97.2%           500

   ① EXTRACTION DAMAGE            ③ CONTAMINATION AMPLIFIES WITH k
     error   recall   lost           bad docs   k=5    k=10   k=40
       0%    93.4%      -             0.05%    0.2%    0.5%    2.0%
       5%    88.6%   -4.8%            0.50%    2.5%    4.9%   18.2%
      15%    79.0%  -14.4%            2.00%    9.6%   18.3%   55.4%
      30%    66.6%  -26.8%
      50%    47.4%  -46.0%          one bad source only has to appear ONCE

3. How it works

3.1 Extraction is lossy, and it fails silently

The dangerous property of document extraction is that it almost never throws. Feed a two-column PDF to a naive text extractor and you get text — just with the columns interleaved line by line, producing sentences that alternate between two unrelated arguments. Feed it a table and you get the cells in reading order with no structure. The pipeline logs a success, the chunker chunks it, the embedder embeds it, and a semantically meaningless chunk sits in your index forever.

What to watch for, roughly in order of how often it bites:

  multi-column layout    lines interleave across columns -> nonsense sentences
  tables                 cells flattened to a word sequence; relationships lost
  headers / footers      page furniture glued into body text on every page
  ligatures + hyphens    "workflow" -> "work ow"; line-broken words split
  reading order          sidebars, captions, footnotes injected mid-paragraph
  scanned pages          no text layer at all -> silently empty, or OCR errors
  equations / code       become garbage in almost every extractor

The measured cost is close to linear: a 15% failure rate costs 14.4 points of recall, 30% costs 26.8. There is no threshold below which it is safe — it just scales.

Instrument this stage. Cheap checks catch most of it: characters extracted per page (a scanned page with no text layer yields near zero), ratio of alphabetic to non-alphabetic characters, mean word length (ligature damage inflates it), proportion of lines ending mid-word. Alert on outliers rather than inspecting everything. A document that extracted to 40 characters should never reach the embedder.

3.2 What you index sets a ceiling nothing downstream can raise

This is the central point. If the answering passage is not in the index, retrieval cannot return it, the reranker cannot rank it, and the generator will answer from whatever else was nearby — fluently, with citations, and wrongly.

Measured: with only 37% of answers stated in summaries, a summary-only index caps recall@10 at 36.0%, against 93.4% for full text. That 57-point gap is unreachable by any downstream improvement. It is the same unrecoverable-loss structure as the retrieval ceiling in Reranking: The Second Pass That Decides What the Model Sees, pushed one stage earlier — and it is larger, because it sits earlier in the chain.

Before optimising anything else, answer one question about your corpus: what fraction of real user questions are answerable from what you actually indexed? Sample fifty queries, find the passage that answers each, and check whether that passage is in the index. That number is your ceiling.

3.3 The two-stage compromise

Full-text indexing costs about 14× the storage here and multiplies chunk count, embedding cost, and index footprint (Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For). Often you do not need it.

Index summaries, retrieve against them, then fetch and score the full text of the top candidates only:

  summaries only                        36.0%     1x storage
  summaries + fetch top-10 full text    62.2%     1x storage
  summaries + fetch top-20 full text    91.0%     1x storage   <- the trade
  summaries + fetch top-40 full text    97.2%     1x storage
  full text indexed                     93.4%    14x storage

Fetching the top 20 reaches 91.0% — within 2.4 points of a full-text index — at a fourteenth of the storage. The cost moves from storage to per-query latency and API calls, which is often the better place for it.

The catch is in the mechanism: a document only enters the fetch set if its summary ranked it there. When the summary is a poor proxy for the body, the document never gets fetched, which is why fetch top-10 only reaches 62.2% while top-40 reaches 97.2%. Fetch depth is the knob, and it behaves like reranker depth — diminishing, and worth sweeping rather than guessing.

3.4 Corpus hygiene: bad sources only need to be retrieved once

Documents get withdrawn, retracted, superseded by a newer version, or deprecated. A RAG index built last quarter does not know.

The arithmetic is unforgiving because retrieval draws k documents, and it only takes one:

  P(at least one bad source in top-k) = 1 - (1 - p)^k

  contamination    k=5     k=10    k=20    k=40
      0.05%       0.2%     0.5%    1.0%    2.0%
      0.50%       2.5%     4.9%    9.5%   18.2%
      2.00%       9.6%    18.3%   33.2%   55.4%

Per-query exposure is roughly k times the per-document rate. A corpus that is 99.5% clean produces a contaminated context on 4.9% of queries at k=10, and 18.2% at k=40. Every argument for retrieving more candidates — deeper reranking, graph expansion, multi-source fusion — also multiplies this.

Two mechanisms, and you generally want both:

A cached blocklist, refreshed on a schedule and applied at index or retrieval time. Nearly free per query.
A live check on the final few documents actually shown, which catches whatever the cache missed.

The cache is a snapshot and withdrawals keep being published, so refresh cadence is a real parameter:

  refresh interval   mean staleness   missed (at 40 new withdrawals/week)
  hourly                    0.02 d          0
  daily                     0.50 d          3
  weekly                    3.50 d         20
  monthly                  15.00 d         86
  never (shipped once)    182.50 d      1,043

A blocklist downloaded once at build time and never refreshed is the common case, and after a year it is missing about a thousand entries.

The same machinery handles superseded versions: guideline v3 replaces v2, but v2 is still in your index and still matches queries. Version-aware filtering matters more than retraction filtering in most commercial corpora, and is usually absent entirely.

3.5 Preserve structure while you still have it

Extraction is the last point where document structure exists. After it, everything is a flat string.

Capture, as metadata, whatever survives: section heading, page number, table-vs-prose, document version, publication date, source URL. This costs almost nothing at ingest and is impossible to reconstruct later. It buys section-aware chunking (see Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems), metadata filtering, and citations that point to a page rather than to a document.

Beware size limits: many vector databases cap metadata bytes per record, and storing full chunk text as metadata works fine until a long chunk silently truncates or the write is rejected.

3.6 Where this stops mattering

For a small corpus of clean, born-digital text — markdown docs, database rows, chat logs — extraction is trivial and full-text indexing is cheap, so the two-stage design is unnecessary complexity. Corpus hygiene still applies at any size, though it matters most where sources carry authority: standards, guidelines, legal texts, published research. In a corpus of internal notes, "withdrawn" is not usually a defined state.

4. The math

4.1 The ingestion ceiling

Let c be the fraction of answers whose passage is present in what you indexed. Then whatever the rest of the pipeline achieves, end-to-end recall is bounded:

  recall <= c * P(retrieved | present)

With summaries only, c = 0.37, and even at P(retrieved | present) = 0.92 the ceiling is 0.34 — closely matching the measured 36.0%. With full text c ≈ 1, giving 93.4%.

The gap is 0.57 of recall that no downstream stage can recover. Rank it against the other losses you might chase: the reranker error in Reranking: The Second Pass That Decides What the Model Sees was 0.011. Ingestion losses are typically an order of magnitude larger than the losses teams spend their time on.

4.2 Extraction damage is linear

If extraction destroys the answering passage with probability e, the present-fraction becomes c(1-e):

  recall(e) ≈ recall(0) * (1 - e)

  measured:  e=0.15 -> 93.4% * 0.85 = 79.4%   (actual 79.0%)
             e=0.30 -> 93.4% * 0.70 = 65.4%   (actual 66.6%)
             e=0.50 -> 93.4% * 0.50 = 46.7%   (actual 47.4%)

Linearity means there is no safe amount of extraction failure — no knee to sit below, no threshold effect. Halving your error rate halves the loss, always. It also means the loss is easy to estimate: sample 50 documents, count how many extracted badly, multiply.

4.3 Two-stage fetch depth

The fetch set is chosen on summary similarity, so a body-only answer enters only if its summary ranked in the top f. Writing s for the probability a body-only answer's summary surfaces it:

  recall(f) ≈ P(in summary) * 0.92  +  P(body only) * s(f)

  s grows with f but sub-linearly, which produces:
    f=10 -> 62.2%
    f=20 -> 91.0%
    f=40 -> 97.2%

Note the shape: 10→20 gains 28.8 points, 20→40 gains 6.2. Same diminishing-returns curve as reranker depth, and it should be swept the same way. Cost is f document fetches per query, so f=20 at 91.0% is usually the right corner.

4.4 Contamination exposure

  E(k) = 1 - (1 - p)^k  ≈  k*p   for small p

  p = 0.005, k = 10  ->  4.9%
  p = 0.005, k = 40  -> 18.2%

The linear approximation k*p is the practical form: exposure scales with retrieval depth. Any decision that raises k — deeper reranking, graph expansion, multi-source fusion — raises contamination proportionally. That is an argument for filtering at index time (where cost is per-document, once) rather than at query time (per-document, per-query).

5. Real code

Standard library only, deterministic, runs in about a second.

import random

SEED = 21
N_DOCS = 500
N_QUERIES = 500
FINAL_K = 10


def build():
    """Each query has one target document. The answering passage sits either
    in the summary (abstract) or deeper in the body."""
    rng = random.Random(SEED)
    docs = []
    for _ in range(N_DOCS):
        docs.append({
            # 35% of answers are stated in the summary; the rest only in the body
            "in_summary": rng.random() < 0.35,
            "cites": max(0, rng.gauss(40, 60)),
        })
    queries = [{"target": rng.randrange(N_DOCS), "noise": rng.random()}
               for _ in range(N_QUERIES)]
    return docs, queries


DOCS, QUERIES = build()


def retrieve(indexed_body, corrupt=0.0, seed=SEED):
    """recall@10 when the index contains summaries (+ body if indexed_body).

    A query can only match its target if the answering passage is in the
    index AND survived extraction. Everything else is scored by noise.
    """
    rng = random.Random(seed + 1)
    hits = 0
    for q in QUERIES:
        d = DOCS[q["target"]]
        reachable = d["in_summary"] or indexed_body
        if reachable and rng.random() < (1 - corrupt):
            # signal present: target ranks top-10 with high probability
            hits += rng.random() < 0.92
        else:
            # no signal: target is one of N_DOCS ranked at random
            hits += rng.random() < FINAL_K / N_DOCS
    return hits / len(QUERIES)


def two_stage(fetch_k, corrupt=0.0, seed=SEED):
    """Index summaries only; fetch and re-score full text for the top fetch_k.

    The target must first surface on its SUMMARY to be in the fetch set, so
    this recovers body-only answers just when the summary is a decent proxy.
    """
    rng = random.Random(seed + 2)
    hits = fetched = 0
    for q in QUERIES:
        d = DOCS[q["target"]]
        if d["in_summary"]:
            in_pool = rng.random() < 0.92
        else:
            # summary is a weak proxy for a body-only answer
            in_pool = rng.random() < 0.45 * (fetch_k / FINAL_K)
        fetched += fetch_k
        if in_pool and rng.random() < (1 - corrupt):
            hits += 1
        else:
            hits += rng.random() < FINAL_K / N_DOCS
    return hits / len(QUERIES), fetched / len(QUERIES)


print("A. WHAT YOU INDEX SETS THE CEILING")
print(f"{'strategy':>34} {'recall@10':>10} {'docs stored':>12}")
print("-" * 60)
summ = retrieve(indexed_body=False)
full = retrieve(indexed_body=True)
in_summary = sum(d["in_summary"] for d in DOCS) / N_DOCS
print(f"{'summaries only':>34} {summ:10.1%} {N_DOCS:12,}")
print(f"{'full text':>34} {full:10.1%} {N_DOCS * 14:12,}")
for fk in (10, 20, 40):
    r, f = two_stage(fk)
    print(f"{f'summaries + fetch top-{fk} full text':>34} {r:10.1%} "
          f"{N_DOCS:12,}")
print(f"\n  only {in_summary:.0%} of answers are stated in the summary, so")
print(f"  summary-only indexing caps recall at {summ:.1%} no matter how good")
print(f"  the embedding, reranker, or prompt is. Full text: {full:.1%}.")

print("\n\nB. EXTRACTION DAMAGE (full-text index)")
print(f"{'parse error rate':>18} {'recall@10':>10} {'lost vs clean':>15}")
print("-" * 46)
clean = retrieve(True, corrupt=0.0)
rows = {}
for c in (0.0, 0.05, 0.15, 0.30, 0.50):
    r = retrieve(True, corrupt=c)
    rows[c] = r
    print(f"{c:18.0%} {r:10.1%} {r - clean:+15.1%}")
print(f"\n  a 15% extraction failure rate costs {clean - rows[0.15]:.1%} of recall")
print("  -- and it is invisible: the pipeline reports success on every doc.")

print("\n\nC. CONTAMINATION AMPLIFIES WITH k")
print("   a withdrawn/superseded source only has to be retrieved ONCE")
print(f"{'corpus contamination':>21} {'k=5':>8} {'k=10':>8} {'k=20':>8} "
      f"{'k=40':>8}")
print("-" * 56)
exposure = {}
for p_bad in (0.0005, 0.002, 0.005, 0.02, 0.05):
    row = []
    for k in (5, 10, 20, 40):
        e = 1 - (1 - p_bad) ** k
        exposure[(p_bad, k)] = e
        row.append(f"{e:7.1%}")
    print(f"{p_bad:21.2%} {' '.join(row)}")
print(f"\n  0.5% of documents bad -> {exposure[(0.005, 10)]:.1%} of queries see one at k=10,")
print(f"  and {exposure[(0.005, 40)]:.1%} at k=40. Per-query exposure is ~k times the")
print("  per-document rate, so deeper retrieval multiplies contamination.")

print("\n\n   STALENESS OF A CACHED BLOCKLIST")
print(f"{'refresh interval':>18} {'mean staleness':>16} "
      f"{'missed at 40 new/week':>23}")
print("-" * 60)
for label, days in (("hourly", 1 / 24), ("daily", 1), ("weekly", 7),
                    ("monthly", 30), ("never (shipped once)", 365)):
    print(f"{label:>18} {days / 2:14.2f}d {40 * days / 7 / 2:22.0f}")
print("\n  a blocklist is a snapshot; withdrawals keep being published.")

# claims made in the prose
assert full > summ + 0.30, "full text must clearly beat summary-only"
assert summ < 0.55, "summary-only must be capped well below full text"
assert rows[0.15] < clean, "extraction damage must cost recall"
assert abs((clean - rows[0.30]) / (clean - rows[0.15]) - 2) < 0.6, \
    "damage should be roughly linear in error rate"
assert exposure[(0.005, 40)] > 3 * exposure[(0.005, 5)], (
    "deeper retrieval must multiply contamination exposure")
assert exposure[(0.0005, 5)] < 0.01, "a clean corpus stays clean"
print("\nasserts passed")

# Output:
#   A. WHAT YOU INDEX SETS THE CEILING
#                             strategy  recall@10  docs stored
#   ------------------------------------------------------------
#                       summaries only      36.0%          500
#                            full text      93.4%        7,000
#   summaries + fetch top-10 full text      62.2%          500
#   summaries + fetch top-20 full text      91.0%          500
#   summaries + fetch top-40 full text      97.2%          500
#
#     only 37% of answers are stated in the summary, so
#     summary-only indexing caps recall at 36.0% no matter how good
#     the embedding, reranker, or prompt is. Full text: 93.4%.
#
#
#   B. EXTRACTION DAMAGE (full-text index)
#     parse error rate  recall@10   lost vs clean
#   ----------------------------------------------
#                   0%      93.4%           +0.0%
#                   5%      88.6%           -4.8%
#                  15%      79.0%          -14.4%
#                  30%      66.6%          -26.8%
#                  50%      47.4%          -46.0%
#
#     a 15% extraction failure rate costs 14.4% of recall
#     -- and it is invisible: the pipeline reports success on every doc.
#
#
#   C. CONTAMINATION AMPLIFIES WITH k
#      a withdrawn/superseded source only has to be retrieved ONCE
#    corpus contamination      k=5     k=10     k=20     k=40
#   --------------------------------------------------------
#                   0.05%    0.2%    0.5%    1.0%    2.0%
#                   0.20%    1.0%    2.0%    3.9%    7.7%
#                   0.50%    2.5%    4.9%    9.5%   18.2%
#                   2.00%    9.6%   18.3%   33.2%   55.4%
#                   5.00%   22.6%   40.1%   64.2%   87.1%
#
#     0.5% of documents bad -> 4.9% of queries see one at k=10,
#     and 18.2% at k=40. Per-query exposure is ~k times the
#     per-document rate, so deeper retrieval multiplies contamination.
#
#
#      STALENESS OF A CACHED BLOCKLIST
#     refresh interval   mean staleness   missed at 40 new/week
#   ------------------------------------------------------------
#               hourly           0.02d                      0
#                daily           0.50d                      3
#               weekly           3.50d                     20
#              monthly          15.00d                     86
#   never (shipped once)         182.50d                   1043
#
#   asserts passed

The in_summary rate is the parameter to change first when adapting this. It is the fraction of real questions answerable from what you chose to index, and it is measurable on your own corpus in an afternoon — sample 50 queries, locate the answering passage, check whether it is in the index. Every other number here moves with it.

6. Real-world example

A team built RAG over a few thousand technical PDFs. Retrieval was mediocre and would not improve. Over two months they swapped embedding models twice, added a reranker, and tuned chunk size — each change producing a point or two.

The problem was in the first hour of the project. Their extractor handled single-column PDFs correctly and interleaved two-column ones line by line. Roughly 40% of the corpus was two-column. Those documents were in the index as fluent-looking nonsense: every line a fragment of one argument followed by a fragment of another.

Nothing had flagged it. The extractor returned text and exited zero. Chunk counts looked plausible. Embeddings were generated without error. The chunks even retrieved sometimes, because the vocabulary was right even though the sentences were meaningless — which is exactly the failure mode a keyword-based sanity check misses.

They found it when someone printed a retrieved chunk to debug a citation and could not parse the sentence.

Switching to a layout-aware extractor moved retrieval more than everything from the preceding two months combined. The lasting change was smaller and more useful: an ingestion report with characters-per-page, alphabetic ratio, and mean word length per document, with outliers held for review. The bug was never hard to fix — it was hard to see, because ingestion was the one stage with no instrumentation.

7. Interview questions companies actually ask

Q1 [easy] "Retrieval quality is poor. Where do you look first?"
  A At what's actually in the index, before touching embeddings or reranking.
    Sample ~50 real queries, find the passage that answers each, and check whether
    it's indexed. That fraction is a hard ceiling. Measured here: if only 37% of
    answers are in the indexed summary, recall@10 caps at 36.0% versus 93.4% for
    full text -- 57 points no downstream stage can recover.

Q2 [easy] "Why is PDF extraction risky?"
  A Because it fails silently. PDF is a rendering format: two-column layouts
    interleave line by line, tables flatten into word sequences, headers glue into
    body text, ligatures corrupt words, scans may have no text layer. You still get
    TEXT, so the pipeline reports success and nonsense chunks enter the index.

Q3 [medium] "How much does a 15% extraction error rate cost?"
  A About 14.4 points of recall -- and it's linear, so there's no safe threshold to
    sit below. recall(e) ≈ recall(0) * (1-e). Halving the error rate halves the loss.
    Estimate it by sampling 50 documents and counting bad extractions.

Q4 [medium] "Full text is 14x the storage. How do you avoid paying it?"
  A Two-stage: index summaries, retrieve against them, then fetch and score full
    text for the top candidates only. Measured: fetch top-20 reached 91.0% versus
    93.4% for a full-text index, at 1x storage. You move cost from storage to
    per-query latency, which is often the better place.

Q5 [medium] "What's the catch with the two-stage design?"
  A A document only enters the fetch set if its SUMMARY ranked it there. When the
    summary is a poor proxy for the body, the document never gets fetched --
    fetch top-10 only reached 62.2% while top-40 reached 97.2%. Fetch depth is a
    knob with diminishing returns; sweep it rather than guessing.

Q6 [medium] "How do you keep withdrawn or superseded sources out?"
  A A cached blocklist applied at index time, refreshed on a schedule, plus a live
    check on the final few documents you actually cite. The cache is a snapshot --
    at 40 new withdrawals a week, monthly refresh misses ~86 and a never-refreshed
    list misses ~1,000 after a year. Version supersession usually matters more than
    retraction in commercial corpora and is almost always missing.

Q7 [hard] "Why does retrieval depth make contamination worse?"
  A Because exposure is P(at least one bad doc in top-k) = 1-(1-p)^k ≈ k*p. A 0.5%
    contaminated corpus gives 4.9% of queries a bad source at k=10 and 18.2% at
    k=40. So every argument for retrieving more -- deeper reranking, graph
    expansion, multi-source fusion -- multiplies contamination. That argues for
    filtering at index time, where the cost is once per document rather than once
    per document per query.

Q8 [hard] "What should ingestion emit besides text?"
  A Structure, because extraction is the last moment it exists: section heading,
    page number, table-vs-prose flag, document version, publication date, source
    URL. Nearly free at ingest, impossible to reconstruct later, and it enables
    section-aware chunking, metadata filters, and page-level citations. Watch
    metadata size limits -- storing full chunk text there silently truncates.

8. When to use / tradeoffs

  ORDER OF OPERATIONS AT INGEST:
    1. measure the ceiling -- what fraction of answers are in what you index
    2. fix extraction before anything downstream (it's linear and invisible)
    3. instrument: chars/page, alpha ratio, mean word length, truncation rate
    4. decide granularity: full text, or summaries + on-demand fetch
    5. filter withdrawn / superseded at INDEX time, live-check the cited few
    6. capture structure as metadata while you still have it

Situation	Why it breaks	Use instead
Tuning embeddings before checking the index	Ceiling is set at ingest; 57 points was unreachable	Measure answer-presence first
Naive PDF text extraction	Columns interleave, tables flatten — silently	Layout-aware extraction + ingest metrics
No ingestion instrumentation	Failures report success; found months later by hand	chars/page, alpha ratio, outlier alerts
Summary-only index	Caps recall at 36.0% here	Full text, or summaries + fetch top-20
Two-stage with shallow fetch	Body-only answers never enter the pool — 62.2% at f=10	Sweep fetch depth; ~20 was the corner
No withdrawal filtering	0.5% contamination → 4.9% of queries at k=10	Cached blocklist + live check on cited docs
Blocklist refreshed never	~1,000 missed entries after a year	Schedule the refresh; measure staleness
Raising `k` for quality	Contamination scales with `k`	Filter at index time, not query time
Discarding structure at parse	Section, page, version unrecoverable later	Store as metadata (mind size limits)

Honest limits. This is a model of ingestion, not a corpus of real PDFs. "Extraction failure" is a single Bernoulli flag per document, whereas real damage is partial and structured — a table garbles while the surrounding prose is fine, so a document is often half-usable rather than lost, which makes the true curve gentler than the linear one in §4.2 but also much harder to detect. The in_summary = 0.35 figure is invented and is the single most load-bearing number here; measure yours. The 14× storage multiplier for full text is a plausible ratio for research papers and will be quite different for short web pages or long books. Part C is exact arithmetic rather than simulation, but its inputs — your contamination rate, your withdrawal publication rate — are things I do not know; the shape (≈ k*p) transfers, the numbers do not. The staleness table assumes withdrawals arrive at a constant rate, which is roughly true in aggregate and false for any individual source. Nothing here measures the cost of layout-aware extraction, which is real: good PDF parsing is often slower than embedding.

Ingestion sets a ceiling nothing downstream can raise. Summary-only indexing capped recall@10 at 36.0% against 93.4% for full text — a 57-point gap no reranker or prompt can recover.
Measure the ceiling first: what fraction of real questions are answerable from what you actually indexed?
Extraction fails silently and linearly. 15% parse failure = −14.4 points; 30% = −26.8. No safe threshold. Instrument chars/page, alpha ratio, word length.
Two-stage beats both extremes: summaries + fetch top-20 full text gave 91.0% at 1× storage instead of 14×. Sweep fetch depth — 10→20 gained 28.8 points, 20→40 gained 6.2.
Contamination scales with k: 1-(1-p)^k ≈ k*p. 0.5% bad documents → 4.9% of queries at k=10, 18.2% at k=40.
Filter at index time, live-check only the few you cite. Refresh the blocklist on a schedule — never-refreshed misses ~1,000 entries a year.
Capture structure while it exists — section, page, version, date. Unrecoverable afterwards.
Boundary: for small, clean, born-digital corpora most of this is unnecessary complexity. Corpus hygiene still applies wherever sources carry authority.

Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — the next step: how the extracted text gets split, and why section metadata makes that better
Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — what the 14× storage multiplier actually costs, and metadata size limits
Reranking: The Second Pass That Decides What the Model Sees — §4.1's ceiling decomposition, one stage later and an order of magnitude smaller than this one
Graph RAG: Walking the Link Graph When Retrieval Comes Back Thin — another way to reach documents the index missed, and another k that multiplies contamination
RAG Evaluation: Attributing Failure and Sizing the Eval Set — how to attribute a bad answer to ingestion rather than to retrieval or generation

Resources

Shen, Zhang, Dell et al., "LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis", ICDAR 2021 (arXiv:2103.15348) — layout-aware extraction, the fix for §3.1.
Smith, "An Overview of the Tesseract OCR Engine", ICDAR 2007 — the standard open OCR engine and its error characteristics.
Pfeiffer, Bressem, Adams et al., "Unstructured document parsing for RAG" — see the unstructured and docling open-source projects for current practical extractors. No single canonical paper; evaluate on your own corpus.
Steen, Yasseri et al. and the Retraction Watch Database — https://retractionwatch.com/retraction-watch-database-user-guide/ — the reference dataset for §3.4 in scholarly corpora.
Crossref REST API — https://api.crossref.org — live per-document status lookup, the "check the cited few" half of §3.4.

← Previous: Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For Next: Generation Pipeline: Assembling Context and Enforcing a Citation Contract →