Tables in RAG: Why Flattening Breaks the Answer, Not the Search

TL;DR — A table that loses its grid usually keeps retrieving perfectly well. Bag-of-words and dense retrievers score a flattened table and an intact one almost identically, because the tokens are the same; what is destroyed is the binding between a value and the row and column that give it meaning. So the failure surfaces one stage later, at generation, as a confident answer with the wrong number attached. The obvious fix — chunk one row at a time — repairs binding but can shift the failure back into retrieval, because a row label rarely matches how a person phrases a question. Neither problem is visible in a similarity score, which is why table handling has to be evaluated on answers, not on chunks.

---

1. Simple explanation

A table means something because of where things sit. 8.90 is only an answer if you know it is in the 1–3 kg row and the Zone 2 column. Take the grid away and you still have the number 8.90 sitting in a sentence full of other numbers — and nothing that says which pairing it belongs to.

Retrieval does not notice this. A retriever matches words, or the direction of a vector built from words. "Zone 2", "1-3 kg" and "8.90" are all still present in the flattened text, so the chunk still looks like an excellent match for a question about zone 2 rates. It gets retrieved. It goes to the model. And the model now has to guess which of the nine numbers in front of it is the one asked for.

Analogy — a seating chart torn into a list of names. A wedding seating chart tells you Priya is at table 4 and Marcus is at table 9. Tear off the table headings and you are left with a list: Priya, Marcus, Anand, table 4, table 9, table 2. Every name and every table number survived. Not one pairing did. If someone now asks "where is Marcus sitting?", the list looks highly relevant — it contains his name and table numbers — and it cannot answer the question. Worse, it will confidently suggest a table, because a plausible-looking pairing is easy to assemble from the parts.

That is exactly what flattening does to a table, and exactly why the failure hides: the retrieved text looks right.

---

2. Diagram

  THE TABLE AS PUBLISHED            WHAT EACH REPRESENTATION KEEPS
  ------------------------          -------------------------------

  | band     | Zone 1 | Zone 2 |
  |----------|--------|--------|      INTACT (grid preserved)
  | <=1 kg   |  4.20  |  5.80  |        position carries meaning
  | 1-3 kg   |  6.50  |  8.90  |        (1-3 kg, Zone 2) -> 8.90   OK
  | 3-5 kg   |  9.10  | 12.40  |

         |                               FLATTENED (text only)
         |  parser drops the grid          "band Zone 1 Zone 2 <=1 kg
         v                                  4.20 5.80 1-3 kg 6.50 8.90"
                                            every value survived
                                            every pairing did not     BROKEN
    tokens are IDENTICAL
    -> retrieval score is
       IDENTICAL                       ROW CHUNKS (label repeated)
    -> the loss is invisible             "band: 1-3 kg; Zone 1: 6.50;
       until generation                   Zone 2: 8.90"
                                          pairing restored ...
                                          ... but now the RIGHT ROW has
                                              to be retrieved first    RISK


  WHERE THE FAILURE LANDS

    representation   retrieval      binding      net result
    --------------   ---------      -------      ----------
    intact           works          works        answer correct
    flattened        works          BROKEN       confident wrong number
    row chunks       AT RISK        works        right answer, or no answer

---

3. How it works

3.1 Two separate properties, routinely confused

A table in a retrieval pipeline has to satisfy two independent conditions, and almost all discussion of "table extraction quality" collapses them into one:

These fail independently. A table can be perfectly structured and never retrieved. A table can be retrieved every time and be unusable. Structure-recovery metrics measure binding only. Retrieval metrics measure findability only. A system needs both and is usually evaluated on neither.

3.2 Why flattening is invisible to a retriever

Both sparse and dense retrieval are, at the level that matters here, functions of the multiset of tokens in a chunk.

BM25 scores a document from term frequencies and document length. Flattening a table changes neither the terms nor, materially, the length — it removes pipe characters and newlines. The score barely moves.

Dense embedding is less obvious but lands in the same place. A sentence encoder is not indifferent to word order, but a table has no syntax for order to carry: | 6.50 | 8.90 | and 6.50 8.90 produce near-identical representations because there was no grammatical relationship to preserve in the first place. The grid was doing the work that syntax does in prose, and the encoder has no channel for it.

The consequence is worth stating plainly: you cannot detect flattening by looking at retrieval scores. Not by inspecting them, not by thresholding them, not by monitoring them in production. The signal is not there.

3.3 Why the obvious fix moves the problem

If the grid is what binds, restore the binding explicitly — emit one chunk per row with the header repeated:

Shipping rate. band: 1-3 kg; Zone 1: 6.50; Zone 2: 8.90; Zone 3: 14.20

Binding is now unambiguous without any grid. But three things changed at once:

1. One table became N chunks, so the right row must now win retrieval, not merely the right table.
2. Each chunk is short, so a single mismatched token moves the score a long way.
3. The row label is written the way the document phrases it (1-3 kg), while the question is phrased the way a person does (a two kilo parcel, 1.8 kg). Nothing in a lexical or embedding match reliably bridges that.

The measured run in §5 shows this precisely: row-chunking scored lower than either single-chunk representation and retrieved the wrong row. The naive fix traded a binding failure for a findability failure, and a pipeline that only watched binding would have recorded it as an improvement.

3.4 Repair, in the order the stages run

Where this stops applying. All of the above assumes the table has a header row and a consistent column count. Merged cells, multi-level headers, rotated column labels, and tables whose first row is a spanning title defeat every representation in this article, including intact markdown — markdown has no syntax for a cell spanning two columns. For those, the honest move is to keep the table as an image alongside its text and let a vision model read it on demand.

---

4. The math

4.1 Findability and binding as separate rates

Let Q be a question set where every answer is one cell of a table. For a pipeline configuration c:

  find(c)  = |{q : a chunk containing q's cell is retrieved}| / |Q|
  bind(c)  = |{q : (row, col) -> value recoverable from that chunk}| / |Q_found|
  answer(c) = find(c) * bind(c)

answer(c) is the only number a user experiences. Reporting find alone flatters a flattened pipeline; reporting bind alone flatters row-chunking.

4.2 Why cosine cannot see the grid

For a bag-of-words embedding E, where t(d) is the token multiset of document d:

  t(intact)     = {band, Zone, 1, Zone, 2, <=1, kg, 4.20, 5.80, ...}
  t(flattened)  = {band, Zone, 1, Zone, 2, <=1, kg, 4.20, 5.80, ...}

  t(intact) == t(flattened)   (pipes and newlines are not tokens)
  => E(intact) == E(flattened)
  => cos(E(q), E(intact)) == cos(E(q), E(flattened))   for every q

The equality is exact for a hashing bag-of-words encoder and approximate for a trained sentence encoder. Either way, no threshold on similarity separates the two.

4.3 Worked example

Take the three-row shipping table, the question "what is the rate for a 1-3 kg parcel to zone 2", and the true answer 8.90. Using a 256-dimensional hashing encoder with sign bits and L2 normalisation, the measured run in §5 gives:

  representation   chunks   top cosine   retrieved        binds?
  intact                1      +0.544    whole table      True
  flattened             1      +0.544    whole blob       False
  row chunks            3      +0.499    row "<=1 kg"     False

Read the first two rows together: the cosine is identical to three decimal places, and one of them is unusable. Then read the third: splitting into rows dropped the score by 0.045 and selected the wrong row, so binding never got a chance to help.

Applying §4.1 to this single question:

  intact       find = 1,  bind = 1  ->  answer = 1
  flattened    find = 1,  bind = 0  ->  answer = 0
  row chunks   find = 0,  bind = -  ->  answer = 0

Two of three representations score a perfect or near-perfect retrieval number and answer the question incorrectly or not at all.

---

5. Real code

"""One table, three representations. What each does to retrieval, and to binding."""
import hashlib, math, re

TABLE_MD = """| Weight band | Zone 1 | Zone 2 | Zone 3 |
|---|---|---|---|
| up to 1 kg | 4.20 | 5.80 | 9.40 |
| 1-3 kg | 6.50 | 8.90 | 14.20 |
| 3-5 kg | 9.10 | 12.40 | 19.80 |"""

def as_intact(md):
    return [md]

def as_flattened(md):
    cells = [c.strip() for c in re.split(r"[|\n]", md)
             if c.strip() and set(c.strip()) != {"-"}]
    return [" ".join(cells)]

def as_row_chunks(md):
    lines = [l for l in md.splitlines() if l.startswith("|")]
    header = [c.strip() for c in lines[0].strip("|").split("|")]
    return ["Shipping rate. " + "; ".join(f"{h}: {v}" for h, v in
            zip(header, [c.strip() for c in l.strip("|").split("|")]))
            for l in lines[2:]]

def embed(text, dim=256):
    v = [0.0] * dim
    for tok in re.findall(r"[a-z0-9.]+", text.lower()):
        h = int(hashlib.sha256(tok.encode()).hexdigest(), 16)
        v[h % dim] += 1.0 if (h >> 8) & 1 else -1.0
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

def retrieve(query, chunks):
    q = embed(query)
    return max(((sum(a*b for a,b in zip(q, embed(c))), c) for c in chunks),
               key=lambda t: t[0])

def binds(chunk, row="1-3 kg", col="Zone 2", val="8.90"):
    """Can (row, col) -> val be recovered from this chunk WITHOUT guessing?"""
    if chunk.lstrip().startswith("|"):                 # a grid: position carries meaning
        for line in chunk.splitlines():
            if line.startswith("|") and row in line:
                cells = [c.strip() for c in line.strip("|").split("|")]
                hdr = [c.strip() for c in chunk.splitlines()[0].strip("|").split("|")]
                return dict(zip(hdr, cells)).get(col) == val
        return False
    if f"{col}: {val}" in chunk:                       # explicit label: pairing survives
        return row in chunk
    return False                                       # bare text: order is all you have

QUERY = "what is the rate for a 1-3 kg parcel to zone 2"

print(f"{'representation':13} {'chunks':>6} {'top_score':>10}  {'retrieved':<22} binds?")
for name, fn in [("intact", as_intact), ("flattened", as_flattened),
                 ("row-chunks", as_row_chunks)]:
    chunks = fn(TABLE_MD)
    score, top = retrieve(QUERY, chunks)
    label = ("whole table" if len(chunks) == 1 and top.lstrip().startswith("|")
             else "whole blob" if len(chunks) == 1
             else top.split(";")[0].replace("Shipping rate. Weight band: ", "row "))
    print(f"{name:13} {len(chunks):>6} {score:>+10.3f}  {label:<22} {binds(top)}")

print()
print("flattened, verbatim  ->", as_flattened(TABLE_MD)[0][:96], "...")
print("row-chunk retrieved  ->", retrieve(QUERY, as_row_chunks(TABLE_MD))[1])

assert binds(as_intact(TABLE_MD)[0]) is True
assert binds(as_flattened(TABLE_MD)[0]) is False

# Output:
#   representation chunks  top_score  retrieved              binds?
#   intact             1     +0.544  whole table            True
#   flattened          1     +0.544  whole blob             False
#   row-chunks         3     +0.499  row up to 1 kg         False
#
#   flattened, verbatim  -> Weight band Zone 1 Zone 2 Zone 3 up to 1 kg 4.20 5.80 9.40 1-3 kg 6.50 8.90 14.20 3-5 kg 9.10 12 ...
#   row-chunk retrieved  -> Shipping rate. Weight band: up to 1 kg; Zone 1: 4.20; Zone 2: 5.80; Zone 3: 9.40

The two asserts are the article's claim in executable form: the grid binds, the blob does not. The identical +0.544 is not a coincidence to be tuned away — it is the arithmetic of §4.2.

---

6. Real-world example

A support-automation team shipped a retrieval assistant over a few hundred policy and pricing documents. Retrieval looked healthy: for shipping questions, the correct document was in the top three results more than nine times out of ten. They tracked that number weekly and it never moved.

Complaints arrived anyway, and they were oddly specific — the assistant quoted real prices that were wrong by one row. A 4 kg parcel was quoted the 1–3 kg rate. A zone 3 delivery was quoted the zone 2 price. Every quoted figure existed somewhere in the rate card, which is why nothing looked hallucinated and why spot-checking the retrieved documents found no problem: the right document really was being retrieved.

The cause was an ingestion change made four months earlier for an unrelated reason — a switch to a faster text extractor for throughput. The new extractor returned the rate cards as plain text. No error, no warning, no change in retrieval metrics, and a 100% rate of broken bindings on every table in the corpus. The regression was invisible to every dashboard they had, because they were all watching findability.

Two details are worth taking from this. First, the defect survived four months precisely because retrieval quality was monitored and stayed green. Second, the fix was not a better parser — it was adding one table-dependent question to the smoke suite, whose answer could only be produced by reading a specific cell. That test would have failed on day one.

---

7. Interview questions companies actually ask

Q1. Your RAG system retrieves the right document but returns a wrong number from it. Where do you look first? At the representation of the source, not the retriever. A wrong-but-plausible number that exists in the document is the signature of a broken binding: the value survived ingestion while the structure that gave it meaning did not. Check what the parser actually emitted for that region — if a table came through as running text, the model was asked to re-derive a grid from a word sequence, and it will do so confidently and sometimes wrongly. Retrieval metrics will look perfect throughout, because the tokens never changed.

Q2. Why can't you detect table flattening by monitoring retrieval scores? Because sparse and dense retrieval are both essentially functions of a chunk's token multiset, and flattening removes only delimiters. BM25 sees the same terms with the same frequencies; a sentence encoder has no channel for grid position, since the grid was substituting for syntax that was never there. The similarity is identical or near-identical, so no threshold separates the two cases. Detection requires an evaluation whose unit is an answer, not a chunk.

Q3. You chunk one table row per chunk with the header repeated. What breaks? Findability. You have replaced one retrieval target with N short ones, so the right row must now win, and short chunks are volatile — one mismatched token moves the score substantially. Worse, row labels are phrased as the document writes them (1-3 kg) while questions are phrased as people ask them (a two kilo parcel), and neither lexical nor embedding matching reliably bridges that gap. The measured example in this article scored lower after row-chunking and retrieved the wrong row.

Q4. What is the minimum change that makes table handling measurable? Add questions whose answers exist only in a table cell and appear nowhere in the surrounding prose, then score answered-versus-not. The "only in a table" check is the part that makes the test valid — if the value also appears in a sentence, the question can be answered without any table handling at all and measures nothing. A handful of such questions in a smoke suite catches the entire class of failure.

Q5. How do you decide between structure-aware markdown, row chunks, and keeping the table as an image? By the table's shape and the question's shape. A clean rectangular table with a single header row is well served by preserved markdown and needs nothing more. Wide tables that would blow a chunk budget, or corpora where questions name a row explicitly, favour row chunks with the header repeated. Merged cells, multi-level headers, and spanning titles defeat both, because markdown cannot express them — there, keep the image and let a vision model read it when a question touches that region.

Q6. Your table chunk is well-formed but never retrieved. What do you change? The text you embed, not the text you store. Give the chunk a separate embed_text containing the caption, the header terms, and a short natural-language gloss of what the table reports, so it matches how a person would ask. Keep the grid in the text the model is shown. Separating the retrieval surface from the display surface is the single highest-leverage change available at the indexing stage.

Q7. Is a structure-recovery metric like a tree-edit-distance score enough to certify table quality? No, and it is important to know why. Those metrics measure binding fidelity against a ground-truth grid, which is exactly the thing retrieval scores miss — they are genuinely useful and complementary. What they do not measure is findability: a perfectly reconstructed table that no question ever retrieves contributes nothing, and a structure score cannot tell you that. The two metrics cover different halves, and neither is a substitute for scoring answers.

---

8. When to use / tradeoffs

Reach for preserved structured output when:

• the corpus contains comparison tables, rate cards, or per-variant outcome tables
• questions ask for a specific value at the intersection of two labels
• the table is rectangular with one header row
• you can keep the structured form through chunking without splitting it

Reach for row chunks with a repeated header when:

• tables are wide or long enough to exceed a sensible chunk budget
• questions tend to name a row explicitly and in the document's own vocabulary
• you are willing to measure findability afterwards rather than assume it

Honest limits. The identical +0.544 in §5 comes from a hashing bag-of-words encoder, where the token-multiset argument of §4.2 is exact. A trained sentence encoder is not perfectly order-blind, so a real system will show a small difference between intact and flattened text — small enough to sit well inside normal score variance, which is the point, but not exactly zero. The binds() function in §5 is a deliberately strict proxy for what a reader can recover, and it treats an explicit label: value pairing as sufficient; a long row with many similar labels can still mislead a model even when that check passes. The failure taxonomy here assumes a table either keeps its grid or loses it, while real parser output often lands in between — a header that survives while one column merges. And the whole argument is about tabular content; documents whose evidence lives in figures fail the same way with none of the same repairs available, because there is no grid to preserve in the first place.

---

9. Summary + related articles

• A table has to be both findable and bindable; these fail independently and most pipelines measure neither.
• Flattening a table does not degrade retrieval scores, because retrieval is a function of tokens and flattening removes only delimiters. The loss appears at generation as a confident wrong value.
• Therefore no similarity threshold, dashboard, or monitor built on retrieval quality can detect this class of failure.
• Row chunks with a repeated header restore binding and put findability at risk — the measured example scored lower and retrieved the wrong row.
• Repair at the stage that actually failed: preserve structure through chunking, give table chunks their own embedding text, and keep the grid in the prompt.
• The boundary: none of this survives merged cells or multi-level headers, where markdown cannot express the structure and an image plus a vision model is the honest answer.
• The minimum viable test is a question whose answer exists only in a cell — verified absent from the prose — scored on whether it was answered.

Related:

• Document Processing: What You Index Sets the Ceiling — what the extractor keeps and discards before any of this applies
• Chunking Strategies — the general strategies this article specialises for tabular content
• Token-Aware Chunking and Reproducible Index Builds — why a table must not be split by a word budget
• Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — where the separate embedding text for a chunk is configured
• RAG Evaluation: Attributing Failure and Sizing the Eval Set — building the answer-level eval set this article argues for
• Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer — what to do when the cell genuinely is not there
• Reranking: The Second Pass That Decides What the Model Sees — the second pass that can rescue a row chunk that lost on first retrieval

Resources

• Zhong, Xu & ShafieiBavani, Elaheh & Jimeno Yepes, Antonio. Image-based Table Recognition: Data, Model, and Evaluation. ECCV 2020. Introduces the PubTabNet dataset and the TEDS (Tree-Edit-Distance-based Similarity) metric for table structure recovery. Dataset: https://github.com/ibm-aur-nlp/PubTabNet
• Smock, Brandon & Pesala, Rohith & Abraham, Robin. PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents. CVPR 2022. Table detection and structure recognition at scale, and the GriTS metric. Code and data: https://github.com/microsoft/table-transformer
• ICDAR Competition on Table Detection and Recognition (cTDaR) — the long-running benchmark series for table detection and structure recognition: https://github.com/cndplab-founder/ICDAR2019_cTDaR
• Docling (IBM) — document conversion with table structure recognition and a chunker that can repeat a table header across a split: https://github.com/docling-project/docling
• Robertson, Stephen & Zaragoza, Hugo. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 2009. The term-frequency formulation behind the claim in §3.2 that delimiters do not affect the score.