Anatomy of a RAG Pipeline

ArchitectureSearchBeginner23 min

By: Anacodic Team

TL;DR — A RAG answer passes through six stages — ingest, chunk, embed, retrieve, augment, generate — and a wrong answer is usually caused by a stage the model never sees. In the worked example below, chunking strands the clause that decides the answer, retrieval returns a chunk scoring a perfect 1.00, and the generator produces the exact opposite of the truth while faithfully citing a real source. Debugging the prompt would find nothing. The negative result is the useful part: a score threshold does not rescue it, because the retrieved chunk is relevant — it is merely incomplete. Thresholds catch irrelevant retrieval, not truncated retrieval, and those need different fixes. So the discipline is to log what each stage produced and bisect, rather than tuning the last stage because it's the visible one.

1. Simple explanation

"RAG" names one behaviour — look it up, then answer — but it is built from six steps, each of which can quietly ruin the result.

Documents come in and get cleaned up. They get cut into pieces small enough to retrieve. Each piece becomes a vector. At question time you find the closest pieces, paste them into a prompt, and the model writes the answer.

What makes debugging hard is that only the last step is visible. When the answer is wrong, the answer is what you're looking at, so the prompt is what you reach for. But the answer is downstream of five other decisions, and the model can only work with what arrived. If the sentence containing the deciding fact was cut in half three stages earlier, no prompt in the world recovers it.

Analogy — a bad restaurant meal. The plate is what you see, so you blame the chef. But the dish depends on the supplier, what was in the delivery, how it was stored, how it was prepped, and only then the cooking. A tomato that arrived rotten cannot be rescued by seasoning. Diagnosing a RAG failure is walking that chain backwards: what did the cook actually receive? Almost always the answer is "not what you thought", and the fix is upstream of the stage you were staring at.

2. Diagram

THE SIX STAGES  —  offline once, online per question

  OFFLINE (when documents change)
  ┌─────────┐   ┌─────────┐   ┌─────────┐
  │1 INGEST │──▶│2 CHUNK  │──▶│3 EMBED  │──▶ [ vector index ]
  │ parse,  │   │ split   │   │ text →  │
  │ clean   │   │ into    │   │ vector  │
  └─────────┘   │ pieces  │   └─────────┘
                └─────────┘
                     ▲
              ⚠ the usual culprit

  ONLINE (per question)
                          ┌─────────┐   ┌─────────┐   ┌──────────┐
   question ─────────────▶│4 RETRIEVE│──▶│5 AUGMENT│──▶│6 GENERATE│──▶ answer
                          │ rank +  │   │ build   │   │ write it │
                          │ filter  │   │ prompt  │   │          │
                          └─────────┘   └─────────┘   └──────────┘
                                                            ▲
                                                  the only visible stage
                                                  — and rarely the cause


THE FAILURE, TRACED

  the rule spans TWO sentences:
    chunk 1: "Sale items may be returned ... only if the item is faulty."
    chunk 2: "Non-faulty sale items are final."
                                        ▲
                       the clause that DECIDES the answer

  A. retrieve k=1  ─▶ chunk 1 only, score 1.00 ─▶ "yes"   ❌ WRONG
  B. retrieve k=2  ─▶ chunks 1+2               ─▶ "no"    ✅ CORRECT
  C. k=1 + strict threshold 0.9 ─▶ chunk 1, score 1.00 still passes ─▶ "yes" ❌

     C is the point: a threshold cannot help. The chunk is
     RELEVANT (1.00). It is just INCOMPLETE.


TWO DIFFERENT FAILURES, TWO DIFFERENT FIXES

  irrelevant retrieval   low score   ─▶ fix with a THRESHOLD  (abstain)
  truncated retrieval    high score  ─▶ fix with CHUNKING or a bigger k

3. How it works

3.1 Stages 1–3: the offline half

Ingest turns source files into clean text: parsing PDFs, stripping navigation, keeping tables and headings intelligible. Boring, and the source of a surprising share of bad answers — a table that becomes a soup of numbers is unretrievable no matter what happens later. Document Processing: What You Index Sets the Ceiling is the depth.

Chunk splits text into retrievable pieces. This is the highest-leverage decision in the pipeline and the most common cause of confidently wrong answers, for the reason §3.3 sets out.

Embed converts each chunk to a vector and stores it. Mechanical, with one trap: change the embedding model and every stored vector is invalid, because vectors from different models aren't comparable. Re-embed everything.

These three run when documents change, not per question — so their cost is amortised, and their mistakes are baked in until you re-run them.

3.2 Stages 4–6: the online half

Retrieve embeds the question, ranks the chunks, and filters. Two decisions: how many (k) and how good is good enough (the score floor). Covered in Vector Search for Retrieval.

Augment builds the prompt: the question, the retrieved chunks, and the instruction restricting the model to them. Also where you decide chunk ordering, how to label sources so citations are possible, and what to do when nothing was retrieved.

Generate writes the answer. This is the only stage a user sees, which is precisely why it absorbs debugging attention it usually doesn't deserve.

3.3 Why chunking causes wrong answers

A chunk can be perfectly good text and still be an incomplete rule.

Real policies are written as a claim plus its qualifications: "you may do X — unless Y." Split those into separate chunks and each is individually true and individually misleading. Retrieve only the first and the model sees permission with no exception, so it answers "yes" — correctly, given what it was shown, and wrongly, given the document.

This is worse than a missing answer, because the output is fluent, definite, and comes with a citation to a real chunk that really does say what it says.

Three mitigations, in rough order of cost: overlap chunks so a split rule appears whole in at least one; retrieve neighbouring chunks alongside the best match (sometimes called sentence-window retrieval); or chunk on structure — sections and clauses — instead of a fixed size. Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems goes deeper.

3.4 The negative result: what a threshold cannot do

A minimum relevance score is the right fix for one failure and useless against another. The distinction is worth memorising:

failure	score looks like	fix
irrelevant — nothing in the corpus answers this	low (0.2)	threshold → abstain
truncated — the right passage, cut short	high (1.00)	chunking, overlap, bigger `k`
confusable — same topic, wrong specifics	high (0.95)	hybrid / exact matching
stale — right passage, obsolete	high	corpus hygiene, date filters

Only the first row is a threshold problem. In §4 case C, a threshold of 0.9 — aggressive — changes nothing, because the misleading chunk scores 1.00. Teams that add a threshold and consider retrieval "handled" have fixed one of four failures.

3.5 Debugging: log the stages, then bisect

The practice that makes RAG debuggable is boring and non-negotiable: record what each stage produced. For every question, store the retrieved chunk IDs and scores, the exact prompt sent, and the answer. Then a bad answer is a bisection instead of a guess:

  Was the deciding fact in the corpus at all?      no  → documentation problem
  Did chunking keep it intact?                     no  → chunking problem
  Did retrieval return that chunk?                 no  → retrieval problem (k, threshold, embedding)
  Was it in the prompt that was actually sent?     no  → augmentation problem
  All yes, and the answer still ignores it?        → now it is a generation problem

Most teams start at the bottom and work up, which is the expensive direction. Note also that these are separately measurable: retrieval quality can be scored without generating anything, which is the cheapest evaluation in the whole system — see RAG Evaluation: Attributing Failure and Sizing the Eval Set.

3.6 Where this six-stage model stops describing things

It assumes one retrieval pass per question. Systems that reason about what to fetch, search repeatedly, or refine a query mid-answer don't fit — that's Agentic RAG: Routing Retrieval to Specialists. It also assumes independent text chunks, so it says little about graph-structured knowledge or tables where the answer spans rows. And it assumes questions map to passages: a question needing aggregation across hundreds of documents ("how many contracts expire this quarter?") is a database query, and no amount of retrieval tuning will make it one.

4. The math

4.1 Failure probability compounds along the chain

Each stage must succeed for the answer to be right, and success is roughly conditional on the previous stage:

  P(correct) ≈ P(in corpus) × P(chunk intact) × P(retrieved) × P(in prompt) × P(model uses it)

  five stages at 95% each:   0.95^5 ≈ 0.774
  five stages at 90% each:   0.90^5 ≈ 0.590

That multiplication is the argument against tuning one stage hard. Lifting generation from 0.95 to 0.98 buys ~2 percentage points; a chunking stage at 0.80 is capping the whole system at 0.80 no matter what follows.

4.2 Where the attention should go

  marginal value of fixing a stage ∝ (1 - P(stage)) × everything downstream

  → fix the WORST stage, not the LAST one

4.3 Worked example

One source document, one question, three retrieval settings. The document is two sentences:

   chunk 1: Sale and clearance items may be returned unused within 30 days only if the item is faulty.
   chunk 2: Non-faulty sale items are final.

Question: "Can I return a sale item?" The truth is no — the default case is a non-faulty item, and chunk 2 decides it.

Case A — retrieve the single best chunk:

   retrieve 1.00 [USED] Sale and clearance items may be returned unused within 30 days only if the item is faulty.
   answer -> 'yes'   (WRONG, truth='no')

A perfect 1.00 similarity, a real citation, and the opposite of the truth. The model behaved correctly on the evidence it was given.

Case B — retrieve the neighbour as well:

   retrieve 1.00 [USED] Sale and clearance items may be returned unused ... only if the item is faulty.
   retrieve 0.67 [USED] Non-faulty sale items are final.
   answer -> 'no'   (CORRECT, truth='no')

Note that the chunk which decides the answer scored lower (0.67) than the one that misleads. Similarity is not importance.

Case C — add a strict score threshold of 0.9:

   retrieve 1.00 [USED] Sale and clearance items may be returned unused ... only if the item is faulty.
   answer -> 'yes'   (WRONG, truth='no')

The threshold changes nothing, because the misleading chunk scores 1.00. This is the result to remember: a score floor is not a general safety net. It catches irrelevance; it is blind to incompleteness.

Same question, same documents, same generator across all three. Only stage 4 changed.

5. Real code

"""A RAG pipeline, and locating the stage that actually caused a bad answer."""

SOURCE = (
    "Sale and clearance items may be returned unused within 30 days "
    "only if the item is faulty. "
    "Non-faulty sale items are final."
)
QUESTION = "Can I return a sale item?"
GROUND_TRUTH = "no"          # the default case is a non-faulty item


# ---- stage 1: chunk ------------------------------------------------------
def chunk(text: str) -> list[str]:
    """One sentence per chunk -- a common default. Each chunk is self-contained as
    TEXT, but not necessarily self-contained as a RULE. That gap is the bug below."""
    return [s.strip() + "." for s in text.split(".") if s.strip()]


# ---- stage 2: embed (stand-in) ------------------------------------------
STOP = {"can", "i", "a", "the", "are", "is", "to", "may", "be", "and",
        "only", "if", "within"}


def keywords(s: str) -> set[str]:
    """Stem to 4 letters so return/returned/returns unify. A crude stand-in for
    embedding similarity, which is what really does stage 2."""
    out = set()
    for w in s.replace("?", " ").replace(".", " ").replace(",", " ").lower().split():
        if len(w) > 2 and w not in STOP:
            out.add(w[:4])
    return out


# ---- stage 3: retrieve --------------------------------------------------
def retrieve(chunks: list[str], q: str, k: int, threshold: float):
    qk = keywords(q)
    scored = sorted(
        ((len(qk & keywords(c)) / max(1, len(qk)), c) for c in chunks),
        reverse=True, key=lambda t: t[0],
    )
    return [(s, c) for s, c in scored[:k] if s >= threshold], scored[:k]


# ---- stage 4: generate (stand-in) --------------------------------------
def generate(context: list[str]) -> str:
    """Answers ONLY from the supplied context, and never from prior knowledge."""
    joined = " ".join(context).lower()
    if "final" in joined:
        return "no"            # the deciding exception is present
    if "may be returned" in joined:
        return "yes"           # the trap: a true-but-partial rule
    return "unknown"


def run(k: int, threshold: float = 0.2, label: str = "") -> str:
    kept, top = retrieve(chunk(SOURCE), QUESTION, k, threshold)
    ans = generate([c for _s, c in kept])
    verdict = "CORRECT" if ans == GROUND_TRUTH else "WRONG"
    print(f"\n{label}   (k={k}, threshold={threshold})")
    for s, c in top:
        mark = "USED" if (s, c) in kept else "drop"
        print(f"   retrieve {s:.2f} [{mark}] {c}")
    print(f"   answer -> {ans!r}   ({verdict}, truth={GROUND_TRUTH!r})")
    return ans


print("The rule spans TWO sentences: a permission, then the exception that")
print("actually decides the answer. Chunk 1 alone is true but incomplete.\n")
for i, c in enumerate(chunk(SOURCE), 1):
    print(f"   chunk {i}: {c}")

bad = run(1, label="A. retrieve the single best chunk")
good = run(2, label="B. retrieve the neighbour as well")
strict = run(1, threshold=0.9, label="C. best chunk + a STRICT score threshold")

print("\n" + "=" * 72)
print("Same question, same documents, same generator. Only STAGE 3 changed.\n")
print("A answered the OPPOSITE of the truth -- confidently, citing a real chunk")
print("  that scored a PERFECT 1.00. Debugging the prompt would find nothing:")
print("  the deciding sentence never reached the generator. The bug is upstream.\n")
print("C is the important negative result. A score threshold does NOT rescue A,")
print("  because the retrieved chunk IS relevant -- it is merely INCOMPLETE.")
print("  Thresholds catch irrelevant retrieval, not truncated retrieval.")
print("  Two different failures, needing two different fixes.\n")
print("B is the fix here: pull in the neighbour so the stranded clause comes along.")

assert bad == "yes" and bad != GROUND_TRUTH, bad
assert good == GROUND_TRUTH, good
assert strict == bad == "yes", strict          # the threshold changes nothing
print("\nall assertions passed")

# Output:
#   The rule spans TWO sentences: a permission, then the exception that
#   actually decides the answer. Chunk 1 alone is true but incomplete.
#
#      chunk 1: Sale and clearance items may be returned unused within 30 days only if the item is faulty.
#      chunk 2: Non-faulty sale items are final.
#
#   A. retrieve the single best chunk   (k=1, threshold=0.2)
#      retrieve 1.00 [USED] Sale and clearance items may be returned unused within 30 days only if the item is faulty.
#      answer -> 'yes'   (WRONG, truth='no')
#
#   B. retrieve the neighbour as well   (k=2, threshold=0.2)
#      retrieve 1.00 [USED] Sale and clearance items may be returned unused within 30 days only if the item is faulty.
#      retrieve 0.67 [USED] Non-faulty sale items are final.
#      answer -> 'no'   (CORRECT, truth='no')
#
#   C. best chunk + a STRICT score threshold   (k=1, threshold=0.9)
#      retrieve 1.00 [USED] Sale and clearance items may be returned unused within 30 days only if the item is faulty.
#      answer -> 'yes'   (WRONG, truth='no')
#
#   ========================================================================
#   Same question, same documents, same generator. Only STAGE 3 changed.
#
#   A answered the OPPOSITE of the truth -- confidently, citing a real chunk
#     that scored a PERFECT 1.00. Debugging the prompt would find nothing:
#     the deciding sentence never reached the generator. The bug is upstream.
#
#   C is the important negative result. A score threshold does NOT rescue A,
#     because the retrieved chunk IS relevant -- it is merely INCOMPLETE.
#     Thresholds catch irrelevant retrieval, not truncated retrieval.
#     Two different failures, needing two different fixes.
#
#   B is the fix here: pull in the neighbour so the stranded clause comes along.
#
#   all assertions passed

The generator is a stand-in with two hard-coded rules, so the experiment isolates stage 4 — swap in a real model and the outcome is the same, because the deciding sentence still never arrives in case A.

6. Real-world example

A team shipped RAG over an internal HR handbook. Accuracy was acceptable, but a category of questions kept coming back wrong: anything involving an eligibility condition. "Can I carry over leave?" → yes, when the real answer was "only in your first year."

They spent two weeks on the prompt. They added "consider all conditions and exceptions", then a chain-of-thought instruction, then a bigger model. The wrong answers moved around but didn't stop, which should have been the clue: if a bigger model doesn't fix it, the model probably isn't the problem.

What finally identified it was logging the retrieved chunks alongside each answer. The handbook was written as a general statement followed by a separate paragraph of exceptions, and the chunker — 500-character windows — reliably put those in different chunks. With k=3, the three most similar chunks were all general statements on the topic, because those are phrased like the question. The exception paragraphs were phrased differently and scored lower, so they never made the cut.

Two properties made this hard to see. The retrieved chunks always looked right — they were on-topic, well-scored, genuinely about the question. And there was no missing-information signal anywhere: nothing scored low, nothing was empty, no threshold tripped.

The fix was chunking on the document's own structure so a rule and its exceptions stayed together, plus retrieving neighbours. Accuracy on that question category went from roughly half to nearly all. The prompt was reverted to its original form — none of the two weeks of prompt work had contributed anything, because it was addressing the wrong stage.

7. Interview questions companies actually ask

Q1. Walk me through the stages of a RAG pipeline. Offline: ingest (parse and clean documents), chunk (split into retrievable pieces), embed (vectorise and index). Online: retrieve (rank and filter by similarity), augment (build the prompt with the question, the chunks, and an instruction restricting the model to them), generate (write the answer). The offline half runs when documents change; the online half runs per question.

Q2. A RAG system gives a confidently wrong answer. How do you debug it? Bisect the stages from the top, not the bottom. Was the fact in the corpus? Did chunking keep it intact? Did retrieval return that chunk? Was it in the prompt actually sent? Only if all four are yes is it a generation problem. The instinct is to start with the prompt because the answer is what you can see, and that is the expensive direction.

Q3. Which stage causes the most bugs? Chunking, and it isn't close. A chunk can be perfectly valid text and an incomplete rule — policies are written as a claim plus its exceptions, and splitting those produces pieces that are individually true and individually misleading. That's worse than missing information, because the answer is fluent, definite, and cites a real chunk.

Q4. Does a minimum relevance score protect you? Against one failure of four. It catches irrelevant retrieval, where nothing scores well. It is blind to truncated retrieval (the right passage cut short — high score), confusable retrieval (same topic, wrong specifics — high score), and stale retrieval (right passage, obsolete — high score). In the worked example a threshold of 0.9 changes nothing, because the misleading chunk scores 1.00.

Q5. Why not just make chunks very large? Because it moves the problem rather than removing it. Large chunks cost more input tokens per question, dilute the prompt with irrelevant surrounding text, and blur retrieval — a chunk covering five topics is similar to queries about all five and precisely relevant to none. The usual resolution is moderate chunks with overlap, or chunking on document structure, plus retrieving neighbours.

Q6. Where should you spend effort if you can only fix one stage? The worst one, not the last one. Because the stages multiply, five stages at 95% gives about 77% end to end, and a single stage at 80% caps the whole system at 80% regardless of what follows. Measure each stage separately — retrieval quality can be scored without generating anything at all, which makes it the cheapest measurement available.

Q7. How do you know retrieval is the problem and not the model? Score retrieval on its own: for a set of questions with known answers, record whether the chunk containing the answer was in what came back. That number is independent of the model entirely. A useful shortcut in the meantime — if swapping in a bigger model doesn't help, the model was probably not the constraint.

8. When to use / tradeoffs

This six-stage model fits when:

One retrieval pass per question is enough
Knowledge is text that divides sensibly into independent passages
Questions map onto passages rather than requiring aggregation
You can instrument each stage separately

Reach for something else when:

The system needs to decide what to fetch, or fetch repeatedly → agentic retrieval
Knowledge is relational, so the answer is a path → graph retrieval
The question is an aggregation → a database query
Sources are tables or images → multimodal ingestion

Situation	Why it breaks	Use instead
Rules with exceptions in separate chunks	Each chunk true, incomplete, misleading	Structure-aware chunking + neighbours
Debugging starts at the prompt	Four stages upstream unexamined	Log every stage, then bisect
Threshold added, retrieval "handled"	Only fixes 1 of 4 failure modes	Also fix chunking; add hybrid; date documents
Chunks made huge to be safe	Token cost, dilution, blurred retrieval	Moderate chunks with overlap
Embedding model swapped in place	Old vectors are incomparable	Re-embed the whole corpus
"How many X are there?"	Needs aggregation, not retrieval	SQL or a metadata query
No per-stage measurement	Cannot tell which stage is worst	Score retrieval independently first

Honest limits. The pipeline in §5 is a stand-in end to end: the embedder is word-stem overlap, the generator is two hard-coded rules. That isolates the mechanism cleanly and it also means the numbers carry no information about real system accuracy — the shape of the failure transfers, the magnitudes do not. The compounding model in §4.1 assumes stage successes are independent, which they are not: bad chunking correlates with bad retrieval because both depend on the same text, so real systems fail together more than the multiplication suggests. The article also presents six tidy stages, and production systems blur them — query rewriting sits between 3 and 4, reranking between 4 and 5, and caching cuts across everything. Finally, "fix the worst stage" assumes you can measure each one, and most teams cannot when they need to; building that instrumentation is usually the actual first task rather than a prerequisite you already have.

Six stages: ingest → chunk → embed (offline), retrieve → augment → generate (per question).
A wrong answer is usually caused upstream of the model. Only generation is visible, which is why it absorbs debugging effort it rarely deserves.
Chunking is the biggest source of confidently wrong answers. A chunk can be valid text and an incomplete rule — a permission separated from its exception.
The measured failure: chunk 1 retrieved at a perfect 1.00, answer was the exact opposite of the truth, with a real citation.
A score threshold does not fix this. It catches irrelevant retrieval; it's blind to truncated, confusable, and stale retrieval — all of which score high.
The chunk that decided the answer scored lower (0.67) than the one that misled. Similarity is not importance.
Stages multiply: 5 × 95% ≈ 77%. Fix the worst stage, not the last one.
Log what every stage produced, then bisect. Retrieval can be scored without generating anything — the cheapest evaluation you have.
The model doesn't fit agentic, graph, or aggregation-shaped problems, and real pipelines blur the stage boundaries.

What RAG Is and When to Use It — why retrieve at all, and when not to
Vector Search for Retrieval — stage 4 in detail, including the threshold this article limits
Document Processing: What You Index Sets the Ceiling — stage 1, and why bad ingest is unrecoverable later
Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — stage 2, the highest-leverage decision
Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — stage 3, storage and indexing
Generation Pipeline: Assembling Context and Enforcing a Citation Contract — stages 5 and 6 done properly
RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring stages separately, which §3.5 depends on
RAG Monitoring: Your Error Rate Will Not Tell You Anything — the per-stage logging this article insists on
Reranking: The Second Pass That Decides What the Model Sees — the extra stage between retrieve and augment
Agentic RAG: Routing Retrieval to Specialists — when one pass isn't enough
Graph RAG: Walking the Link Graph When Retrieval Comes Back Thin — when the answer is a relationship, not a passage

Resources

Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 — the original architecture: https://arxiv.org/abs/2005.11401
Gao et al. (2023) — Retrieval-Augmented Generation for Large Language Models: A Survey, arXiv:2312.10997 — a stage-by-stage map of the variants: https://arxiv.org/abs/2312.10997
Liu et al. (2023) — Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172 — evidence that chunk ordering in the prompt changes whether the model uses it: https://arxiv.org/abs/2307.03172
Es et al. (2023) — RAGAS: Automated Evaluation of Retrieval Augmented Generation, arXiv:2309.15217 — metrics that separate retrieval quality from answer quality: https://arxiv.org/abs/2309.15217
Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 8 (evaluation) — free online; precision and recall for retrieval predate LLMs and still apply: https://nlp.stanford.edu/IR-book/

Runnable notebook

Run it end to end — the mock model needs no API key; add your own key for the real Claude section.

← Previous: Vector Search for Retrieval