What RAG Is and When to Use It

TL;DR — Retrieval-Augmented Generation looks up relevant text at question time and hands it to the model, instead of relying on what the model absorbed during training. That buys three things a trained-in fact cannot give you: it updates the instant you edit a document, it can cite which document it used, and it can say "I don't know" when nothing relevant exists. In the worked example below, changing a policy makes the memorised answer silently stale while the retrieved answer is correct and traceable — at the cost of one document edit rather than a retraining run. It stops helping the moment the answer isn't written down anywhere, because retrieval can only find what you already have; and it makes abstention possible, not automatic — a system that isn't allowed to say "I don't know" will still make something up.

---

1. Simple explanation

A language model learns from a huge pile of text, and what it learns is frozen at the moment training stops. Ask it something that was in that text and it will often answer well. Ask about your company's refund policy, or anything that changed last week, and it has two options: not know, or guess. It usually guesses, fluently.

RAG changes the order of operations. When a question arrives, you first search your own documents for the relevant passage, then hand that passage to the model along with the question and an instruction: answer using this text. The model stops being the source of facts and becomes the thing that reads and phrases them.

That distinction is the whole idea. The model supplies language ability; your documents supply truth.

Analogy — a closed-book exam versus an open-book one. In a closed-book exam you answer from memory: fast, and confidently wrong when your memory is out of date. In an open-book exam you look the answer up first — slower, but correct, and you can point at the page. Crucially, when a rule changes, the open-book student needs no re-education; you just update the book. And if the answer isn't in the book at all, the honest open-book student says so. The analogy carries the mechanism exactly: the change is where the facts come from, not how clever the reader is.

---

2. Diagram

WITHOUT RETRIEVAL — the model answers from frozen memory

  "What is the refund window?"
        │
        ▼
  ┌───────────────────┐
  │  model (trained   │──▶ "14 days"      confident. no source. possibly stale.
  │  months ago)      │
  └───────────────────┘


WITH RETRIEVAL — look it up first, then phrase it

  "What is the refund window?"
        │
        ├──▶ 1. SEARCH your documents ──▶ finds: policy.md
        │                                 "Refunds accepted within 30 days."
        ▼
  ┌───────────────────────────────────────────┐
  │  model + the retrieved passage            │
  │  "Answer using ONLY the text above."      │
  └───────────────────┬───────────────────────┘
                      ▼
       "30 days."  + a citation to policy.md
        ▲                    ▲                    ▲
        │                    │                    └─ auditable
        │                    └─ current: you edited the doc, not the model
        └─ and if nothing was found: "I don't know"


WHAT CHANGES WHEN A FACT CHANGES

  retrieval    edit 1 document              ~$0    live immediately
  fine-tuning  rebuild → retrain → deploy   hours-days, every single time

---

3. How it works

3.1 The three steps

Retrieve. Take the user's question, search your document collection, return the few most relevant passages. "Search" here usually means comparing meaning rather than exact words, so "can I send this back?" finds a passage about returns — that mechanism is the subject of Vector Search for Retrieval.

Augment. Build a prompt containing the question, the retrieved passages, and an instruction restricting the model to that text. This is the step people underestimate: the instruction is what turns a suggestion into a constraint.

Generate. The model writes an answer from the supplied passages and, if you have asked for it, names which passage it used.

The whole pipeline stage-by-stage — and the places it breaks — is Anatomy of a RAG Pipeline.

3.2 The three things retrieval actually buys

Freshness. Facts live in documents you control. Editing a document changes the answer on the next question, with no model work at all.

Attribution. The answer came from a specific passage, so you can show it. In a regulated setting this is frequently the requirement that decides the architecture, ahead of accuracy.

Abstention becomes possible. If the search finds nothing relevant, the honest output is "I don't know, here's how to reach a human." Note the word possible — see §3.5, because this is the benefit teams most often fail to collect.

3.3 RAG versus fine-tuning — different tools, common confusion

The two get compared as if they solve the same problem. They don't.

Rule of thumb: RAG for knowledge, fine-tuning for behaviour. If you want it to know your prices, retrieve. If you want it to always reply as a terse JSON object, fine-tune — or often just show it examples, which is cheaper still.

They also compose. A fine-tuned model that reliably produces your output format, fed retrieved facts, is a common and sensible production shape.

3.4 What retrieval does not fix

It doesn't make the model smarter at reasoning. It doesn't fix a bad question. And it doesn't stop the model contradicting the passage it was given — models do sometimes ignore supplied context in favour of what they "know," which is why you check the answer against the source rather than assuming.

Most importantly, retrieval cannot find what isn't there. If the policy was decided in a meeting and never written down, no retrieval system will surface it. RAG converts a knowledge problem into a documentation problem, and if your documentation is thin, RAG will faithfully expose that.

3.5 The failure that wastes the biggest benefit

Retrieval makes "I don't know" available. It does not make it happen.

The common shape: retrieval finds nothing above threshold, and instead of abstaining the code falls back to "use the best match anyway" or "answer without context." Now you have a wrong answer that is indistinguishable, in your logs, from a right one — and you've paid for a retrieval system while keeping the failure mode it was supposed to remove.

Getting this right needs two deliberate decisions: a minimum relevance score below which you refuse to answer, and a distinct status in your telemetry for "abstained" versus "answered." Neither is automatic. Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer is the full treatment.

3.6 Where RAG is the wrong choice

• The answer isn't written down. Fix the documentation first; RAG has nothing to retrieve.
• The task is reasoning, not recall — a maths problem, a plan, a code refactor. There's no passage to fetch.
• The corpus is tiny and static. Ten facts that never change belong in the prompt directly. A retrieval system is machinery you'd be maintaining for nothing.
• The task is a style transformation. Summarise, translate, reformat — the input is already in the request.
• Latency is brutally tight and answers repeat. A cache in front is cheaper and faster than retrieving the same passage all day.

---

4. The math

4.1 Cost of keeping a fact current

  RAG:          C_update = cost of one document edit          ~ 0
                (plus re-embedding that document, seconds)

  fine-tuning:  C_update = C_data_prep + C_train + C_eval + C_deploy
                paid per BATCH of changes, and you wait for the batch

The asymmetry compounds: N fact changes cost about N × 0 under retrieval and roughly ceil(N / batch) × C_train under fine-tuning. The real cost of the second is rarely the compute — it's that nobody wants to retrain for one price change, so the model stays wrong until the next batch.

4.2 Per-question cost

  C_question = C_search + C_model(in_tokens + out_tokens)

  where in_tokens = prompt + RETRIEVED PASSAGES + question

Retrieval makes each question more expensive, because you are paying for the passages as input tokens. That's the trade: more tokens per question, in exchange for correctness and a citation. RAG Cost Optimization: Find the Step That Runs Forty Times covers keeping this bounded; the shortest version is retrieve fewer passages, and cache repeats.

4.3 Worked example

Three documents. One question. The policy then changes, and nobody retrains anything.

BEFORE any change
  memory   : Refunds are accepted within 14 days of delivery.
  retrieval: Refunds are accepted within 14 days of delivery.

Both correct — the frozen snapshot happens to match reality. Now the business changes the policy to 30 days and edits the document:

AFTER the policy changes to 30 days (no retraining)
  memory   : Refunds are accepted within 14 days of delivery.
             ^ stale, and it sounds just as confident
  retrieval: Refunds are accepted within 30 days of delivery.
             ^ correct, and traceable to source 'policy-v1'

The memorised answer is wrong and indistinguishable from right — same fluency, same certainty, no source to check. The retrieved answer is correct because the document changed, and it can name the document.

And the boundary, asked something absent from the corpus entirely:

BOUNDARY — asking something absent from the corpus
  retrieval: I don't know.

That is the correct output, and it is only useful if the surrounding system is permitted to return it.

---

5. Real code

"""Memorised answers vs retrieved answers, when the underlying facts change."""

# A tiny knowledge base the "organisation" controls and edits.
DOCS = {
    "policy-v1": "Refunds are accepted within 14 days of delivery.",
    "hours":     "The support desk is open 9am to 5pm on weekdays.",
    "shipping":  "Standard shipping takes 3 to 5 working days.",
}

# What a model "knows" after training: a frozen snapshot, taken at train time.
def train_snapshot(docs: dict) -> dict:
    return {k: v for k, v in docs.items()}


def answer_from_memory(snapshot: dict, question: str) -> str:
    """Parametric: answers from the frozen snapshot. Cannot see later edits."""
    for text in snapshot.values():
        if _matches(question, text):
            return text
    return "I don't know."


def answer_from_retrieval(docs: dict, question: str) -> tuple[str, str | None]:
    """Retrieval: reads the CURRENT documents and cites which one it used."""
    for key, text in docs.items():
        if _matches(question, text):
            return text, key
    return "I don't know.", None


STOPWORDS = {"what", "when", "where", "does", "do", "is", "the", "a", "an", "to",
             "of", "in", "on", "for", "your", "you", "my", "i", "how", "are"}


def _content_stems(s: str) -> set[str]:
    """Crude stemming: keep meaningful words, compare on their first 5 letters.
    A stand-in for embedding similarity, which is the real tool for this job."""
    out = set()
    for w in s.lower().replace("?", " ").replace(".", " ").replace(",", " ").split():
        if len(w) >= 4 and w not in STOPWORDS:
            out.add(w[:5])
    return out


def _matches(question: str, text: str) -> bool:
    return bool(_content_stems(question) & _content_stems(text))


Q = "What is the refund window?"
snapshot = train_snapshot(DOCS)          # frozen here
print("BEFORE any change")
print("  memory   :", answer_from_memory(snapshot, Q))
print("  retrieval:", answer_from_retrieval(DOCS, Q)[0])

# The business changes the policy. Documents are edited; the model is NOT retrained.
DOCS["policy-v1"] = "Refunds are accepted within 30 days of delivery."

print("\nAFTER the policy changes to 30 days (no retraining)")
mem = answer_from_memory(snapshot, Q)
ret, cite = answer_from_retrieval(DOCS, Q)
print(f"  memory   : {mem}")
print(f"             ^ stale, and it sounds just as confident")
print(f"  retrieval: {ret}")
print(f"             ^ correct, and traceable to source '{cite}'")

# Cost of making the answer correct, per fact changed.
print("\nCOST OF ONE FACT CHANGE")
print(f"  {'retrieval':<12} edit 1 document                 ~$0, live immediately")
print(f"  {'fine-tuning':<12} rebuild + retrain + redeploy    hours-days, repeat every change")

# And the boundary: retrieval cannot answer what is not in the corpus.
Q2 = "Do you ship to Antarctica?"
print(f"\nBOUNDARY — asking something absent from the corpus")
print(f"  retrieval: {answer_from_retrieval(DOCS, Q2)[0]}")
print("             ^ correct behaviour. Retrieval turns 'wrong' into 'I don't know',")
print("               which is only useful if the system is allowed to say it.")

assert "14 days" in mem, "memory should still hold the stale value"
assert "30 days" in ret, "retrieval should see the edit"
assert cite == "policy-v1", "retrieval must be able to name its source"
assert answer_from_retrieval(DOCS, Q2)[1] is None, "absent facts must not be invented"
print("\nall assertions passed")

# Output:
#   BEFORE any change
#     memory   : Refunds are accepted within 14 days of delivery.
#     retrieval: Refunds are accepted within 14 days of delivery.
#
#   AFTER the policy changes to 30 days (no retraining)
#     memory   : Refunds are accepted within 14 days of delivery.
#                ^ stale, and it sounds just as confident
#     retrieval: Refunds are accepted within 30 days of delivery.
#                ^ correct, and traceable to source 'policy-v1'
#
#   COST OF ONE FACT CHANGE
#     retrieval    edit 1 document                 ~$0, live immediately
#     fine-tuning  rebuild + retrain + redeploy    hours-days, repeat every change
#
#   BOUNDARY — asking something absent from the corpus
#     retrieval: I don't know.
#                ^ correct behaviour. Retrieval turns 'wrong' into 'I don't know',
#                  which is only useful if the system is allowed to say it.
#
#   all assertions passed

The _matches function is deliberately crude — word-stem overlap, not meaning. Real retrieval compares meaning, which is why "can I send this back?" finds a passage about refunds even with no shared words. That is the next article.

---

6. Real-world example

A team built an internal assistant over a few thousand pages of operations documentation. It worked well in testing and shipped.

Two months later, support noticed staff acting on procedures that had been replaced. The assistant was confidently describing a superseded escalation path.

The cause was not the model. The documentation had been reorganised, and the old pages were still in the index alongside the new ones. Retrieval was doing its job perfectly: it found a highly relevant passage describing the old procedure, because that passage was highly relevant — it was simply obsolete. The answer cited its source, and nobody had looked at the citation, because the answer read correctly.

Two lessons. First, retrieval inherits the state of your corpus, including its contradictions. If two documents disagree, the system will confidently pick one, and "which one" is decided by a similarity score rather than by which is current. Deleting or dating superseded documents is part of running RAG, not housekeeping you can defer.

Second, a citation nobody checks is not a safeguard. The attribution was there and unread. What eventually caught it was adding a document date to the answer, so a stale source became visible in the reply itself rather than one click away.

Worth noting what wasn't wrong: no model was at fault, no prompt needed tuning, and a bigger model would have made the same mistake with more polish.

---

7. Interview questions companies actually ask

Q1. What problem does RAG actually solve? It changes where facts come from. A model's knowledge is frozen at training time, so anything private, recent, or changeable is either unknown or guessed at fluently. Retrieval fetches the relevant passage at question time, which gives you three things training cannot: the answer updates when you edit a document, it can cite the source it used, and it can abstain when nothing relevant exists.

Q2. When would you fine-tune instead? When you want to change behaviour rather than supply facts — a consistent output format, a house style, a specialised task shape. Fine-tuning is a poor tool for knowledge: each fact change needs a training run, it cannot cite a source, and it does not abstain. The short version is RAG for knowledge, fine-tuning for behaviour, and they compose fine together.

Q3. Your RAG system gives a confidently wrong answer. Where do you look first? Upstream of the model, and in this order: did retrieval return the right passage, and did the passage actually contain the deciding fact? The most common cause is that the answer's basis never reached the model — because chunking split it, or the top result was relevant-but-incomplete. Check the retrieved context before touching the prompt; you will usually find the bug there.

Q4. Does RAG stop hallucination? It reduces it and does not eliminate it. Models can still contradict supplied context, and — more commonly — the system is configured to answer even when retrieval found nothing useful, which reproduces exactly the failure RAG was meant to remove. You need a minimum relevance score, permission to abstain, and a distinct log status for having abstained. Retrieval makes honesty possible; it does not make it automatic.

Q5. Is RAG always the right call for a company's internal knowledge? No. If the corpus is small and static, put the facts in the prompt and skip the machinery. If the knowledge isn't written down, RAG has nothing to retrieve and you have a documentation problem wearing an AI costume. And if the task is reasoning rather than recall, there's no passage to fetch.

Q6. What ongoing work does a RAG system need that a plain model doesn't? Corpus hygiene, mainly, and it's usually underestimated. Superseded documents must be removed or dated, or retrieval will confidently serve them — it ranks by similarity, not by currency. Add re-embedding when documents change, monitoring retrieval quality separately from answer quality, and a scored test set so you can tell whether a change helped.

Q7. How does RAG change your cost per question? It raises it. You pay for a search plus the retrieved passages as input tokens on every question, so a RAG answer costs more than an unaugmented one. The levers are retrieving fewer passages, keeping them tight, caching repeated questions, and not calling the model at all when a deterministic rule can answer.

---

8. When to use / tradeoffs

Reach for RAG when:

• The facts are private, or change more often than you'd retrain
• You must be able to show where an answer came from
• The corpus is too large to fit in a prompt
• "I don't know" is a better outcome than a plausible guess

Reach for something else when:

• The corpus is small and static → put it in the prompt
• You need a format or style → few-shot examples, or fine-tuning
• The task is reasoning or transformation → no passage to retrieve
• The knowledge isn't documented → write it down first

Honest limits. The comparison in §4 flatters retrieval in ways worth naming. It treats a document edit as free, when in practice you also re-embed, invalidate caches, and keep the index consistent — small per change, real at scale. It says nothing about retrieval quality, which is the hard part and the subject of most of the rest of this module; a RAG system with poor retrieval is worse than no RAG, because it adds cost and latency while still being wrong. The toy matcher in §5 uses word overlap and would fail on any paraphrase, so it demonstrates the architecture rather than the technique. And the freshness argument assumes your corpus is actually kept current — as §6 shows, retrieval inherits your documentation's problems faithfully, including its contradictions.

---

9. Summary + related articles

• RAG looks facts up at question time instead of relying on what a model absorbed during training. The model supplies language; your documents supply truth.
• Three things it buys: freshness (edit a document, not a model), attribution (it can cite its source), and abstention becomes possible.
• RAG for knowledge, fine-tuning for behaviour. They solve different problems and compose well.
• Cost of one fact change: a document edit versus a training run. That asymmetry, not accuracy, is usually what decides the architecture.
• Per-question cost goes up — you pay for the passages as input tokens. That's the trade.
• Abstention is available, not automatic. Without a score threshold and permission to say "I don't know", you keep the failure you were trying to remove.
• Retrieval cannot find what isn't written down, and it inherits your corpus's contradictions — it ranks by similarity, not by currency.
• A citation nobody reads is not a safeguard.

Related:

• Vector Search for Retrieval — how "find the relevant passage" actually works
• Anatomy of a RAG Pipeline — the stages end to end, and which one causes a bad answer
• Embeddings and Cosine Similarity — what a vector representation of meaning is
• Document Processing: What You Index Sets the Ceiling — getting real documents into a usable state
• Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — splitting documents, and the score floor §3.5 argues for
• Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — where the vectors live and how search scales
• Generation Pipeline: Assembling Context and Enforcing a Citation Contract — building the augmented prompt properly
• Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer — making "I don't know" actually happen
• RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring retrieval quality separately from answer quality
• RAG Cost Optimization: Find the Step That Runs Forty Times — keeping the per-question cost bounded
• Agentic RAG: Routing Retrieval to Specialists — when one retrieval pass isn't enough
• RAG at Scale — the same ideas at millions of documents

Resources

• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 — the paper that named the pattern: https://arxiv.org/abs/2005.11401
• Karpukhin et al. (2020) — Dense Passage Retrieval for Open-Domain Question Answering, arXiv:2004.04906 — why dense vector retrieval replaced keyword search for this job: https://arxiv.org/abs/2004.04906
• Guu et al. (2020) — REALM: Retrieval-Augmented Language Model Pre-Training, arXiv:2002.08909 — the closely related earlier work: https://arxiv.org/abs/2002.08909
• Gao et al. (2023) — Retrieval-Augmented Generation for Large Language Models: A Survey, arXiv:2312.10997 — a map of the variants before you pick one: https://arxiv.org/abs/2312.10997
• Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 1 and 6 — free online; retrieval did not begin with LLMs and the fundamentals still apply: https://nlp.stanford.edu/IR-book/