Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer

TL;DR — A retrieval-augmented system that always generates an answer, regardless of how weak or irrelevant the retrieved context actually is, will confidently answer questions its source material never addressed at all. The fix is a design decision made before generation ever runs: check whether retrieval actually found something relevant enough to answer from, and if it didn't, refuse — "not found in the source material" — rather than letting the generator produce a fluent, plausible-sounding answer built on a weak match. In the harness below, a naive design confidently answers 2 out of 2 out-of-scope test questions anyway; a design that checks retrieval strength before generating correctly abstains on both, while still answering all 3 in-scope questions normally. This is a design choice, not a detection technique applied afterward — the boundary it draws is only as good as the retrieval score it's based on, and it fails exactly where retrieval itself fails.

---

1. Simple explanation

A language model asked to answer a question, given some retrieved context, will generally produce a fluent, confident-sounding answer — that's what generation does, regardless of whether the retrieved context actually supports that answer or barely relates to the question at all. Left unchecked, this means a retrieval-augmented system answers questions its source material never covered with exactly the same confident tone it uses for questions the material answers well, and a user has no way to tell the difference from the answer's tone alone.

Analogy — a reference librarian versus someone who feels obligated to answer. A good reference librarian, asked a question the library's collection genuinely doesn't cover, says "we don't have anything on that" rather than improvising a plausible-sounding answer from loosely related material just to seem helpful. Someone who feels they must always produce an answer, by contrast, will reach for the closest thing on the shelf and present it with the same confidence as a genuinely well-supported answer — and a person asking the question has no way to tell, from the answer's tone alone, which situation they're in. Designing a system to say "I don't have that" is designing it to behave like the good librarian on purpose, rather than hoping the model happens to hedge appropriately on its own.

---

2. Diagram

NAIVE DESIGN                              GROUNDED-BY-DESIGN

  question                                  question
     |                                         |
     v                                         v
  retrieve best match                       retrieve best match
     |                                         |
     v                                         v
  generate an answer                        score >= threshold?
  REGARDLESS of match quality                  |         |
     |                                        YES        NO
     v                                         |          |
  confident-sounding answer,                   v          v
  EVEN for an out-of-scope              generate       "not found in
  question with a weak match            an answer      the source material"

  MEASURED (harness in §5, 3 in-scope + 2 out-of-scope questions):

    design            in-scope answered     out-of-scope answered anyway
    -----------------------------------------------------------------
    naive                   3/3                      2/2   <- the problem
    grounded-by-design      3/3                      0/2   <- correct abstention

---

3. How it works

3.1 Generation doesn't know when it shouldn't happen — a check has to decide that first

An LLM generating an answer from a prompt containing a question and some retrieved context will produce something fluent, because fluent generation is what the underlying model does regardless of input quality. Nothing about the generation step itself carries a reliable signal for "this context doesn't actually support an answer" — that judgment has to be made before generation runs, using a proxy the system can actually measure, most commonly the retrieval step's own similarity or relevance score for its best match.

3.2 The threshold gate runs before generation, not as a check on its output

The design places a decision point between retrieval and generation: if the best-matching retrieved content scores below a chosen threshold, the system returns a fixed refusal message and never calls the generator at all. This is different from generating an answer and then trying to detect, after the fact, whether it was well-supported — checking beforehand means an ungrounded answer is never produced in the first place, rather than being produced and then (hopefully) caught. Placing the gate before generation also saves the cost of a generation call for every question the system was always going to refuse anyway.

3.3 The refusal has to say something useful, not just fail silently

A well-designed abstention doesn't return an empty response or a generic error — it tells the user plainly that the source material doesn't address their question, which is itself useful information (it means "look elsewhere" or "rephrase," not "something broke"). This is the same design decision as choosing what a search engine shows for zero results: a blank page looks like a bug, while an explicit "no results found" message is a correct, informative answer to a query that legitimately has none.

Where this stops working: the entire design rests on the retrieval score being a trustworthy proxy for "this content actually supports an answer," and it inherits every weakness retrieval itself has. A question phrased in different vocabulary than the source material can score below threshold even though the material genuinely covers it — a false abstention, refusing to answer something it actually could have. A question that happens to share surface vocabulary with an unrelated passage can score above threshold without the passage actually answering it — a false pass, generating an answer from context that doesn't really support it. The threshold gate is only as good as the retrieval score feeding it; it doesn't fix a bad retrieval step, it just acts honestly on whatever that step reports.

---

4. The math

There's no formula to derive beyond the threshold rule itself:

if best_match_score < threshold:
    return "not found in the source material"
else:
    generate_answer(question, best_match)

The one number worth deriving is the outcome, not a formula: in a well-separated case (in-scope questions scoring meaningfully higher than out-of-scope ones), a single threshold correctly splits both categories, as shown in §5. The harder case — not measured here, but worth stating plainly — is when in-scope and out-of-scope scores overlap; no single threshold can then separate them perfectly, and the choice of threshold becomes a tradeoff between false abstentions (refusing answerable questions) and false passes (answering unsupported ones), tuned against whichever error the system can tolerate less.

---

5. Real code

import re
import math
from collections import Counter


CORPUS = {
    "d1": "The recycling program accepts glass, aluminum cans, and cardboard "
          "every Tuesday morning starting at 7am.",
    "d2": "Parking permits for residents cost 40 dollars a year and can be "
          "renewed online or at the town office.",
    "d3": "Noise complaints after 10pm on weekdays can be reported through "
          "the town's non-emergency line.",
}

QUESTIONS = [
    ("When does the recycling program pick up glass and cardboard?", True),
    ("How much does a resident parking permit cost per year?", True),
    ("How do I report a noise complaint after 10pm?", True),
    ("What is the property tax rate for a house in this town?", False),
    ("Is street parking free on national holidays?", False),
]

STOPWORDS = {"the", "a", "an", "is", "does", "do", "how", "what", "on",
             "for", "in", "at", "to", "and", "this", "i", "my", "if"}


def vectorize(text):
    words = re.findall(r"[a-z]+", text.lower())
    return Counter(w for w in words if w not in STOPWORDS)


def cosine(a, b):
    common = set(a) & set(b)
    dot = sum(a[w] * b[w] for w in common)
    mag_a = math.sqrt(sum(v * v for v in a.values()))
    mag_b = math.sqrt(sum(v * v for v in b.values()))
    return dot / (mag_a * mag_b) if mag_a and mag_b else 0.0


DOC_VECS = {k: vectorize(v) for k, v in CORPUS.items()}


def best_match(question):
    q_vec = vectorize(question)
    scored = sorted(
        ((cosine(q_vec, v), k) for k, v in DOC_VECS.items()),
        reverse=True,
    )
    return scored[0]  # (score, doc_id)


def answer_naive(question):
    """Always generates a confident-sounding answer, regardless of whether
    the retrieved context actually supports one -- the failure mode this
    article is about."""
    score, doc_id = best_match(question)
    return f"Based on {doc_id}: [a confident-sounding answer, generated " \
           f"regardless of score={score:.2f}]"


GROUNDING_THRESHOLD = 0.15


def answer_grounded(question):
    """Checks the retrieval score BEFORE generating anything. Below the
    threshold, refuses rather than lets the generator produce a plausible
    but unsupported answer."""
    score, doc_id = best_match(question)
    if score < GROUNDING_THRESHOLD:
        return f"Not found in the source material (best match {doc_id}, " \
               f"score={score:.2f} below threshold {GROUNDING_THRESHOLD})"
    return f"Based on {doc_id} (score={score:.2f}): [a real answer, " \
           f"generated because retrieval cleared the threshold]"


print(f"{'in-scope?':10}{'question':52}{'best score':11}")
naive_confident_on_oos = 0
grounded_correct_abstentions = 0
oos_count = 0

for question, in_scope in QUESTIONS:
    score, doc_id = best_match(question)
    print(f"{str(in_scope):10}{question:52}{score:<11.3f}")
    if not in_scope:
        oos_count += 1
        naive_result = answer_naive(question)
        grounded_result = answer_grounded(question)
        if "confident-sounding" in naive_result:
            naive_confident_on_oos += 1
        if "Not found" in grounded_result:
            grounded_correct_abstentions += 1

print(f"\nout-of-scope questions: {oos_count}")
print(f"naive design: confidently answered {naive_confident_on_oos}/{oos_count} "
      f"out-of-scope questions anyway")
print(f"grounded design: correctly abstained on {grounded_correct_abstentions}/{oos_count} "
      f"out-of-scope questions")

assert naive_confident_on_oos == oos_count
assert grounded_correct_abstentions == oos_count

in_scope_grounded_answers = sum(
    1 for q, s in QUESTIONS if s and "real answer" in answer_grounded(q)
)
print(f"in-scope questions still answered normally by the grounded design: "
      f"{in_scope_grounded_answers}/{sum(1 for _, s in QUESTIONS if s)}")
assert in_scope_grounded_answers == sum(1 for _, s in QUESTIONS if s)

print("\nasserts passed: naive design confidently answers every "
      "out-of-scope question anyway; grounded design correctly abstains "
      "on all of them while still answering every in-scope question normally")

# Output:
# in-scope? question                                            best score
# True      When does the recycling program pick up glass and cardboard?0.436
# True      How much does a resident parking permit cost per year?0.314
# True      How do I report a noise complaint after 10pm?       0.359
# False     What is the property tax rate for a house in this town?0.124
# False     Is street parking free on national holidays?        0.124
#
# out-of-scope questions: 2
# naive design: confidently answered 2/2 out-of-scope questions anyway
# grounded design: correctly abstained on 2/2 out-of-scope questions
# in-scope questions still answered normally by the grounded design: 3/3
#
# asserts passed: naive design confidently answers every out-of-scope question anyway; grounded design correctly abstains on all of them while still answering every in-scope question normally

Both asserts passed on the run that produced this output: the naive design's failure mode (confidently answering both out-of-scope questions) and the grounded design's fix (correctly abstaining on both, while still answering every genuinely in-scope question) are both directly verified, not just described.

---

6. Real-world example

A team shipped a document-grounded Q&A assistant that generated an answer for every question it received, with no check on retrieval quality — the reasoning at the time was that the retrieval step would naturally surface the most relevant document, and the generator would naturally produce a reasonable answer from whatever it was given. In testing against questions the team wrote themselves, all of which were genuinely covered by the source material, this worked fine.

Once real users started asking questions, a meaningful fraction of them fell outside what the source material actually covered — questions about a related but different topic, questions the documentation simply hadn't been written to address yet. The assistant answered every one of them anyway, in the same confident tone it used for well-supported answers, because nothing in the pipeline distinguished "I found something relevant" from "I found the closest thing available, which isn't very relevant at all." Several of these confidently-wrong answers were forwarded to other users as if they were authoritative, since nothing about their presentation signaled otherwise.

The fix was adding exactly the threshold gate in §5: retrieval score checked before generation, with a real "not found" response below threshold rather than a generated answer regardless of score. The team's retrospective distinguished this clearly from adding a "hallucination detector" after the fact — checking beforehand meant the system never generated the ungrounded answer at all, rather than generating it and then trying to catch it, which is both cheaper (no wasted generation call) and more reliable (there's no detection step that itself might miss a case).

---

7. Interview questions companies actually ask

Q1. Why doesn't a language model just naturally say "I don't know" when it lacks good context, without a special design for it? Generation produces fluent text regardless of whether the underlying context actually supports a confident answer — nothing about the generation process itself carries a reliable, built-in signal that the retrieved material is weak or irrelevant. Getting reliable refusal behavior requires an explicit check, made using something the system can actually measure (typically the retrieval step's own relevance score), rather than hoping the model happens to hedge appropriately on its own.

Q2. Why check retrieval quality before generation instead of checking the generated answer's quality afterward? Checking beforehand means an ungrounded answer is never produced at all, which is both cheaper (no wasted generation call for a question that was always going to be refused) and more reliable, since there's no separate detection step after generation that could itself fail to catch a bad answer. It also means the user never sees a fluent, confident-sounding wrong answer even briefly — refusal happens before anything misleading is generated.

Q3. What should a system say when it abstains, and why does it matter? It should say plainly that the source material doesn't cover the question, not fail silently or return a generic error — an explicit "not found" is itself useful information, telling the user to look elsewhere or rephrase, the same way a search engine's honest "no results found" is a correct answer to a query that has none, not a failure of the search engine.

Q4. A grounding threshold correctly abstains on clearly out-of-scope questions but incorrectly abstains on an in-scope question phrased in unusual vocabulary. What's actually wrong? The threshold gate is only as reliable as the retrieval score it depends on — if a genuinely answerable question happens to use different vocabulary than the source material, retrieval itself may score it too low, and the abstention gate correctly (from its own perspective) refuses a question it actually could have answered. This isn't a flaw in the abstention design itself; it's the abstention design faithfully surfacing a retrieval-quality problem that exists upstream of it.

Q5. How would you choose the actual threshold value, rather than picking an arbitrary number? Collect a set of known in-scope and known out-of-scope questions, score them all through the actual retrieval mechanism, and look for where the two score distributions separate — the threshold should sit in the gap between the lowest scores among genuinely answerable questions and the highest scores among genuinely unanswerable ones. If those two distributions overlap significantly, no single threshold value can cleanly separate them, and the choice becomes a deliberate tradeoff between the two error types rather than a value that eliminates both.

Q6. What's the tradeoff between a stricter (higher) threshold and a looser (lower) one? A stricter threshold reduces false passes (generating an answer from weak, unsupported context) but increases false abstentions (refusing questions that were actually answerable but scored lower than expected, often due to vocabulary mismatch). A looser threshold does the opposite. The right setting depends on which error costs more in the specific system — a system where a wrong answer is dangerous should lean strict; one where over-refusing frustrates users more than an occasional weak answer should lean loose.

Q7. Why is this called a design decision rather than an evaluation technique? Because it changes what the system does at answer time — refusing to generate — rather than only measuring, after the fact, how well an already-generated answer was supported. Evaluating groundedness (as covered in RAG evaluation methodology) tells you how well your system is doing; building the threshold gate in changes the system's actual behavior so that ungrounded answers are architecturally prevented from being produced, not just detected afterward.

---

8. When to use / tradeoffs

Reach for a pre-generation grounding gate when:

• the system will realistically receive questions outside what its source material covers
• a confidently wrong answer is more costly than an honest "I don't know"
• retrieval scores meaningfully separate in-scope from out-of-scope questions for your actual corpus and question patterns

Honest limits. This design is exactly as reliable as the retrieval score it depends on — it doesn't independently verify that a passage actually supports an answer, it only checks that retrieval considered it a strong match, and those are not the same thing; a passage can score highly on lexical or semantic similarity to a question without actually answering it, producing a confident but subtly wrong answer that clears the gate. The numbers in §5 come from a small, deliberately well-separated synthetic case; real question sets rarely separate this cleanly, and the harder, more common case is tuning a threshold against genuinely overlapping score distributions, which requires an explicit tradeoff rather than a clean cutoff. Finally, this design prevents unsupported answers, not wrong ones in general — a retrieved passage that is itself outdated or incorrect will still produce an answer that clears the gate and is confidently wrong for a completely different reason.

---

9. Summary + related articles

• A generator produces fluent, confident-sounding text regardless of whether the retrieved context actually supports an answer — nothing about generation itself signals "I shouldn't answer this."
• Checking retrieval strength against a threshold before generation, and refusing below it, prevents an ungrounded answer from ever being produced, rather than trying to detect one after the fact.
• Measured: a naive design confidently answered 2/2 out-of-scope test questions; a grounded-by-design system correctly abstained on both, while still answering all 3 in-scope questions normally.
• Boundary: the gate is only as reliable as the retrieval score it's based on — it inherits every failure mode of retrieval itself, including vocabulary mismatches that cause false abstentions and coincidental lexical overlap that causes false passes.

Related:

• Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — the retrieval-quality fundamentals that determine how reliable the grounding score feeding this gate actually is
• RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring groundedness and faithfulness after generation, the complementary evaluation-side discipline to this article's before-generation design decision