Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems

TL;DR — A retrieval-augmented system can't compare a question against a whole document at once, so the document is split into overlapping chunks that get embedded and searched independently. Chunk size trades context against precision, overlap stops a sentence from being cut in half at a chunk boundary, and a similarity score is only meaningful relative to the specific scoring function that produced it — never as a universal number. This stops working the moment you swap embedding models or scoring functions without re-tuning the threshold, and it stops working entirely for documents whose structure (tables, code, a mix of one-line FAQ entries and 50-page manuals) doesn't fit a single fixed chunk size.

---

1. Simple explanation

A search assistant that answers questions from a document corpus can't feed the whole corpus into a similarity comparison at once — it has to break every document into pieces small enough to compare word-for-word (or vector-for-vector) against the question, one piece at a time. The size of those pieces, how much they overlap with their neighbors, and the number you call "a good match" are three separate decisions, and getting any one of them wrong produces the same symptom: the assistant fails to find something that is clearly in the source material.

Analogy — looking something up with index cards, not chapters. Imagine you hand a librarian one page at a time and ask them to hold up the page that answers your question. If you photocopied whole chapters and handed those over one at a time, a five-page chapter that mentions your topic once wouldn't stand out — the one relevant paragraph is diluted by four pages of unrelated material, and the whole chapter's average "relevance" barely rises above chance. If instead you copied every sentence onto its own index card, the librarian could find the single card with the exact wording — but a card holding one isolated sentence might not tell you the qualifying condition stated two sentences earlier, so you'd have the words but not their meaning. Retrieval systems face exactly this tension, and the size of the "card" is the chunk size. There is no chunk size that is correct in general — only one that is correct for a particular corpus and a particular kind of question.

---

2. Diagram

RAW DOCUMENT (285 words)
+---------------------------------------------------------------+
| section A | section B | section C | section D | section E     |
+---------------------------------------------------------------+

CHUNKING with overlap — each window repeats the tail of the
previous one, so no sentence falls entirely on a boundary and
gets split between two chunks that individually look irrelevant:

  chunk0 [words   0.. 60)
        chunk1 [words  45..105)
                chunk2 [words  90..150)
                        chunk3 [words 135..195)
                                chunk4 [words 180..240)
                                        chunk5 [words 225..285)

RETRIEVAL AT QUERY TIME
  query   --embed-->  q_vec
  chunk_i --embed-->  c_vec_i                     (for every chunk)
  score_i = similarity(q_vec, c_vec_i)

  chunk 2  |################################|  0.256   <- best match
  chunk 1  |###################|              0.177
  chunk 0  |##########|                       0.092
  chunk 3  |#########|                        0.086
  chunk 4  |######|                           0.054
  chunk 5  |######|                           0.052

THRESHOLD GATE — meaningful only for THIS scoring function
  score >= threshold  ->  answer, citing the winning chunk
  score <  threshold  ->  abstain: "not found in the source material"

---

3. How it works

3.1 Chunk size trades context against precision

A small chunk (one or two sentences) is precise about where a match is, but it can be lexically on-topic while missing the qualifying clause stated in the surrounding sentences — a chunk that says "rollbacks restore the previous image" might be retrieved confidently even though the very next sentence says the restore excludes database migrations, and that sentence lives in a different chunk. A large chunk (a full section or page) keeps that context together, but dilutes the retrieval signal: a 500-word chunk containing one relevant sentence and 490 irrelevant ones scores only slightly higher than a chunk with no relevant sentence at all, because the similarity function measures the whole chunk, not the one sentence inside it that actually matters. Every corpus sits somewhere on this line, and the right chunk size is the point where a single chunk usually contains one complete idea — no more, no less.

3.2 Overlap prevents boundary loss

If a chunk boundary happens to land in the middle of the one sentence that answers the question, the sentence is split across two chunks and neither one retrieves well, because neither contains the complete idea. Overlap — repeating the last N words of one chunk at the start of the next — means that as long as the important sentence is shorter than the overlap window, it appears whole in at least one chunk. This costs storage and embedding time roughly in proportion to how large the overlap fraction is relative to the chunk size, which is why overlap is usually set as a fraction of chunk size (a quarter to a third is a common starting point) rather than a fixed constant that stops making sense when chunk size changes.

3.3 A similarity score means nothing outside the scoring function that produced it

This is the part that causes real production incidents, and it's demonstrated directly in §5's code: two different, entirely reasonable ways of scoring the same six chunks against the same query — term-frequency cosine similarity and word-overlap (Jaccard) similarity — produce numbers on completely different scales for the identical retrieval problem. In the actual run, cosine similarity ranged from 0.052 to 0.256, while Jaccard similarity for the same six chunks ranged from 0.017 to 0.109 — roughly half the numeric range, on the same underlying data, for the same "best" answer. Both methods agree on which chunk is the best match; they disagree completely on what number counts as "confidently relevant."

Real embedding models behave the same way for the same underlying reason: each model learned its own geometry, so one model's "this is obviously the right passage" score might be 0.85 and a different model's score for the identical match might be 0.4. A threshold tuned against one scoring function is a property of that function, not a property of "relevance" in general. The moment the scoring function underneath it changes — a new embedding model, a provider migration, a different similarity metric — the old threshold either rejects everything (if the new scores run lower) or accepts everything (if they run higher), and it will look, from the outside, exactly like the assistant "got worse" or "started making things up," when the actual fault is a stale number.

Where this stops applying: the whole design assumes that lexical or semantic similarity between a question and a passage is a reasonable proxy for "this passage answers the question." That assumption breaks for questions that require combining two chunks that individually share no vocabulary with the question (multi-hop reasoning), or for questions whose answer is a negation or an inference the source never states directly. At that point the fix isn't a better chunk size or a re-tuned threshold — it's a different retrieval strategy (query decomposition, multi-step retrieval, or a graph-structured index over the corpus).

---

4. The math

4.1 The chunk-count formula

Given a document of total_words, a chunk size chunk_size, and an overlap overlap (both in words, overlap < chunk_size):

step        = chunk_size - overlap
num_chunks  = ceil( max(total_words - overlap, 0) / step )

The intuition: the first chunk covers chunk_size words for free. Every chunk after that only has to cover step new words, because it re-covers overlap words from the previous chunk. So after subtracting the one-time overlap from the total, the remaining words are covered in steps of size step.

4.2 Worked example

Document length: 285 words (verified by len(DOC.split()) in the code below).

Case 1 — chunk_size=60, overlap=15:

step       = 60 - 15 = 45
num_chunks = ceil( (285 - 15) / 45 ) = ceil( 270 / 45 ) = ceil(6.0) = 6

The actual chunker produced exactly 6 chunks, matching the formula.

Case 2 — chunk_size=30, overlap=15:

step       = 30 - 15 = 15
num_chunks = ceil( (285 - 15) / 15 ) = ceil( 270 / 15 ) = ceil(18.0) = 18

The actual chunker produced exactly 18 chunks, again matching the formula. Note that halving the chunk size (60 -> 30) more than tripled the chunk count (6 -> 18), not doubled it — because overlap stayed fixed at 15 words, so it went from 25% of the chunk (60-word case) to 50% of the chunk (30-word case). Shrinking chunk size without shrinking overlap proportionally makes consecutive chunks repeat more of each other, and the corpus needs more chunks to make the same net forward progress through the document.

---

5. Real code

import re
import math
from collections import Counter

DOC = """
Setting up the deployment pipeline. Every service in the platform ships through
a three-stage pipeline: build, stage, and release. The build stage compiles the
service and runs the unit test suite. A build that fails any test is rejected
before it reaches the stage environment. The stage environment mirrors
production traffic patterns at ten percent scale, and every deploy sits there
for a minimum soak time before promotion is allowed.

Rolling back a release. If an error budget burns down faster than the alerting
threshold allows, the on-call engineer can trigger an automatic rollback to the
last known-good release. Rollbacks restore the previous container image and
the previous configuration bundle together, since restoring only one of the two
has caused mismatched-schema incidents in the past. A rollback does not restore
database migrations; those are forward-only and must be reverted by a separate,
manually reviewed migration.

Configuring the service mesh. The mesh routes traffic between services using
weighted routing rules stored in the control plane. A canary release starts at
five percent of traffic and increases in fixed steps only if the error rate and
p99 latency both stay under their configured thresholds for the full observation
window. If either metric breaches its threshold at any point in the window, the
canary is aborted and traffic reverts to one hundred percent on the previous
version.

Handling secrets. Secrets are never stored in the service repository. Each
service reads its secrets from a mounted volume populated at container start
by the secrets manager, and the volume is backed by tmpfs so secrets never
touch disk. Rotating a secret requires a restart of every pod that mounted it,
which the deployment controller schedules automatically within a maintenance
window.
""".strip()


def chunk_text(text, chunk_size_words, overlap_words):
    words = text.split()
    if overlap_words >= chunk_size_words:
        raise ValueError("overlap must be smaller than chunk size")
    chunks, start = [], 0
    step = chunk_size_words - overlap_words
    while start < len(words):
        chunks.append(" ".join(words[start:start + chunk_size_words]))
        if start + chunk_size_words >= len(words):
            break
        start += step
    return chunks


def expected_chunk_count(total_words, chunk_size_words, overlap_words):
    step = chunk_size_words - overlap_words
    return math.ceil(max(total_words - overlap_words, 0) / step)


def term_frequency_vector(text):
    return Counter(re.findall(r"[a-z]+", text.lower()))


def cosine_similarity(vec_a, vec_b):
    common = set(vec_a) & set(vec_b)
    dot = sum(vec_a[w] * vec_b[w] for w in common)
    mag_a = math.sqrt(sum(v * v for v in vec_a.values()))
    mag_b = math.sqrt(sum(v * v for v in vec_b.values()))
    return dot / (mag_a * mag_b) if mag_a and mag_b else 0.0


def jaccard_similarity(vec_a, vec_b):
    set_a, set_b = set(vec_a), set(vec_b)
    return len(set_a & set_b) / len(set_a | set_b) if set_a and set_b else 0.0


total_words = len(DOC.split())
print(f"Document length: {total_words} words")

for size, overlap in [(60, 15), (30, 15)]:
    chunks = chunk_text(DOC, size, overlap)
    predicted = expected_chunk_count(total_words, size, overlap)
    print(f"chunk_size={size} overlap={overlap} -> {len(chunks)} chunks "
          f"(formula predicted {predicted})")
    assert len(chunks) == predicted

query = "how does an automatic rollback work and what does it restore"
chunks = chunk_text(DOC, 60, 15)
q_vec = term_frequency_vector(query)

print(f"\nQuery: {query}")
print(f"{'chunk':<7}{'cosine':<10}{'jaccard':<10}")
cos_scores, jac_scores = [], []
for i, c in enumerate(chunks):
    c_vec = term_frequency_vector(c)
    cos, jac = cosine_similarity(q_vec, c_vec), jaccard_similarity(q_vec, c_vec)
    cos_scores.append(cos)
    jac_scores.append(jac)
    print(f"{i:<7}{cos:<10.3f}{jac:<10.3f}")

print(f"\ncosine  range: {min(cos_scores):.3f} - {max(cos_scores):.3f}")
print(f"jaccard range: {min(jac_scores):.3f} - {max(jac_scores):.3f}")

best_chunk = cos_scores.index(max(cos_scores))
assert "rollback" in chunks[best_chunk].lower()
print(f"\nTop cosine match is chunk {best_chunk} "
      f"(contains 'rollback': {'rollback' in chunks[best_chunk].lower()})")

# Output:
# Document length: 285 words
# chunk_size=60 overlap=15 -> 6 chunks (formula predicted 6)
# chunk_size=30 overlap=15 -> 18 chunks (formula predicted 18)
#
# Query: how does an automatic rollback work and what does it restore
# chunk  cosine    jaccard
# 0      0.092     0.041
# 1      0.177     0.083
# 2      0.256     0.109
# 3      0.086     0.017
# 4      0.054     0.018
# 5      0.052     0.038
#
# cosine  range: 0.052 - 0.256
# jaccard range: 0.017 - 0.109
#
# Top cosine match is chunk 2 (contains 'rollback': True)

Both assert statements passed on the run that produced this output — the chunk counts matched §4.2's formula exactly, and the top-scoring chunk under cosine similarity genuinely contained the word the query was asking about, not just a nearby one.

---

6. Real-world example

A team building an internal documentation search assistant tuned their retrieval threshold during development against one embedding provider: they tried several cutoff values against a small set of known question/answer pairs and settled on a threshold that correctly answered their test questions while abstaining on a handful of out-of-scope ones. Everything worked in staging.

Weeks later, they swapped embedding providers for cost reasons — the new provider was cheaper per call and had lower latency. Nobody re-ran the threshold-tuning exercise, because the swap looked like a drop-in replacement: same interface, same vector dimensionality, same integration code. Within a day of the swap reaching production, the assistant started abstaining on nearly every question, including ones it had answered correctly for months. The team's first hypothesis was a regression in the retrieval index itself, and they spent most of a day checking document ingestion, re-verifying the corpus was intact, and confirming the vector store hadn't silently dropped entries.

The actual cause was simpler and easy to miss precisely because nothing had "broken" in the traditional sense: the new embedding model's cosine similarity scores for genuine matches topped out meaningfully lower than the old model's did for the same documents, so the old threshold — a number that had never been tied to a specific model in anyone's mental model of the system — now rejected almost everything as "not confident enough." The fix was not a code change; it was re-running the same threshold-selection process against the new model's actual score distribution and shipping a new number. The lesson the team took away: a retrieval threshold should always be checked into configuration next to a note recording which embedding model and scoring function it was tuned against, so that "we changed the model" and "we need a new threshold" are the same mental step, not two separate discoveries a day apart.

---

7. Interview questions companies actually ask

Q1. Why not just embed an entire document as a single vector instead of splitting it into chunks? A single vector for a long document averages together the meaning of everything in it, so a query about one narrow topic buried in a large document produces a weak, diluted similarity score against the whole-document vector — the signal from the relevant part gets outvoted by the rest of the content. Chunking keeps each embedded unit narrow enough that its vector actually represents one coherent idea, at the cost of needing to search many vectors per document instead of one.

Q2. What goes wrong if your chunk boundaries consistently land in the middle of important sentences? The sentence gets split across two chunks, and neither half contains the complete idea, so neither chunk retrieves as a strong match even though the answer was present in the source text the whole time. This shows up as false negatives that are hard to debug because a human reading the source document sees the answer immediately and assumes the retrieval system is broken, when the real issue is that the chunk boundaries and overlap window don't guarantee the sentence survives intact in at least one chunk.

Q3. How would you pick chunk size and overlap differently for a corpus of short FAQ entries versus a corpus of long structured manuals? Short FAQ entries are usually already at the right granularity — one question and answer is typically one complete idea — so the natural chunk boundary is the entry boundary itself, with little or no overlap needed. Long structured manuals benefit from a fixed word-count window with meaningful overlap, because a section boundary in a manual doesn't reliably align with a complete idea the way an FAQ entry does; the manual case is where the chunk-size/overlap tradeoff in §3.1 and §3.2 actually matters.

Q4. Why can't you compare a cosine-similarity score from one embedding model directly against a score from a different model? Each embedding model is trained to place text in its own vector space with its own geometry, so the numeric range that represents "highly relevant" for one model has no guaranteed relationship to the range for another model — as demonstrated in §5, even two simple non-learned scoring methods (term-frequency cosine and Jaccard) produce different numeric ranges for the identical retrieval problem. Any threshold is a property of the specific scoring function it was tuned against, not a universal measure of relevance.

Q5. Your retrieval system starts abstaining far more often right after you switch embedding providers. What's your first hypothesis, and how would you confirm it? The first hypothesis should be that the similarity-score distribution shifted under the new model and the existing threshold no longer matches it — not that the corpus or retrieval logic broke, since neither of those changed. To confirm it, re-run retrieval for a small set of known-good query/answer pairs against the new model and compare the resulting score distribution to the one the current threshold was originally tuned against; if the new "confidently relevant" scores sit below the old threshold, that confirms the diagnosis and points directly at the fix.

Q6. Overlap increases both storage and embedding cost. How do you decide how much overlap is worth paying for? Overlap only needs to be large enough that the longest single sentence or idea you expect in the corpus can't be fully cut in half by a chunk boundary — beyond that point, additional overlap increases cost without reducing boundary-loss risk further. A quarter to a third of chunk size is a reasonable starting point for prose, and the right number for a specific corpus should come from checking the length of the longest sentences actually present in it, not from a general rule of thumb alone.

Q7. How would you detect that a chunk is lexically on-topic but missing context that changes its actual meaning? This is hard to detect from the retrieval score alone, since a lexically similar chunk scores well regardless of whether its meaning is complete — it requires either a downstream faithfulness check on the generated answer (verifying the answer's claims are actually supported by the retrieved chunk, not just topically related to it) or a chunking strategy with enough overlap and small enough chunk size that qualifying clauses stay attached to the sentences they modify.

Q8. Design a way to validate a new similarity threshold before it ships to production. Build a small, held-out set of query/answer pairs with known correct source passages, run retrieval with the new model or scoring function, and check two things separately: whether the correct passage is still the top-ranked (or near-top-ranked) result, and what score range the correct matches fall into versus the score range for clearly irrelevant chunks. The new threshold should sit in the gap between those two ranges for the new scoring function — never reused from the old function without checking that gap first.

---

8. When to use / tradeoffs

Reach for chunk-size/overlap tuning and per-model threshold tuning when:

• building any retrieval-augmented system over a real document corpus that doesn't fit in a single prompt
• migrating or A/B testing between embedding models or providers
• diagnosing a retrieval system that "used to work and now doesn't" after any change upstream of the retrieval step

Honest limits. This entire design assumes a similarity score is a reasonable proxy for "this chunk answers the question," which is false for negations, comparisons across multiple chunks, and answers the source implies but never states directly — no amount of chunk-size or threshold tuning fixes a problem that is actually a reasoning-strategy problem. It also assumes chunk boundaries drawn on word count roughly align with meaningful units of the source, which is false for tables and code. And overlap is a real, ongoing cost — proportional to the overlap fraction — not a free improvement, so more overlap than the corpus's longest meaningful unit requires is pure waste.

---

9. Summary + related articles

• A retrieval system chunks documents because it can only compare a question against pieces small enough to score meaningfully, not a whole corpus at once.
• Chunk size trades context (large chunks) against precision (small chunks); overlap exists specifically to stop a chunk boundary from splitting the one sentence that matters.
• A similarity score is a property of the specific scoring function or embedding model that produced it — never treat a threshold as portable across models without re-checking the score distribution first.
• Boundary condition: this approach assumes similarity tracks relevance and that word-count chunk boundaries roughly align with meaningful units of the source; it breaks for multi-hop reasoning, negations, and structured data like tables or code.

Related:

• Hallucination Detection & Grounding — once retrieval returns a chunk, this is how you check whether the generated answer is actually faithful to what that chunk says, rather than just topically related to it
• Memory Management — for a system that needs to remember what was retrieved across a long-running, multi-step session rather than a single one-shot query

Resources

• Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158. https://aclanthology.org/2024.eacl-demo.16/
• Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 338–354. https://aclanthology.org/2024.naacl-long.20/