Semantic Chunking: Letting the Boundary Follow the Topic

TL;DR — Fixed-size chunking cuts at a word or token count, which lands mid-topic roughly whenever a topic does not happen to be that length. Semantic chunking instead embeds each sentence, measures similarity between neighbours, and starts a new chunk where that similarity drops — so the boundary follows the subject rather than the ruler. It costs one embedding call per sentence instead of one per chunk, and it introduces a threshold that is corpus-specific and genuinely brittle: in the measured run below, the same document yields 2 chunks at a 0.10 threshold and 4 at 0.30. The boundary condition worth knowing before you adopt it is that the largest similarity drop is not necessarily the topic change — an unusual sentence with little vocabulary in common with its neighbour outscores a real change of subject, so the method finds lexical seams, not meaning.

---

1. Simple explanation

Fixed-size chunking is a ruler. Every 400 words, cut. It knows nothing about what it is cutting through, so sooner or later it cuts a paragraph in half, or separates a claim from the number that supports it, and whichever half gets retrieved is missing the other.

Semantic chunking replaces the ruler with a question asked at every sentence: does this still sound like what came before? Embed each sentence, compare each one with its predecessor, and when the resemblance falls off a cliff, put the boundary there. Topics end up whole, whatever their length.

The cost is that you now embed every sentence rather than every chunk — an order of magnitude more calls — and you have acquired a threshold. "Falls off a cliff" needs a number, that number differs by corpus, and picking it badly is worse than not having tried: too high and every sentence becomes its own chunk, too low and the whole document stays one.

Analogy — chapter breaks versus page breaks. A page break happens every 300 words because that is how much fits, and lands wherever it lands — mid-sentence, mid-argument. A chapter break happens where the subject changes. Nobody would index a book by page and expect coherent excerpts, but they would by chapter. Semantic chunking is trying to find the chapter breaks in a document that never marked any — and the honest caveat is that it finds them by style discontinuity, which usually coincides with a change of subject and sometimes just means the author wrote one unusual sentence.

---

2. Diagram

  FIXED-SIZE: cut every 3 sentences         SEMANTIC: cut where similarity drops

  s0 rice, pan, simmer      ]              s0 rice, pan, simmer      ]
  s1 rice, pan, stir         } chunk 1     s1 rice, pan, stir         }
  s2 rice, pan, liquid      ]              s2 rice, pan, liquid       } chunk 1
                                           s3 rice, liquid, grains   ]
  s3 rice, liquid, grains   ]                      ---- 0.218 ----   <- boundary
  s4 oil, sunlight           } chunk 2     s4 oil, sunlight          ]
  s5 oil, oxidation         ]              s5 oil, oxidation          } chunk 2
                                           s6 oil, bottle             }
  s6 oil, bottle            ]              s7 oil, smell             ]
  s7 oil, smell              } chunk 3
                            ]              the topic survives intact


  CHUNK 2 OF THE FIXED SPLIT IS THE PROBLEM

    "The rice is done when the liquid has gone ... Store the oil away
     from direct sunlight ... Oxidation gives stored oil a stale smell."
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     end of topic A                  start of topic B
     retrieved for a rice question -> two thirds of it is about oil


  MEASURED SIMILARITY BETWEEN NEIGHBOURS (lexical overlap, §5)

    0->1  0.585  #######################
    1->2  0.585  #######################
    2->3  0.723  ############################
    3->4  0.218  ########                     <- TRUE topic shift
    4->5  0.335  #############
    5->6  0.224  ########
    6->7  0.000                               <- BIGGEST drop, and NOT a topic shift

  the deepest trough is not the seam you wanted.

---

3. How it works

3.1 The algorithm

Four steps, and only the third involves a decision:

1. Split the document into sentences.
2. Embed each sentence independently.
3. For each adjacent pair, compute cosine similarity. Where it falls below a threshold, place a boundary.
4. Join the sentences between boundaries into chunks, usually with a minimum and maximum size enforced afterwards so pathological output is clipped.

Variants differ mainly in step 3. Comparing a sentence to a running centroid of the current chunk rather than to its immediate predecessor is more stable against a single odd sentence. Comparing windows of three sentences rather than one smooths the signal further. Choosing the threshold as a percentile of the observed similarity distribution, rather than a fixed constant, adapts it per document — which trades a brittle global constant for a brittle assumption that every document contains roughly the same proportion of boundaries.

3.2 What it costs

The last row of the middle block is the one that catches teams out. Chunk boundaries become a function of the embedding model, so upgrading the model silently re-cuts the corpus. Two indexes built two months apart are no longer comparable, and a retrieval regression traced back to "we changed the embedding model" is usually really "we changed where the chunks start".

3.3 The threshold is the whole method, and it is brittle

The measured run in §5 uses one short document and two thresholds:

  threshold 0.10  ->  2 chunks   (only the 0.000 link breaks; split in the wrong place)
  threshold 0.30  ->  4 chunks   (finds the real boundary, plus two spurious ones)

Both are defensible-looking numbers and neither gives the intended two-topic split. The right answer for this document sits between 0.218 and 0.224 — a window six thousandths wide, which no one would find except by already knowing the answer.

This is not an argument against the method; it is the reason it needs calibrating on a sample of the actual corpus before use, and re-calibrating when the corpus or the model changes. A threshold copied from a blog post is a random number.

3.4 The failure that matters: seams are not subjects

The most useful thing in the measured output is the last line. The largest similarity drop in the document is 6->7 at 0.000 — and it is not a topic change. Both sentences are about storing oil. They simply share no vocabulary:

  s6  "An opaque bottle slows oxidation more than a clear one."
  s7  "Discard stored oil once the smell turns sharp."

Meanwhile the genuine topic change, rice to oil, scores 0.218 — higher, therefore less of a break by this measure.

So an algorithm that cuts at the deepest trough cuts in the wrong place, and an algorithm that cuts at every trough below a threshold cuts in the right place and two wrong ones. The method detects lexical or stylistic discontinuity, which correlates with topic change and is not the same thing. A trained sentence encoder narrows this gap considerably — it has a notion of oil storage that survives a change of vocabulary — but it does not close it, and the failure mode is the same shape: one unusual sentence produces a boundary nobody wanted.

Where this stops applying. Semantic chunking assumes the document is prose whose topics run in contiguous spans. It has nothing to offer a table, where every row is lexically similar to every other and the correct boundary is structural rather than semantic — a table wants to be one atomic chunk, and a similarity-based splitter will happily cut it between rows. It also degrades on documents with headings, because a heading is a short, vocabulary-poor sentence that reads as a boundary to the algorithm and as a boundary to a much cheaper structural rule that was available for free.

---

4. The math

4.1 Boundary rule

For sentences s_0 .. s_{n-1} with embeddings e_i, and threshold tau:

  sim(i) = cos(e_{i-1}, e_i) = (e_{i-1} . e_i) / (|e_{i-1}| |e_i|)

  boundary before s_i   iff   sim(i) < tau

  chunk count = 1 + |{ i in 1..n-1 : sim(i) < tau }|

Chunk count is a monotone non-decreasing step function of tau. At tau = 0 there is one chunk; at tau = 1 there are n. Every distinct value of sim is a step, so a corpus with tightly clustered similarities has a cliff-edge response to the threshold — exactly the case in §4.3.

4.2 Centroid variant

Comparing against the running chunk instead of the previous sentence resists a single odd sentence:

  c = mean(e_j for s_j currently in the chunk)
  boundary before s_i   iff   cos(c, e_i) < tau

This trades sensitivity for stability: a genuine topic change that arrives gradually over three sentences is less likely to be caught, because the centroid drifts with it.

4.3 Worked example

Eight sentences, two topics, boundary between s_3 and s_4. Measured similarities from §5:

  i      1      2      3      4      5      6      7
  sim  0.585  0.585  0.723  0.218  0.335  0.224  0.000
                            ^^^^^                ^^^^^
                            true shift           biggest drop

Chunk count as a function of tau:

  tau = 0.10   ->  breaks at i=7            ->  2 chunks   (wrong place)
  tau = 0.22   ->  breaks at i=4, 7         ->  3 chunks
  tau = 0.30   ->  breaks at i=4, 6, 7      ->  4 chunks   (right + 2 spurious)
  tau = 0.50   ->  breaks at i=4,5,6,7      ->  5 chunks
  tau = 0.60   ->  breaks at i=1..7 except 3 -> 7 chunks

The interval that produces exactly the intended split is 0.218 <= tau < 0.224:

  width = 0.224 - 0.218 = 0.006

A six-thousandth-wide window on an eight-sentence document. That number is the honest case against treating the threshold as a constant, and the case for calibrating it per corpus as a percentile of the observed distribution.

---

5. Real code

The block below uses a bag-of-words hashing encoder so it runs anywhere with no dependencies. A production implementation replaces embed() with a trained sentence encoder — sentence-transformers with a model such as all-MiniLM-L6-v2, or a hosted embedding endpoint — and nothing else in the algorithm changes. The consequence of the substitution is discussed in §3.4: a trained encoder narrows the gap between lexical seam and topic change without closing it.

"""Semantic chunking: let the boundary follow the topic, not the word count."""
import hashlib, math, re

DOC = (
 "Bring the pan to a steady simmer before adding the rice. "
 "Stir the rice once, then leave the pan alone. "
 "Cover the pan and let the rice absorb the liquid slowly. "
 "The rice is done when the liquid has gone and the grains separate. "
 "Store the oil away from direct sunlight to slow oxidation. "
 "Oxidation gives stored oil a stale, cardboard smell. "
 "An opaque bottle slows oxidation more than a clear one. "
 "Discard stored oil once the smell turns sharp."
)
TOPIC_SHIFT = 4          # sentence 4 starts the second topic

def sentences(text):
    return [s.strip() for s in re.split(r"(?<=\.)\s+", text.strip()) if s.strip()]

def embed(text, dim=128):
    v = [0.0] * dim
    for tok in re.findall(r"[a-z]+", text.lower()):
        h = int(hashlib.sha256(tok.encode()).hexdigest(), 16)
        v[h % dim] += 1.0
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

def cos(a, b):
    return sum(x * y for x, y in zip(a, b))

def fixed_chunks(sents, per_chunk=3):
    return [" ".join(sents[i:i + per_chunk]) for i in range(0, len(sents), per_chunk)]

def semantic_chunks(sents, threshold):
    """Start a new chunk when adjacent sentences stop resembling each other."""
    e = [embed(s) for s in sents]
    chunks, cur = [], [sents[0]]
    for i in range(1, len(sents)):
        (cur.append(sents[i]) if cos(e[i-1], e[i]) >= threshold
         else (chunks.append(" ".join(cur)), cur.clear(), cur.append(sents[i])))
    chunks.append(" ".join(cur))
    return chunks

sents = sentences(DOC)
e = [embed(s) for s in sents]
links = [(i, cos(e[i-1], e[i])) for i in range(1, len(sents))]

print(f"{len(sents)} sentences. Topic shifts at sentence {TOPIC_SHIFT}.\n")
print("adjacent-sentence similarity (lexical overlap):")
for i, s in links:
    mark = "  <- TRUE topic shift" if i == TOPIC_SHIFT else ""
    print(f"  {i-1}->{i}  {s:.3f}  {'#' * int(s * 40):<29}{mark}")

print(f"\nfixed, 3 sentences/chunk -> {len(fixed_chunks(sents))} chunks")
for c in fixed_chunks(sents):
    print("   |", c[:70], "...")

for t in (0.10, 0.30):
    ch = semantic_chunks(sents, t)
    print(f"\nsemantic, threshold={t:.2f} -> {len(ch)} chunks")
    for c in ch:
        print("   |", c[:70], "...")

weakest = min(links, key=lambda p: p[1])
shift = dict(links)[TOPIC_SHIFT]
print(f"\ntopic-shift link 3->4 = {shift:.3f}")
print(f"weakest link overall  = {weakest[0]-1}->{weakest[0]} at {weakest[1]:.3f}")

assert shift < 0.30, "the real topic shift is detectable at a 0.30 threshold"
assert weakest[0] != TOPIC_SHIFT, (
    "the weakest link is NOT the topic shift - low vocabulary overlap "
    "is not the same thing as a change of subject")

# Output:
#   8 sentences. Topic shifts at sentence 4.
#
#   adjacent-sentence similarity (lexical overlap):
#     0->1  0.585  #######################
#     1->2  0.585  #######################
#     2->3  0.723  ############################
#     3->4  0.218  ########                       <- TRUE topic shift
#     4->5  0.335  #############
#     5->6  0.224  ########
#     6->7  0.000
#
#   fixed, 3 sentences/chunk -> 3 chunks
#      | Bring the pan to a steady simmer before adding the rice. Stir the rice ...
#      | The rice is done when the liquid has gone and the grains separate. Sto ...
#      | An opaque bottle slows oxidation more than a clear one. Discard stored ...
#
#   semantic, threshold=0.10 -> 2 chunks
#      | Bring the pan to a steady simmer before adding the rice. Stir the rice ...
#      | Discard stored oil once the smell turns sharp. ...
#
#   semantic, threshold=0.30 -> 4 chunks
#      | Bring the pan to a steady simmer before adding the rice. Stir the rice ...
#      | Store the oil away from direct sunlight to slow oxidation. Oxidation g ...
#      | An opaque bottle slows oxidation more than a clear one. ...
#      | Discard stored oil once the smell turns sharp. ...
#
#   topic-shift link 3->4 = 0.218
#   weakest link overall  = 6->7 at 0.000

The second assert is the article's most important claim in executable form: the weakest link is not the topic shift. If a future edit to the document or the encoder made that accidentally true, the block would fail rather than quietly support a conclusion the data no longer shows.

---

6. Real-world example

A documentation team moved a knowledge base of several thousand support articles from fixed 400-word chunking to semantic chunking. On a sample of thirty questions the new index was clearly better: answers stopped being truncated mid-procedure, and the team shipped it.

Two problems emerged over the following month.

The first was cost, and it was predictable in hindsight. Fixed chunking had made roughly 40,000 embedding calls to ingest the corpus. Semantic chunking embedded every sentence: about 600,000. A one-off ingest cost went up by an order of magnitude, and because the pipeline re-ingested changed articles nightly, so did the recurring cost. Nothing broke; the bill simply moved, and it moved in a line item nobody had flagged as sensitive to a chunking change.

The second was worse and took longer to see. The threshold had been tuned on a sample of long, well-structured product guides. The knowledge base also contained short release notes — three or four terse sentences each, with almost no shared vocabulary between them. Against that threshold, nearly every sentence of a release note became its own chunk. Retrieval on release-note questions degraded sharply, because a single-sentence chunk has too little context to match a question and too little content to answer one. The corpus average looked fine; one document class had been quietly destroyed.

The fix was not a better threshold. It was routing: long prose went to the semantic chunker, short structured documents were kept whole, and tables were made atomic. The threshold itself was then set as a percentile of each document's own similarity distribution rather than as a global constant, which removed the class-sensitivity without requiring anyone to tune per document.

---

7. Interview questions companies actually ask

Q1. What problem does semantic chunking solve that overlap does not? Overlap mitigates a bad boundary; it does not move it. If a 400-word cut lands mid-procedure, a 50-word overlap means both chunks contain a little of the other side, and neither contains the whole procedure — you have paid for duplicate storage and still cannot retrieve the complete unit. Semantic chunking puts the boundary where the subject changes, so the unit stays intact and overlap becomes a much smaller safety margin rather than the primary defence.

Q2. What does semantic chunking cost, in the terms that actually matter? One embedding call per sentence at ingest instead of one per chunk — in practice an order of magnitude more, and recurring if you re-ingest changed documents. It also adds a threshold and a comparison window to the configuration surface, and it makes chunk boundaries a function of the embedding model, which means a model upgrade silently re-cuts the entire corpus and invalidates comparisons against any index built before it.

Q3. How do you pick the threshold? Not from a blog post. Calibrate it on a sample of the real corpus: compute the distribution of adjacent-sentence similarities, and set the threshold as a percentile of that distribution rather than as a global constant, so it adapts per document. Then check it against documents of each class you actually hold — the failure mode is not a bad average but one document type, typically short and terse, where the threshold produces single-sentence chunks.

Q4. Is the biggest similarity drop always the topic boundary? No, and this is the method's main honest limitation. The measure detects lexical or stylistic discontinuity, which correlates with topic change without being it. In the worked example, the deepest trough is between two sentences that are both about the same subject and simply share no vocabulary, while the real rice-to-oil transition scores higher. A trained sentence encoder narrows that gap substantially, because it has a representation of the subject that survives a change of wording, but the failure shape persists: one unusual sentence yields a boundary nobody wanted.

Q5. Would you use semantic chunking on a document full of tables? No. Every row of a table is lexically similar to every other row, so the similarity signal is flat and the algorithm will happily cut between rows — destroying exactly the structure that made the table meaningful. Tables want a structural rule: treat the table as one atomic chunk, and repeat the header if it must be split. Semantic chunking is for prose whose topics run in contiguous spans.

Q6. Your document has headings. Does semantic chunking still earn its cost? Usually not. A heading is an explicit, free, author-supplied boundary marker, and splitting on headings first captures most of the benefit at no embedding cost. Headings also confuse the algorithm, since a short vocabulary-poor line reads as a discontinuity in its own right. The sensible order is structural first — headings, then tables as atomic units — and semantic chunking only within the long prose sections that remain.

Q7. How do you know the change helped? Not by inspecting chunks, which is the trap. Boundaries look better under semantic chunking almost by definition, since that is what was optimised. Score it on retrieval: build a set of questions whose answers live at a known place in the corpus, and measure whether the chunk containing the answer is retrieved. And evaluate per document class, not on the average — the realistic outcome is an improvement on long prose and a regression on short documents, which an average will hide.

---

8. When to use / tradeoffs

Reach for semantic chunking when:

• documents are long-form prose with topics running in contiguous spans
• there are no reliable headings to split on for free
• fixed-size chunking is demonstrably cutting retrievable units in half
• you can afford one embedding call per sentence at ingest, and again on every re-ingest
• you are able to calibrate the threshold on the real corpus and re-check it per document class

Honest limits. The measured similarities in §5 come from a bag-of-words hashing encoder, which makes the lexical seam versus topic change failure especially stark; a trained sentence encoder would score the rice-to-oil transition lower and the vocabulary-poor pair higher, narrowing but not eliminating the gap. The eight-sentence document is deliberately small enough to reason about by hand, and the six-thousandth-wide threshold window it produces is more extreme than a long document would show — on realistic text the usable range is wider, though still corpus-specific. The method assumes sentence segmentation is reliable, which it is not for text containing abbreviations, decimal numbers, or citation markers, and a mis-split sentence corrupts two similarity measurements at once. And none of this addresses whether chunk size is appropriate for the embedding model's input limit; semantic boundaries can produce a chunk far longer than the model accepts, so a size clamp is still required afterwards.

---

9. Summary + related articles

• Fixed-size chunking cuts at a count and lands mid-topic; semantic chunking cuts where adjacent-sentence similarity drops, so topics stay whole.
• The cost is one embedding call per sentence instead of per chunk, plus a threshold and a comparison window.
• Boundaries become a function of the embedding model, so a model upgrade re-cuts the corpus and breaks comparability with older indexes.
• The threshold is brittle: the same document gives 2 chunks at 0.10 and 4 at 0.30, and the window producing the intended split is 0.006 wide.
• The largest similarity drop is not the topic boundary. The method finds lexical and stylistic seams, which correlate with subject changes without being them.
• Calibrate the threshold as a percentile of the corpus's own similarity distribution, and check it per document class — the realistic failure is one class destroyed while the average looks fine.
• Do structural splitting first: headings are free boundaries, and tables must stay atomic.
• Evaluate on retrieval of known answers, per document class. Inspecting boundaries only confirms what was optimised.

Related:

• Chunking Strategies — the full menu this article is one option from
• Token-Aware Chunking and Reproducible Index Builds — the size clamp semantic boundaries still need, and why a chunk budget in words is the wrong unit
• Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — tuning the parameters this method adds to
• Tables in RAG: Why Flattening Breaks the Answer, Not the Search — why a table must be exempt from any similarity-based splitter
• Document Processing: What You Index Sets the Ceiling — the structural signals worth using before paying for embeddings
• Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — what changes downstream when boundaries move
• RAG Evaluation: Attributing Failure and Sizing the Eval Set — building the per-class retrieval test this article's §7 Q7 argues for
• RAG Cost Optimization: Find the Step That Runs Forty Times — finding the ingest step that runs once per sentence

Resources

• Hearst, Marti A. TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages. Computational Linguistics, 23(1), 1997. The original lexical-cohesion approach to topic segmentation; the similarity-trough idea in §3.1 predates embeddings by decades.
• Beeferman, Doug & Berger, Adam & Lafferty, John. Statistical Models for Text Segmentation. Machine Learning, 34, 1999. Introduces the P_k segmentation metric, which is the right way to score a boundary-placement algorithm against reference segmentation.
• Pevzner, Lev & Hearst, Marti A. A Critique and Improvement of an Evaluation Metric for Text Segmentation. Computational Linguistics, 28(1), 2002. The WindowDiff metric and a careful account of why near-miss boundaries should not be scored as total failures.
• Reimers, Nils & Gurevych, Iryna. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019. The sentence-encoder family that replaces embed() in a production implementation: https://www.sbert.net
• LangChain — SemanticChunker, including its percentile, standard-deviation, and interquartile threshold strategies, which are the adaptive alternatives to a global constant discussed in §3.1: https://python.langchain.com/docs/how_to/semantic-chunker/