Turn-Taking in Voice Agents: Endpointing, VAD and Barge-In

TL;DR — Before a voice agent can answer, something must decide the caller has finished speaking. Silence is ambiguous — a mid-sentence hesitation is acoustically identical to the end of a turn — so the usual solution is a fixed timer: commit after N milliseconds of quiet. That timer is an unconditional floor under every reply, and it is a dial with a cliff at each end: too short and the agent talks over people who pause to think, too long and callers assume the line dropped. The measured result below is that a safe fixed threshold is an expensive one — the lowest value that never cuts anyone off charges ~700 ms to every caller, about 3.5x the latency a completeness-aware detector needs for the same safety. It stops working entirely when the caller's speech rhythm is not what you tuned against, which is why code-mixed and disfluent speakers get the worst experience from a system that tests fine in the office.

---

1. Simple explanation

Two people on a phone call take turns, and they manage it almost perfectly without discussing it. They use grammar (a finished sentence sounds finished), pitch (a falling tone means "your turn"), and content (a question invites an answer). A machine listening to raw audio has none of that by default. It has volume over time.

So the standard approach is a stopwatch. When the caller stops making sound, start counting. If they are still quiet after some threshold, assume they are done and start answering.

The problem is that people pause in the middle of sentences constantly — to think, to remember a date, to count. "I'd like an appointment for… um… the fourteenth." There are two silences there and only the last one means "your turn." A stopwatch cannot tell them apart, because in the audio they are the same thing.

Analogy — a colleague on a bad video call. You have all learned to over-wait, because the lag means you cannot tell a thinking pause from a finished thought. Wait too little and you both talk at once and have to restart. Wait too much and there is an awkward gap, and eventually someone says "you go." That is the whole problem: the cost of guessing early is a collision, the cost of guessing late is dead air, and you cannot get both from a single fixed waiting time. Humans escape it by listening to grammar and pitch, not just silence — which is exactly the upgrade path for a machine.

---

2. Diagram

THE AMBIGUITY  —  identical audio, opposite meanings

  "an appointment for ......... the fourteenth" ......... [done]
                     ^^^^^^^^^                 ^^^^^^^^^
                     480 ms quiet              900 ms quiet
                     MID-SENTENCE              END OF TURN
                     (do not answer)           (answer now)


FIXED TIMER  —  one threshold for every caller

  min_delay = 200 ms
     "an appointment for ..|.. the fourteenth"
                           ▲ FIRES HERE — agent talks over the caller ✗

  min_delay = 705 ms   (the lowest value that is safe for ALL callers)
     "an appointment for ......... the fourteenth" .......|
                                                          ▲ fires, correct
                                                          but +705 ms for
                                                          EVERY caller ✗


SEMANTIC  —  ask whether the sentence sounds finished

  "an appointment for"        -> incomplete -> wait long  (800 ms) -> no fire ✓
  "an appointment for the 14th" -> complete -> wait short (200 ms) -> fire   ✓

     zero cut-ins, and 200 ms instead of 705 ms


WHY THE WAIT HURTS TWICE  —  idle vs overlapped

  NOT overlapped   |░░ 350 ms doing nothing ░░|── model ──|── speak
                    ▲ pure loss

  overlapped       |░░ 350 ms ░░|── model ──|── speak
                   |─ model already running on the partial transcript ─|
                    ▲ same safety, 350 ms reclaimed

---

3. How it works

3.1 Three different jobs, often confused

Voice activity detection (VAD) answers "is there speech in this audio frame right now?" It is a small, fast, local classifier — a few milliseconds on CPU — and it produces a stream of speech/no-speech flags. It does not know about turns.

Endpointing answers "has this turn finished?" It consumes VAD flags, or transcript events, or both, and applies a policy — usually a silence threshold.

Interruption detection (barge-in) answers "is the caller talking while we are talking?" and if so, whether to stop speaking. Same inputs, different decision, and it needs extra care because the agent's own audio can leak back through the line and be detected as the caller.

These are separate concerns and get wired together wrongly all the time. A configuration that requests VAD-based interruption without actually loading a VAD is a real and common bug: it either silently degrades to a different strategy or never fires, and the symptom is "the agent won't let me interrupt" — which looks like a model problem and is not.

3.2 Where the timer starts matters as much as its length

A local VAD detects speech-stop in roughly 10–30 ms because it runs on your own audio frames. Without one, endpointing has to wait for the transcription service to tell you the turn looks finished, which is a network round trip. The threshold is the same number in both cases, but in the second the clock starts late, so the whole budget shifts right by the round-trip time. Two systems with identical min_delay can differ by 150 ms for this reason alone.

3.3 The two thresholds, and which one is the trap

Most endpointing configurations expose a pair:

• min_delay — the shortest silence that counts as end-of-turn. This is a floor under every single reply.
• max_delay — the longest the system will wait when it is unsure, before committing anyway.

min_delay gets all the attention and max_delay causes the incidents. A caller experiences max_delay as a dropped call — two seconds of total silence on a phone line and people say "hello? are you there?" and start repeating themselves, at which point you have two speakers and a mess.

max_delay also fires far more often than teams expect, because the "unsure" condition correlates with exactly the callers you most want to serve well. A transcriber configured for automatic language detection or code-mixed speech is less confident about whether an utterance is complete, so those turns drift toward the ceiling. The people who get the worst experience are the ones speaking the way the system was specifically configured to accept.

3.4 Barge-in, and the false-interruption problem

Letting callers interrupt is table stakes — it is how humans repair a misunderstanding without waiting. But a naive implementation stops speaking on any detected sound, which on a phone line means it stops for a cough, a car horn, or the caller's own "mm-hm" of agreement.

The usual controls:

That last one is worth dwelling on. If a false interruption is detected and you resume the interrupted answer, but the caller has meanwhile asked something new, the agent now delivers a stale reply on top of a fresh question. Leaving resume off means the occasional truncated sentence, which callers barely notice, instead of an answer to a question nobody asked, which they notice immediately.

3.5 Overlapping the wait instead of shortening it

The subtle point: the cost of endpointing is not the waiting, it is the idle waiting. During that silence the system usually does nothing at all, then starts working once the timer expires.

You can instead begin generating on the partial transcript while the timer runs, and discard the work if the caller resumes. You keep the full safety margin and get the time back. The trade is that some generations are thrown away — which is a real cost with an expensive model and a rounding error with a cheap one, so this optimisation and model-tier choice are coupled decisions rather than independent ones.

3.6 Where this stops working

All of the above assumes a two-party, turn-based call. It describes nothing useful about conference calls with several speakers, simultaneous interpretation, or agents designed to speak over the user. It also assumes the caller is trying to have an orderly conversation — abusive callers, hold music, IVR trees, and voicemail greetings all defeat silence-based turn detection and need to be handled as their own explicit cases rather than tuned around.

---

4. The math

4.1 The two error types

Endpointing is a binary decision on every silence, so it has the usual two failure modes:

  cut-in     : fired during a hesitation      -> agent talks over the caller
  dead air   : waited longer than necessary   -> caller thinks the line dropped

  for a fixed threshold d, over a set of pauses P:
      cut_ins(d)   = |{ p in P : p >= d  AND  p is not a turn boundary }|
      latency(d)   = d                        (paid on EVERY correct turn)

Note the asymmetry: cut_ins depends on the distribution of hesitation lengths, which varies per caller, while latency is paid uniformly. Lowering d trades a certain, universal cost for a probabilistic, per-caller one.

4.2 Why a single threshold cannot win

A fixed threshold d is safe only if it exceeds every hesitation pause in your caller population:

  d_safe = max(hesitation pauses)  +  epsilon

Since a thinking pause and a turn-final pause are drawn from overlapping distributions, d_safe is close to the longest turn-final pause too — so being safe means being about as slow as the slowest caller, for everyone.

A completeness-aware policy escapes this by using two thresholds selected by a signal that is not silence:

  d = d_short  if transcript looks grammatically complete
      d_long   otherwise

  cut-ins go to zero when d_long > max(hesitation after INCOMPLETE text)
  and latency is d_short on the common case

4.3 Worked example

Five callers, with real hesitation patterns — a fluent one, a hesitant one, a very slow one counting out loud, a code-mixed one where the transcriber is unsure, and a one-word confirmation. Sweeping a fixed threshold across them:

   min_delay   cut-ins   avg added latency
      200 ms         3             200 ms
      300 ms         3             300 ms
      350 ms         3             350 ms
      500 ms         2             500 ms
      700 ms         1             700 ms
     1000 ms         0            1000 ms

Read the two ends. At 200 ms, three of five callers get talked over. At 700 ms — a setting that feels generous — the slowest caller is still cut off, because one of their thinking pauses is exactly 700 ms. Searching at 5 ms resolution, the lowest threshold with zero cut-ins is 705 ms, and it charges that to all five callers including the one who said a single word.

Now the completeness-aware version, 200 ms when the transcript parses as finished and 800 ms when it does not:

   200/800 ms         0             200 ms

Zero cut-ins at 200 ms of added latency — about 3.5x less than the cheapest safe fixed timer. The per-caller detail shows why: the hesitant, slow, and code-mixed callers all pause after text that is visibly incomplete ("an appointment for"), so the long threshold applies exactly where it is needed and nowhere else.

Note how sensitive the safe threshold is to a single caller. It is 705 ms because one person paused for 700 ms mid-sentence. Add one more deliberate speaker and it moves again — which is the real argument against tuning a constant: you are fitting to the slowest example you happen to have collected.

---

5. Real code

"""Fixed-timer endpointing vs semantic endpointing on a set of caller utterances.

Each utterance is a list of (spoken_ms, following_pause_ms, text_so_far_is_complete).
The final pause is the real end of turn; earlier pauses are hesitations.
"""

Utterance = list[tuple[int, int, bool]]

CALLS: dict[str, Utterance] = {
    # fluent: one clean pause at the end
    "fluent":      [(1400, 900, True)],
    # hesitant mid-sentence pause -- the classic false cut-in
    "hesitant":    [(700, 480, False), (900, 900, True)],
    # counting people out loud, two long thinking pauses
    "very slow":   [(600, 620, False), (400, 700, False), (500, 900, True)],
    # code-mixed, transcriber unsure, so the pause reads as longer
    "code-mixed":  [(1100, 520, False), (600, 900, True)],
    # short confirmation, ends fast
    "one word":    [(300, 900, True)],
}


def fixed_timer(utt: Utterance, min_delay_ms: int) -> tuple[int, int]:
    """Return (cut_ins, added_latency_ms). Fires on the first pause >= min_delay."""
    for i, (_spoken, pause, _complete) in enumerate(utt):
        if pause >= min_delay_ms:
            is_last = i == len(utt) - 1
            return (0, min_delay_ms) if is_last else (1, 0)
    return 0, min_delay_ms


def semantic(utt: Utterance, short_ms: int, long_ms: int) -> tuple[int, int]:
    """Wait `short_ms` when the transcript looks complete, `long_ms` when it does not."""
    for i, (_spoken, pause, complete) in enumerate(utt):
        need = short_ms if complete else long_ms
        if pause >= need:
            is_last = i == len(utt) - 1
            return (0, need) if is_last else (1, 0)
    return 0, long_ms


print("FIXED TIMER — one delay for every caller")
print(f"  {'min_delay':>9} {'cut-ins':>8} {'avg added latency':>18}")
rows = {}
for d in (200, 300, 350, 500, 700, 1000):
    cuts = sum(fixed_timer(u, d)[0] for u in CALLS.values())
    lats = [fixed_timer(u, d)[1] for u in CALLS.values() if fixed_timer(u, d)[0] == 0]
    avg = sum(lats) / len(lats) if lats else 0.0
    rows[d] = (cuts, avg)
    print(f"  {d:>7} ms {cuts:>8} {avg:>15.0f} ms")

print("\nSEMANTIC — short wait when the sentence is complete, long wait when not")
sem_cuts = sum(semantic(u, 200, 800)[0] for u in CALLS.values())
sem_lats = [semantic(u, 200, 800)[1] for u in CALLS.values()]
sem_avg = sum(sem_lats) / len(sem_lats)
print(f"  {'200/800 ms':>10} {sem_cuts:>7} {sem_avg:>15.0f} ms")

print("\nPer-call detail at the two interesting settings:")
print(f"  {'call':<12} {'fixed 200':>12} {'fixed 700':>12} {'semantic':>12}")
for name, u in CALLS.items():
    f2, f7, sm = fixed_timer(u, 200), fixed_timer(u, 700), semantic(u, 200, 800)
    fmt = lambda r: "CUT IN" if r[0] else f"+{r[1]}ms"
    print(f"  {name:<12} {fmt(f2):>12} {fmt(f7):>12} {fmt(sm):>12}")

# Search for the lowest fixed threshold that never cuts anyone off, rather than
# reading it off the sampled grid above.
first_safe = next(
    d for d in range(100, 1501, 5)
    if sum(fixed_timer(u, d)[0] for u in CALLS.values()) == 0
)

# A fast fixed timer cuts callers off ...
assert rows[200][0] == 3, rows[200]
assert rows[350][0] == 3, rows[350]
# ... and even a slow one still cuts off the slowest caller.
assert rows[700][0] == 1, rows[700]
# The cheapest safe fixed threshold is well above any "snappy" setting.
assert first_safe == 705, first_safe
# Semantic endpointing reaches the same safety at a fraction of the latency.
assert sem_cuts == 0
assert sem_avg == 200
print(f"\nlowest SAFE fixed threshold : {first_safe} ms (paid by every caller)")
print(f"semantic, same 0 cut-ins    : {sem_avg:.0f} ms")
print(f"                            : {first_safe / sem_avg:.1f}x less added latency")
print("all assertions passed")

# Output:
#   FIXED TIMER — one delay for every caller
#     min_delay  cut-ins  avg added latency
#         200 ms        3             200 ms
#         300 ms        3             300 ms
#         350 ms        3             350 ms
#         500 ms        2             500 ms
#         700 ms        1             700 ms
#        1000 ms        0            1000 ms
#
#   SEMANTIC — short wait when the sentence is complete, long wait when not
#     200/800 ms       0             200 ms
#
#   Per-call detail at the two interesting settings:
#     call            fixed 200    fixed 700     semantic
#     fluent             +200ms       +700ms       +200ms
#     hesitant           CUT IN       +700ms       +200ms
#     very slow          CUT IN       CUT IN       +200ms
#     code-mixed         CUT IN       +700ms       +200ms
#     one word           +200ms       +700ms       +200ms
#
#   lowest SAFE fixed threshold : 705 ms (paid by every caller)
#   semantic, same 0 cut-ins    : 200 ms
#                               : 3.5x less added latency
#   all assertions passed

The is_complete flag stands in for what a real system infers — in production it comes from a small classifier over the partial transcript, and its accuracy is what determines whether the right-hand column survives contact with real callers.

---

6. Real-world example

A pharmacy's repeat-prescription line launched with min_delay set in the low hundreds of milliseconds and a max_delay ceiling several times larger. Internal testing was clean. Within a fortnight the complaint pattern was clear and contradictory: some callers said the system interrupted them constantly, others said it "hung up" mid-conversation. Same configuration, opposite symptoms.

The transcripts separated the two groups cleanly. Callers reading a prescription reference aloud paused between digit groups — "two nine one… three oh…" — and those pauses comfortably exceeded min_delay, so the agent answered before they finished. Meanwhile the transcription service was configured for automatic language detection, and for callers switching between languages mid-sentence it returned lower-confidence partial results for longer. Those turns fell through to the ceiling, which callers read as a dead line.

So one threshold was too aggressive for one population and the ceiling was too generous for another, and both groups were being failed by the same config. Worse, there was no local voice activity detector configured, so the endpointing clock only started once the transcription service reported a pause — pushing every measurement later than the settings implied.

Three changes fixed it. A local VAD, so the clock started on the caller's actual silence rather than a network event. max_delay cut to well under a second, on the reasoning that committing to a wrong guess quickly beats a silence that reads as a disconnection. And a completeness check on the partial transcript, so digit sequences — which never look like finished sentences — got the long threshold while ordinary questions got the short one. The instructive part is that none of it involved the language model, and the team had spent the first week tuning prompts.

---

7. Interview questions companies actually ask

Q1. Why can't you just lower the silence threshold to make the agent feel snappier? Because the threshold is doing two jobs and you only get to optimise one. It sets how fast the agent responds and how tolerant it is of mid-sentence pauses, and those pull in opposite directions. Lowering it reduces latency for every caller by a certain amount while increasing collisions for the subset who hesitate — and hesitation length varies far more between people than turn-final silence does, so you tend to lose more than you gain. The way out is a second signal, usually whether the transcript so far looks grammatically complete.

Q2. What's the difference between VAD and endpointing? VAD is a per-frame classifier answering "is anyone speaking right now," running locally in a few milliseconds. Endpointing is a policy answering "has this turn ended," consuming VAD output or transcript events and applying a threshold. VAD is a detector, endpointing is a decision. Conflating them is how you end up with a config that asks for VAD-based interruption without a VAD loaded.

Q3. The agent stops talking whenever a door slams. How do you fix it? Gate interruptions on more than acoustic energy: require a minimum duration to reject blips, require at least one actually transcribed word rather than just sound, and set a false-interruption timeout so an "interruption" that yields no words is reclassified as noise. Also check for echo — the agent's own output leaking back through the line is a frequent cause, and the fix is acoustic echo cancellation rather than interruption tuning.

Q4. Should the agent resume an answer that was interrupted by noise? Usually not. If the interruption was real, the caller has moved on and resuming delivers a stale answer over a fresh question — the most confusing possible outcome. If it was noise, you lose the tail of one sentence, which callers barely register. Truncation is a much cheaper error than answering a question that was never asked.

Q5. Your p50 latency is 700 ms but p95 is 3 seconds. Where would you look? A bimodal split like that usually means a ceiling is firing on a subset of turns, not that everything got uniformly slower — so check how often the maximum-wait threshold is being hit, and what correlates with it, since low transcriber confidence and unusual speech patterns concentrate there. The other classic cause is contention rather than turn-taking: a synchronous call on an async event loop, or an undersized connection pool, gives you exactly this shape once concurrency rises while single-call testing looks fine.

Q6. How do you measure whether turn-taking is any good? Instrument both error types separately, because they need opposite fixes. Count cut-ins via a proxy — the caller speaking again within a short window of the agent starting, or the agent being interrupted within its first second. Track added latency as the gap from the caller's last audio to the agent's first audio, at p50 and p95. Then slice both by detectable speaker characteristics, since aggregate numbers hide exactly the populations that suffer most.

Q7. Where does turn-taking fit relative to the language model in a latency budget? Before it, and it is a floor the model cannot compensate for. However fast your model is, first-audio latency cannot go below the endpointing threshold plus transcript finalisation plus time-to-first-audio from synthesis. Teams routinely optimise the model while leaving a fixed several-hundred-millisecond wait untouched — and the wait is often reclaimable for free by starting generation on the partial transcript while the timer runs.

---

8. When to use / tradeoffs

Reach for a fixed silence threshold when:

• You are building a first version and need something working today
• Callers are cooperative and speak one predictable way
• The utterances are short and self-contained — confirmations, yes/no, single commands
• You have no way yet to measure cut-ins, so tuning anything smarter is guesswork

Reach for completeness-aware (semantic) endpointing when:

• Callers dictate structured data — numbers, dates, spellings, addresses
• Your population is linguistically mixed, disfluent, elderly, or stressed
• You have both p95 latency and cut-in complaints and cannot fix one without the other
• Latency is a product requirement rather than a preference

Honest limits. The numbers in §4 come from five hand-written utterances chosen to illustrate the mechanism, not from a corpus — the shape of the result (a safe fixed threshold being an expensive one; a completeness signal dominating) is robust and reported widely, but the specific 705 ms and 3.5x are properties of this toy set and yours will differ. The 705 ms figure in particular is set by a single caller's single 700 ms pause, which is exactly how overfitting to a small sample looks. The model also assumes a clean binary completeness signal, and real classifiers are wrong a few percent of the time; a completeness detector that misfires on your domain can be worse than a well-tuned constant, so measure before switching. It ignores network jitter, packet loss, and echo, all of which move real thresholds more than the policy choice does. And it assumes hesitation and turn-final pauses are distinguishable at all — for a caller who trails off without ever finishing, no endpointing strategy is correct, and the right behaviour is a gentle prompt rather than a better timer.

---

9. Summary + related articles

• Silence is ambiguous: a hesitation and a finished turn are acoustically identical, so turn-taking is a decision problem, not a signal-processing one.
• Three distinct jobs get conflated — VAD (is there speech now), endpointing (has the turn ended), barge-in (should we stop talking). Wiring them wrongly produces symptoms that look like model problems.
• min_delay is a floor under every reply. max_delay is the one that causes incidents, because callers experience it as a dropped line — and it fires most for the speakers you configured the system to accommodate.
• Measured: a safe fixed threshold is an expensive one. Zero cut-ins needed 705 ms charged to everyone; a completeness-aware policy got there at 200 ms — and the 705 was set by one caller's one long pause.
• Without a local VAD the clock starts on a network event instead of the caller's silence, shifting the whole budget later at identical settings.
• The cost of endpointing is the idle wait — overlap generation with the timer and you keep the safety and reclaim the time.
• None of this applies to multi-party, simultaneous, or non-conversational audio, and no timer is correct for a caller who never finishes their sentence.

Related:

• Voice Agent Architectures: Cascaded vs Speech-to-Speech — where the endpointing term sits in the end-to-end latency budget
• Reasoning Budgets: When Thinking Tokens Are Waste — why overlapping the wait is cheap on one model tier and expensive on another
• ML Inference Systems — p50 vs p95 and the contention patterns behind bimodal latency
• Production Agents — latency budgets and keeping deterministic decisions in code
• Agent Evaluation — building the scored set needed to compare two endpointing policies honestly
• Streaming Modes in LangGraph — emitting partial output, the mechanism behind overlapping the wait

Resources

• Sacks, Schegloff & Jefferson (1974) — A Simplest Systematics for the Organization of Turn-Taking for Conversation, Language 50(4) — the foundational description of how humans manage turns; the transition-relevance-place idea is what semantic endpointing approximates.
• Jurafsky & Martin — Speech and Language Processing (3rd edition draft), the chapters on Automatic Speech Recognition and on Dialogue Systems — free online: https://web.stanford.edu/~jurafsky/slp3/
• Silero VAD — small permissively licensed voice activity detector, CPU-friendly, the common choice for local speech-stop detection: https://github.com/snakers4/silero-vad
• WebRTC VAD — the long-standing lightweight alternative, widely wrapped for Python: https://webrtc.googlesource.com/src/
• LiveKit Agents — turn detection and interruption configuration, including completeness-aware turn detection: https://docs.livekit.io/agents/
• Pipecat — open-source real-time voice pipeline framework; its interruption handling is readable and worth studying: https://github.com/pipecat-ai/pipecat
• Companion notebook — sweep thresholds against your own pause distributions and plot the cut-in/latency frontier.