TL;DR — A phone-answering agent is built one of two ways: cascaded (speech-to-text → language model → text-to-speech, three services with readable text between them) or speech-to-speech (one model takes audio in and emits audio out). Cascaded is slower per hop but roughly half the cost, fully auditable, and the production default; speech-to-speech wins on latency and loses on price, observability, and control over what the agent is allowed to say. The decision flips on one question: do you need to prove what the agent said and why? Both stop working the same way — if you wait for a complete response before speaking, no architecture saves you, because the dominant term is never the transport, it is the time spent generating tokens nobody hears.
1. Simple explanation
A voice agent has to do three things: understand speech, decide what to say, and say it. You can either hire three specialists and hand work between them, or hire one generalist who does all three inside their head.
The three-specialist version is called cascaded. Audio arrives, a transcription service turns it into text, a language model reads that text and writes a reply, and a synthesis service turns the reply back into audio. Every hand-off is text you can log, diff, and replay.
The generalist version is called speech-to-speech (S2S). Audio goes into a single model and audio comes out. Nothing in the middle is text, so there is nothing to read. It is faster, because there are no hand-offs, and it hears things the text pipeline throws away — hesitation, sarcasm, someone getting upset.
Analogy — a doctor's receptionist versus a doctor who answers their own phone. The receptionist writes down what the caller said, checks the appointment book, and reads back an answer. Slower, but there is a written record of every step, and you can change the appointment rules without retraining the receptionist. The doctor who answers directly is quicker and picks up on a worried tone — but you cannot audit the conversation, and if they start quoting prices they made up, your only recourse is asking them nicely not to. The analogy carries the actual mechanism: what you gain is speed and paralinguistic signal, what you lose is the written intermediate that makes control and auditing possible.
2. Diagram
CASCADED — three services, text in the middle
┌── you can log this
▼
audio ──▶ [ STT ] ──▶ "what time do you close" ──▶ [ LLM ]
│
▼
audio ◀── [ TTS ] ◀── "We close at six on weekdays." ─┘
▲
└── and this, and diff it, and replay it
+ cheap, auditable, swap any stage independently
- 3 network hops; tone/emotion discarded at the first arrow
SPEECH-TO-SPEECH — one model, nothing readable inside
┌─────────────────────────────┐
audio ───────────▶│ audio in ▶▶▶ audio out │──────────▶ audio
└─────────────────────────────┘
▲ ▲
│ └── no text to inspect
└── hears hesitation, stress, interruption
+ fewest hops, lowest latency, keeps paralinguistic signal
- ~2x price, opaque, token-based billing grows with call length
WHERE THE TIME ACTUALLY GOES (cascaded, not streamed vs streamed)
not streamed │endpoint│ STT │········ LLM writes it ALL ········│TTS│
▲ 1630 ms
streamed │endpoint│ STT │·· 1st sentence ··│TTS│
└ rest still generating ┘ ▲ 1030 ms
600 ms saved,
same cost
3. How it works
3.1 The cascade, stage by stage
Four stages sit between a caller finishing a sentence and hearing a reply.
Endpointing decides the caller has stopped talking. This is pure waiting — the system holds a stopwatch and commits when silence exceeds a threshold. It is a floor under every reply and it is the stage most teams forget to count. Covered in depth in Turn-Taking in Voice Agents: Endpointing, VAD and Barge-In.
Speech-to-text converts audio to a transcript. Streaming ASR emits partial hypotheses as it goes and a final transcript when the turn closes; the finalisation step costs a network round trip.
The language model reads the transcript plus whatever context you supply and writes a reply. This is almost always the largest single term in the budget, and the one most under your control.
Text-to-speech converts the reply to audio. The number that matters is time to first audio, not total synthesis time — you only need the first chunk to start playing.
3.2 Why the text in the middle is the actual product
The text intermediate is not an implementation detail, it is the feature that makes the architecture governable:
| capability | cascaded | speech-to-speech |
|---|---|---|
| read a transcript of any call | yes | needs a separate ASR pass |
| assert the agent never quoted a wrong price | grep the text | prompt-level hope |
| replay yesterday's calls against a new model | trivial | re-synthesise audio first |
| swap one stage for a cheaper vendor | independent | all or nothing |
| route cheap questions past the model entirely | yes | no hook to do it |
That last row is the big one. Because the transcript exists before any model runs, you can intercept it. Most phone traffic is a handful of repeated questions — opening hours, whether you take a particular insurance, where to park. Those can be answered from a lookup table for nothing, and only the leftovers need a model. Speech-to-speech gives you nowhere to stand to make that decision; the audio is already inside the model.
3.3 Streaming is not optional
The single most common failure is a pipeline that waits for a complete response before speaking. It reads naturally in code and it is fatal on a phone line:
wait for full transcript → wait for full reply → wait for full audio
Each wait for full is dead air. Streamed properly, partial transcripts flow to the model, model tokens flow to synthesis as they are produced, and synthesis begins on the first sentence boundary. The published range for this saving is roughly 300–500 ms, and the code above measures 600 ms on one representative budget.
A tell that a team has not done this: the agent says "Sure." or "One moment" before answering. Those filler phrases exist to cover the gap. They are a symptom, not a feature, and they bring their own bugs — you then need logic to stop the agent saying "Sure" twice when the real reply also begins with an acknowledgement.
3.4 What speech-to-speech genuinely buys you
Two things a cascade structurally cannot do. First, latency: no hand-offs means published p50 figures around 500–600 ms, versus roughly 700 ms for a well-built cascade. Second, paralinguistic signal — the first arrow of a cascade discards everything except words, so hesitation, distress, and sarcasm are gone before the model sees anything. If the product is a companion, a language tutor, or crisis triage, that signal may be the product.
3.5 Where this framing stops applying
The cascaded/S2S distinction assumes a turn-based conversation: someone speaks, someone answers. It describes nothing useful about simultaneous interpretation, always-listening ambient assistants, or agents that must speak over the user by design. It also assumes a telephone-grade constraint — roughly one second of tolerance. Relax that to a walkie-talkie or an async voice note and the entire latency argument evaporates, at which point you pick on cost and auditability alone and cascaded wins on both.
4. The math
4.1 First-audio latency
Latency composes additively along the critical path, so the budget is a sum, not an average:
T_first_audio = T_endpoint + T_stt_final + T_model_to_first_speakable + T_tts_first_chunk
where, unstreamed: T_model_to_first_speakable = time to generate the WHOLE reply
streamed: T_model_to_first_speakable = time to the first sentence boundary
The important consequence: T_endpoint and T_tts_first_chunk are close to fixed, so all of your leverage is in the middle term, and streaming shrinks it by a factor of roughly the number of sentences in a typical reply.
4.2 Cost per connected minute
C_total = C_telephony + C_stt + C_model + C_tts + C_orchestration [$/min]
C_model = turns_per_min * (in_tok * rate_in + out_tok * rate_out) / 1e6
C_model is the only term that varies by more than about 3x between sane choices, which is why it is where cost work starts. Published all-in figures for 2026 production deployments cluster at $0.07–$0.21 per connected minute, with cascaded stacks at the low end and speech-to-speech near the top.
4.3 Worked example
Take one representative cascaded budget, unstreamed:
endpointing 350 ms $0.000 /min
STT final 200 ms $0.020
LLM (whole reply) 900 ms $0.030
TTS first chunk 180 ms $0.035
telephony 0 ms $0.015
-------
T_first_audio 1630 ms $0.100 /min
Now stream it. The reply is about three sentences, so the first speakable chunk arrives in roughly a third of the time — 300 ms instead of 900 ms:
T_first_audio = 350 + 200 + 300 + 180 = 1030 ms cost unchanged at $0.100
600 ms removed for zero additional spend. And compare speech-to-speech on the same call:
endpointing (in-model) 120 ms $0.000 /min
audio-in -> audio-out 440 ms $0.175
telephony 0 ms $0.015
-------
T_first_audio 560 ms $0.190 /min
S2S is 470 ms faster and 1.9x the price. The honest reading: streaming buys most of the available latency for free, and the remaining 470 ms costs you double. Note also that 1030 ms is still above a comfortable target — streaming alone does not get you there, because the model tier is still wrong. That is the subject of Reasoning Budgets: When Thinking Tokens Are Waste.
5. Real code
"""Latency and cost budget for a cascaded vs speech-to-speech voice pipeline."""
from dataclasses import dataclass
@dataclass
class Stage:
name: str
ms: float # time before the NEXT stage can begin
usd_per_min: float
def cascaded(streaming: bool) -> list[Stage]:
"""STT -> LLM -> TTS. With streaming, TTS starts on the first sentence."""
# An unstreamed pipeline waits for the whole LLM answer before speaking.
# A streamed one only waits for the first sentence (~1/3 of the tokens).
llm_ms = 900.0 if not streaming else 300.0
return [
Stage("endpointing", 350.0, 0.000),
Stage("STT final", 200.0, 0.020),
Stage("LLM", llm_ms, 0.030),
Stage("TTS first audio", 180.0, 0.035),
Stage("telephony", 0.0, 0.015),
]
def speech_to_speech() -> list[Stage]:
"""One model consumes audio and emits audio; no text hand-offs."""
return [
Stage("endpointing (in-model)", 120.0, 0.000),
Stage("audio-in -> audio-out", 440.0, 0.175),
Stage("telephony", 0.0, 0.015),
]
def budget(stages: list[Stage]) -> tuple[float, float]:
return sum(s.ms for s in stages), sum(s.usd_per_min for s in stages)
def report(label: str, stages: list[Stage]) -> tuple[float, float]:
ms, usd = budget(stages)
print(f"{label}")
running = 0.0
for s in stages:
running += s.ms
if s.ms:
print(f" {s.name:<24} +{s.ms:6.0f} ms (t={running:6.0f})")
print(f" {'TOTAL to first audio':<24} {ms:7.0f} ms")
print(f" {'cost per minute':<24} ${usd:6.3f}")
print()
return ms, usd
naive_ms, naive_usd = report("CASCADED, no streaming", cascaded(streaming=False))
strm_ms, strm_usd = report("CASCADED, streamed", cascaded(streaming=True))
s2s_ms, s2s_usd = report("SPEECH-TO-SPEECH", speech_to_speech())
print(f"streaming saves {naive_ms - strm_ms:.0f} ms at identical cost")
print(f"S2S is faster by {strm_ms - s2s_ms:.0f} ms")
print(f"S2S costs more by ${s2s_usd - strm_usd:.3f}/min "
f"({s2s_usd / strm_usd:.1f}x)")
# The claims made in the prose must hold.
assert naive_ms == 1630, naive_ms
assert strm_ms == 1030, strm_ms
assert s2s_ms == 560, s2s_ms
assert naive_ms - strm_ms == 600
assert round(s2s_usd / strm_usd, 1) == 1.9
# Streaming alone does NOT reach the 800 ms target; the model tier must also change.
assert strm_ms > 800
print("\nall assertions passed")
# Output:
# CASCADED, no streaming
# endpointing + 350 ms (t= 350)
# STT final + 200 ms (t= 550)
# LLM + 900 ms (t= 1450)
# TTS first audio + 180 ms (t= 1630)
# TOTAL to first audio 1630 ms
# cost per minute $ 0.100
#
# CASCADED, streamed
# endpointing + 350 ms (t= 350)
# STT final + 200 ms (t= 550)
# LLM + 300 ms (t= 850)
# TTS first audio + 180 ms (t= 1030)
# TOTAL to first audio 1030 ms
# cost per minute $ 0.100
#
# SPEECH-TO-SPEECH
# endpointing (in-model) + 120 ms (t= 120)
# audio-in -> audio-out + 440 ms (t= 560)
# TOTAL to first audio 560 ms
# cost per minute $ 0.190
#
# streaming saves 600 ms at identical cost
# S2S is faster by 470 ms
# S2S costs more by $0.090/min (1.9x)
#
# all assertions passed
Swap the constants for numbers you have measured and the table becomes a decision tool rather than an illustration. If you cannot fill them in from your own logs, that is the first problem to fix — see §8.
6. Real-world example
A dental practice replaced its overflow answering service with a voice agent. Cascaded pipeline, sensible vendors, and on a quiet line it worked: callers asked about opening hours and insurance and got correct answers.
Then the receptionist started reporting that callers were arriving confused about prices the practice does not charge. The transcripts explained it in an afternoon. The team had wired the reply path so that if the model call raised any exception — quota, timeout, a transient network fault — the code silently fell back to "return the best-scoring entry from the FAQ table." The scorer counted shared words, with no minimum. A caller asking "do you do payment plans?" shared the words "do" and "you" with an entry about a treatment fee, scored above zero, and was read that entry as if it were an answer. It was logged with the same success status as a correct reply.
Three things made this expensive. The fallback had no relevance floor, so a bad match was indistinguishable from a good one. There was no separate status for "we degraded," so dashboards showed a healthy service. And nothing recorded which path each turn took, so the only way to find it was reading transcripts by hand.
The fix was small — require a minimum score, and when nothing clears it, say so and offer a human — but it was only findable because the architecture was cascaded. The transcript existed. In a speech-to-speech deployment, the same failure would have been audio in a bucket that nobody had a reason to listen to.
7. Interview questions companies actually ask
Q1. Why do most production voice agents still use a cascaded pipeline when speech-to-speech models have lower latency? Because latency is one requirement among several and the others favour cascade. Cascade produces a text intermediate, which is what makes the system auditable, replayable, cheaply testable, and — most importantly — interceptable, so you can answer common questions without invoking a model at all. It is also roughly half the cost per minute, with predictable token accounting rather than billing that scales with conversation length. A well-streamed cascade lands near 700 ms p50, which is inside the comfortable range, so the remaining latency gap is not worth doubling the bill and giving up the transcript for most workloads.
Q2. A caller says the agent talks over them. Where do you look? Turn-taking, not the model. Something is deciding the caller has finished too early — usually an endpointing threshold that is too short, or an interruption detector that is triggering on line noise or on the agent's own audio leaking back through the line. Check whether a voice activity detector is actually loaded, since asking for VAD-based interruption without configuring a VAD is a common misconfiguration that silently degrades to something else.
Q3. Your agent's first word arrives 4 seconds after the caller stops. Walk me through diagnosis. Instrument the four stages separately before changing anything: endpointing wait, transcript finalisation, model time to first speakable token, and synthesis time to first audio. In practice the model term dominates, and there are usually two causes stacked: the response is not streamed, so synthesis waits for the last token, and the model tier is doing hidden reasoning work that the reply does not need. Fix streaming and the tier, then re-measure before touching anything else.
Q4. How would you cut the cost per minute of an existing agent by half without changing vendors? Count calls before counting tokens. Most turns are repeated questions, so route them deterministically and skip the model; add a response cache keyed on the normalised question plus a version of the knowledge it depends on; put the static part of the prompt in a cached prefix; and cap output length, since output tokens are billed several times higher than input. Only then consider a smaller model tier — and validate it against a scored test set rather than by ear.
Q5. What does speech-to-speech let you build that a cascade cannot? Anything that depends on how something was said rather than what was said. The first stage of a cascade discards prosody, so emotional state, hesitation, overlapping speech, and accent-level detail are gone before the reasoning step. Companion products, pronunciation tutors, and distress triage are the cases where that signal is the product rather than a nicety.
Q6. Where do you put business rules — in the prompt or in code? In code, wherever the fact is already known. A conditional in a template is deterministic, testable, and free; the same rule expressed as a sentence of instruction is probabilistic, costs tokens on every call, and fails silently. Reserve the model for genuine language work: understanding an unusual phrasing, or composing a sentence you could not have written in advance.
Q7. How do you test a voice agent, given that the interface is audio? In two layers. Most coverage should be text-level: feed transcripts to the decision layer and assert on the route taken and the reply, which is fast, deterministic, and catches nearly all logic regressions. Then a smaller synthetic voice layer where generated callers dial the real system, to catch what only appears in audio — endpointing behaviour, interruptions, and cases where the agent goes silent. Track per-stage latency as a regression metric alongside correctness.
8. When to use / tradeoffs
Reach for cascaded when:
- You must be able to prove what the agent said, or restrict it to approved facts
- Cost per minute matters, or volume is high enough for it to matter later
- Most calls are a small set of predictable questions you could answer from a table
- You want to swap transcription or synthesis vendors independently
- The domain is regulated and text intermediates simplify audit
Reach for speech-to-speech when:
- Sub-600 ms is a genuine product requirement, not a preference
- Tone, hesitation, or emotional state changes what the agent should do
- Conversations are short enough that token-based billing stays bounded
- You can absorb roughly double the per-minute cost
| Situation | Why it breaks | Use instead |
|---|---|---|
| Regulated domain, must show what was said | No text intermediate to audit | Cascaded |
| High volume, thin margin | ~2x per-minute cost; grows with call length | Cascaded, with routing and caching |
| Long calls | Token billing grows non-linearly with history | Cascaded with a bounded context window |
| Emotion drives the decision | Cascade discards prosody at the first stage | Speech-to-speech |
| Simultaneous / always-listening | Both models assume turn-taking | Purpose-built streaming architecture |
| You cannot measure per-stage latency yet | Any architecture choice is a guess | Instrument first, then choose |
Honest limits. Every number in §4 is a budget, not a measurement of your system: they assume co-located services, a warm connection pool, an uncontended event loop, and a model that streams. Break any of those and the arithmetic understates reality — a cross-region hop adds 200–500 ms per exchange that appears in no vendor's latency chart, and a synchronous call on an async event loop makes p95 collapse under concurrency while p50 looks fine. The 1/3-of-tokens estimate for a first sentence is a rule of thumb that fails for one-sentence replies, where streaming buys almost nothing. Published per-minute figures also vary by an order of magnitude across sources because they draw the boundary differently — some include telephony and orchestration, some do not. Treat all of it as a framework for your own measurements, and be suspicious of any comparison, including this one, where the two options did not do the same amount of work.
9. Summary + related articles
- Two architectures: cascaded (STT → LLM → TTS, text in the middle) and speech-to-speech (audio in, audio out). Cascaded is the production default.
- Cascaded costs roughly half as much and gives you a transcript — which is what makes auditing, replay, cheap testing, and skipping the model entirely possible.
- Speech-to-speech is 400–500 ms faster and keeps paralinguistic signal. Choose it when tone drives behaviour or sub-600 ms is a hard requirement.
- Latency is additive along the critical path, and the model term dominates. Streaming to the first sentence boundary is the highest-leverage fix and costs nothing.
- Filler phrases like "Sure, one moment" are a symptom of an unstreamed pipeline, not a feature.
- Both architectures stop working the same way: waiting for a complete response before speaking. And neither framing applies to non-turn-based interaction, or when you have not yet instrumented per-stage latency.
Related:
- Turn-Taking in Voice Agents: Endpointing, VAD and Barge-In — the endpointing term in §4.1, and why it is a floor under every reply
- Reasoning Budgets: When Thinking Tokens Are Waste — why the model term is usually larger than it needs to be
- ML Inference Systems — serving latency percentiles and caching patterns that apply to the model stage
- RAG Cost Optimization: Find the Step That Runs Forty Times — §3.5 on not making the call at all, and §3.2 on per-role model routing
- Production Agents — latency budgets, cost control, and keeping decisions in code
- Streaming Modes in LangGraph — mechanics of emitting partial output to a consumer
- Agent Evaluation — building the scored test set §8 says you need before changing tiers
Resources
- Jurafsky & Martin — Speech and Language Processing (3rd edition draft), the Automatic Speech Recognition and Text-to-Speech chapters — free online, the standard reference for the two ends of the cascade: https://web.stanford.edu/~jurafsky/slp3/
- Radford et al. (2022) — Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), arXiv:2212.04356 — the weak-supervision approach behind most current multilingual ASR: https://arxiv.org/abs/2212.04356
- RFC 3261 — SIP: Session Initiation Protocol — the signalling layer beneath almost every telephony integration: https://www.rfc-editor.org/rfc/rfc3261
- LiveKit Agents documentation — turn detection, interruption handling, and pipeline composition for cascaded agents: https://docs.livekit.io/agents/
- Pipecat — open-source framework for real-time voice pipelines; useful for reading how the stages are wired in practice: https://github.com/pipecat-ai/pipecat
- Silero VAD — small, permissively licensed voice activity detector, the usual choice for local speech-stop detection: https://github.com/snakers4/silero-vad
- Per-minute cost and latency figures quoted in §4 are drawn from published 2026 vendor and practitioner benchmarks; they vary widely by how the boundary is drawn, so verify against your own invoices before using them in a decision.