Cruxly › AI engineering › AI System Design Case Studies

Case Study: A Voice Agent for Inbound Calls

850 milliseconds to first audio, an endpointing dial with no free setting, and a cost model that is all audio

Prompt: "Design a voice agent for a bank's customer-care line. Callers speak Hindi, English and a mix of the two, over ordinary phone lines. The agent should answer balance and card questions, block a lost card, and hand over to a human when it cannot help. Should it be a cascade of speech-to-text, an LLM and text-to-speech, or a speech-to-speech model? How do you handle interruptions, what is the latency budget, and how do you test it?"

40 minutes first. Write the latency budget as five numbers that sum to a target, decide cascaded versus speech-to-speech with reasons, and list what the law and the bank will require before the first call. The trap is to treat voice as chat with a microphone: turn-taking, telephone audio quality and cost per minute are different problems.

The latency budget: five numbers that must sum to a target

People tolerate a pause of roughly a second before a phone conversation feels broken, so voice systems are designed backward from a budget to first audio, and every stage gets a number. The budget below uses assumed round figures; vendors publish their own, but those are vendor-authored and directional, so treat them as hypotheses and benchmark on your stack.

Stage	What happens	Budget
Network	audio from phone network to your server and back	80 ms
Endpointing	deciding the caller has finished	200 ms
STT final	last words of the transcript	100 ms
LLM first token	time to first token, as in TTFT, TPOT, and the Latency Budget	350 ms
TTS first audio	first playable chunk	120 ms
Total	caller stops talking to agent starts	850 ms

Shave it with overlap, not faster hardware. The stages need not run strictly one after another.

Stream partial transcripts. STT emits words as the caller speaks, so by the time endpointing decides the turn is over, all but the last word or two are already transcribed: final STT falls from 100 to about 30 ms (saving 70).
A better end-of-turn decision. A model that looks at the words and the pause together (a caller who says "my card number is..." is clearly not finished) can use a shorter silence threshold safely: 200 down to about 150 ms (saving 50).
Start speaking on the first clause. Instruct the model to open with a short clause ("Sure, let me check that."), stream tokens to TTS as they arrive, and hand off at the first sentence or comma instead of waiting for the whole reply; with a cached system prompt (see Prompt Caching: Paying for the Same Tokens Once) LLM first token falls to about 300 ms (saving 50).

New total: 80 + 150 + 30 + 300 + 120 = 680 ms, a saving of 170 ms.

Percentiles, not medians. Five stages each with their own tail add up in a way medians hide. In a quick simulation with assumed lognormal stage times (medians 80, 200, 100, 350 and 120 ms, spreads of 0.3 to 0.45 in log space), the median total came out about 885 ms rather than 850, the 95th percentile about 1,300 ms and the 99th about 1,560. The shape is the point: p95 is roughly 1.5x the sum of medians, so set targets on p95 and watch the slowest stage. Where the budget is blown, the usual culprit is the LLM stage: long prompts, uncached prefixes or an overloaded endpoint.

The tail is where callers hang up. A single 3-second silence reads as a dropped call, and the caller says "hello?" over the agent's reply. Add a filler or earcon if generation exceeds a threshold (for example, 1.2 seconds), and keep tool calls (looking up a balance) off the hot path by speaking first ("Let me check that for you") and fetching in parallel.

Cascaded or speech-to-speech: a decision with reasons

A cascade (speech-to-text, then an LLM, then text-to-speech) is the debuggable default; a speech-to-speech model is a different architecture that trades visibility for naturalness. Say both, then pick for the constraints.

Cascade. Audio goes to STT, the transcript to a text LLM that can call tools, and the reply text to TTS. Strengths: every hop produces text you can log, redact, test and compare; you can swap components independently (a better Hindi STT, a different voice); the LLM is a normal text model with the whole tool-calling and evaluation toolkit of the earlier lessons; the transcript is the audit record. Weaknesses: latency adds up across hops; tone and emotion in the caller's voice are lost at the STT boundary; the TTS speaks only what the text says.

Speech-to-speech (an audio-native model). One model takes audio in and produces audio out, in one session, with built-in turn detection and interruption handling. OpenAI's Realtime guide, for example, describes exactly this, using WebRTC in a browser and WebSocket on a server, for voice agents that need barge-in, tool calls and handoffs inside one session. Strengths: fewer hops, so potentially lower latency, and more natural prosody and timing. Weaknesses: less visibility (you must still get a transcript for audit), fewer components to tune, language and accent coverage you must test yourself, and audio-based pricing.

:::key Verified October 2026 OpenAI's Realtime API guide (retrieved October 2026) describes speech-to-speech voice agents over WebRTC and WebSocket, with barge-in and handoffs inside a session, and gives no latency or price figures on that page. Prices used in this lesson: OpenAI's pricing and model pages (checked via search results on 2026-10-24, not fetched in full) list the GPT-Realtime family's audio at $32 per million input tokens ($0.40 cached) and $64 per million output tokens, the same on gpt-realtime and gpt-realtime-1.5. Models and rates change, so re-check the live page, and never quote a vendor's latency claims as your own measurement. :::

Decide with a table of constraints, for a bank.

Auditability and redaction: cascade wins, since text is available between hops. A speech-to-speech model can emit transcripts, but you then redact after the fact.
Latency: speech-to-speech may win on the median; a well-engineered streaming cascade closes much of the gap. Benchmark both on your calls.
Languages and code-switching: the answer is empirical. Test Hindi, English and Hinglish with real callers' audio on both.
Control of wording: regulated statements (a fee disclosure) must be spoken verbatim. In a cascade you play a pre-approved text or recording; in an audio-native model you must constrain it.
Cost: see section 4; the structure differs.

A defensible answer: start with a cascade for the regulated paths and keep the components swappable; run a speech-to-speech model as a measured experiment on non-regulated flows (branch locator, opening hours), and move traffic only if it wins on task success, latency and cost on your own call set. If you are asked to pick one without data, pick the cascade and say what result would change your mind. The Whisper paper (Radford et al., arXiv 2212.04356) shows why component choice matters: its models were trained on 680,000 hours of weakly supervised multilingual audio and reported robustness near human levels on standard benchmarks, which says nothing about 8 kHz Indian-accented banking calls. Evaluate on yours.

Turn-taking: endpointing, barge-in and backchannels

Turn-taking is the part of voice that has no chat equivalent, and its central dial, the endpointing silence threshold, has no free setting. Too short and the agent talks over callers who paused to think; too long and every reply feels slow.

Voice activity detection (VAD) is a small model that labels each 20-30 ms frame of audio as speech or not. On its own it only knows silence, so a pause inside a sentence looks like a finished turn. End-of-turn models add the words and prosody: "my account number is" is incomplete, "yes, that is correct" is complete. They let you use a shorter silence threshold when the sentence is clearly finished and a longer one when it is not.

The cost of getting it wrong, by arithmetic. Suppose 8 percent of caller turns contain a mid-sentence pause longer than your threshold (callers reading out a 16-digit card number do this constantly). Over a 12-turn call, the chance of at least one cut-off is 1 - 0.92^12 = 63 percent. Nearly two thirds of calls would include one interruption of the caller, which is why digit strings are collected by keypad (DTMF tones, which carry no ambiguity and keep the number out of the transcript) or with a longer threshold in "collecting digits" mode. Meanwhile each extra 100 ms of threshold adds 100 ms to every turn's latency.

Barge-in: the caller interrupts the agent. Detect speech while audio is playing and then do five things within about 100-200 ms:

Stop audio playback immediately on the client or media server.
Cancel the TTS stream and the in-flight LLM generation, which also stops spending tokens.
Truncate the conversation history to the words the caller actually heard. If the model planned a 60-word reply and was cut off after 15 words, the history must say only those 15 words were spoken; otherwise the model believes it already told the caller something they never heard.
Treat the new audio as the next turn.
Apply echo cancellation, otherwise the agent hears its own voice through the caller's speakerphone and interrupts itself.

Backchannels and noise. A caller's "hmm", "haan", "okay" while the agent talks is not an interruption. Require a minimum duration or word count before cancelling, or classify short acknowledgements separately. Background speech (a TV, another person) needs a speaker or energy filter; otherwise every cough cancels a sentence.

Silence handling. If the caller is silent for 5 seconds, prompt once ("Are you still there?"), then again after another 5, then offer a human or end politely. Never loop forever; a call that sits silent for ten minutes is a billed call.

Telephony and cost: 8 kHz audio, DTMF, and a bill made of audio

Phone audio is a harder input than the benchmarks, and the bill is dominated by audio, not by language-model tokens.

Telephony realities.

8 kHz narrowband audio. Phone networks carry the speech band at 8 kHz (commonly G.711 compression), while many speech models are developed mostly on wider-band audio. Accuracy on telephone audio is lower, so test on recorded calls, not studio clips.
Jitter and packet loss. Streaming audio over SIP arrives unevenly; a jitter buffer smooths it at the cost of tens of milliseconds, which comes out of your budget.
DTMF. Keypad tones are the right channel for card numbers, PINs and OTPs: no recognition errors, and the digits never enter a transcript.
Accents and code-switching. Callers mix Hindi and English in one sentence ("mera card block kar do please"). Choose an STT that supports it and measure it, with a custom vocabulary for product and branch names. Read back critical entities ("card ending 4821, correct?") before acting.
Noise. Streets, vehicles, a television; use noise suppression, and treat low-confidence transcripts as "please repeat".

Cost model, with assumptions labelled. A call lasts 5 minutes with 12 agent turns. A caller speaks about 150 words a minute and the agent about 120, and each side speaks for about 45 percent of the call (the rest is pauses and silence), so the agent talks for 135 seconds and says 2.25 x 120 = 270 words, roughly 360 tokens, and 1,620 characters. Cascade costs:

LLM: a 3,000-token prompt, re-sent each turn with 360 new tokens of history per turn: 12 x 3,000 + 360 x (0 + 1 + ... + 11) = 36,000 + 23,760 = 59,760 input tokens. At $2 per million input that is $0.12, but with the 3,000-token prefix read from cache at $0.10 per million it is 12 x 3,000 x 0.1 / 1M + 23,760 x 2 / 1M = $0.0036 + $0.0475 = $0.051, and 360 output tokens at $10 per million add $0.0036, so $0.055.
STT: assumed $0.01 per minute, so $0.05.
TTS: assumed $15 per million characters, so 1,620 x 15 / 1M = $0.024.
Total about $0.129 a call, $0.026 a minute, roughly ₹11 a call at an assumed ₹88 per dollar. The LLM tokens are about 40 percent; STT and TTS are about 60 percent. Substitute your vendor's actual rates.

The speech-to-speech structure differs. Audio-native models bill audio tokens, and each response re-reads the conversation so far. Assume (check the provider's documentation; some count output audio at a higher rate) 10 audio tokens per second, so a turn pair adds 25 s x 0.45 x 10 x 2 = 225 tokens. Re-reading history over 12 turns is 225 x (0 + 1 + ... + 11) = 14,850 tokens, 5.5 times the 2,700 tokens of actual audio. At the reported $32 per million input, that re-read alone is $0.475 a call if uncached and $0.006 if cached at $0.40; fresh audio in and out adds 1,350 x $32 / 1M + 1,350 x $64 / 1M = $0.13. Total about $0.60 a call uncached versus $0.14 cached. The lesson is structural, not the figures: in speech-to-speech, caching the conversation is the cost lever, and cost grows faster than linearly with call length.

Scale. At 30,000 calls a day with 12 percent in the busy hour, concurrency is 3,600 x 5 / 60 = 300 simultaneous calls, so size telephony channels and STT/TTS capacity for about 450 with headroom. The cascade at $0.129 costs 30,000 x 0.129 = $3,870 a day; compare with your measured cost per human-handled minute and your containment rate.

Compliance, testing, and the half-the-cost answer

Regulation and bank policy decide what the agent may say and record, and you must ask before the first call, not after.

Requirements to confirm with legal and compliance (this lesson is not legal advice, and I could not verify the primary texts for India in this pass):

Recording consent and disclosure: whether calls may be recorded, the notice played, and that the caller is told they are speaking to an automated system. A law-firm article by S.S. Rana & Co. (republished on Bar & Bench; I saw it only through search-result summaries, not the full text, in October 2026) reports that TRAI amended its telemarketing rules in 2025 to require explicit consent for auto-dialed and robocalls and that a further amendment on AI-based telemarketing and AI disclosure was reported as under consideration, not confirmed as notified. For banking calls, RBI and the bank's own policy apply; I did not find primary RBI text, so take the current requirements from compliance, not from this page.
Personal data: India's Digital Personal Data Protection Act applies to voice and transcripts. Define retention, redact card numbers and OTPs from transcripts, restrict access to recordings, and plan deletion, as in Red-Teaming and Compliance as Code.
Identity verification: before sharing account details, verify the caller (OTP to the registered mobile entered by DTMF, a known-answer question). The model never decides that verification passed; a tool returns a verified flag the model cannot set, as in Guardrails for an Agent That Takes Real Actions.
Handoff: when the caller asks for a human, is frustrated, fails verification twice, or the model is unsure, transfer with a summary (intent, verified status, what was tried) so the caller does not repeat themselves. Measure the handoff rate; it is also your quality signal.
Blocking a lost card is a real action: read-back confirmation, then a tool with idempotency, and an audit entry.

Testing without calling real customers.

Scripted call simulations: 200 scenarios (balance, card block, dispute, angry caller, wrong number), each played by a synthetic caller (an LLM plus TTS with accent and noise variants) against the full stack. Score task success, turns to completion, correct handoff and compliance statements spoken.
Word error rate by accent: WER = (substitutions + deletions + insertions) / reference words; with 6 substitutions, 2 deletions and 2 insertions in 100 words it is 10 percent. Report it per accent and per audio condition on real recorded calls (with consent), because an average conceals the dialect that fails.
Latency percentiles: p50, p95 and p99 time to first audio under load, as in section 1.
Interruption tests: inject speech at 20 positions in an agent utterance and check playback stops, history truncates correctly and no double replies occur.
Safety: prompts to reveal another customer's data, to skip verification and to issue actions without confirmation; zero tolerated.
Cost per call and per minute, tracked next to containment rate, via Observability and Cost Governance for LLM Systems.

If asked to halve the cost: cache the prompt prefix (and in speech-to-speech, the conversation); shorten agent turns (fewer TTS characters and output tokens, and callers prefer short replies); use a smaller LLM on simple intents with route-level metrics; play pre-recorded audio for fixed phrases (greeting, disclosures) so TTS is not billed; end silent calls early; and negotiate STT/TTS volume pricing. Do not cut the verification, disclosure or testing budget.

How to say it in the interview

"A voice agent has to answer within about 800 milliseconds of the caller stopping, so I start from a budget: 80 for the network, 200 for deciding the caller has finished, 100 for the final transcript, 350 for the language model to produce its first token, 120 for the first audio, which is 850, and I bring it to about 680 by streaming partial transcripts into the model and starting speech on the first clause. I choose a cascade for a regulated bank because I can see, test and redact the transcript at every hop and swap each component, and I would evaluate a speech-to-speech model against it on the same call scripts. Interruptions are handled by detecting speech during playback, stopping audio within about 100 milliseconds, cancelling generation and keeping only the words actually spoken in the history. Telephony is 8 kilohertz audio with accents and code-switching, so I test word error rate by accent on real phone audio. The cost is mostly audio, not language-model tokens. Disclosure that it is an AI, recording consent, identity verification and a handoff with context are launch requirements."

Checklist

Wrote the latency budget as five stage numbers that sum to a target, and measured p95
Overlapped stages: streamed partial transcripts, started TTS on the first clause, cached the prompt
Chose cascade for regulated paths and benchmarked speech-to-speech on the same call set
Used an end-of-turn model plus a tuned silence threshold, with a longer mode for digit collection
Implemented barge-in: stop audio, cancel generation, truncate history to heard words, echo cancellation
Collected card numbers, PINs and OTPs by DTMF and kept them out of transcripts
Tested on real 8 kHz phone audio, with WER reported per accent and Hindi-English code-switching
Priced the call from STT, TTS and LLM or audio-token rates, with the history re-read in speech-to-speech
Played disclosure and recording notices, and confirmed current rules with compliance
Verified identity with a tool-set flag the model cannot write, and read back before actions
Handed off with a summary and tracked handoff rate
Ran 200 simulated calls including interruptions and attacks as a release gate

Practice this interactively