Cruxly

Cruxly › AI engineering › Portfolio Projects

Project: A Hindi and Hinglish Voice Assistant With Endpointing, Barge-In and a Latency Budget

Speech in, grounded speech out, with a per-turn latency budget, barge-in and a document-photo path

Open the interactive lesson

Project: "A clinic helpline gets calls in Hindi, English and Hinglish, mostly after 6 pm. A caller asks when the clinic is open, or how to pay a bill, and expects a spoken answer in a few seconds. I want a voice assistant that answers only from the clinic and utility FAQ, stops talking when the caller speaks, and tells me each turn's time budget. I also want to know what I cannot measure without real recordings."

Time-box: about 38 hours for the core, 55 with the stretch goals. You will have a voice pipeline package with about 470 lines of standard-library Python, 25 offline tests, WAV fixtures with known boundaries, a turn loop on a simulated clock, a document-photo quality gate, and an eval harness that prints a summary. You will not have a WER number for any real speech. The lesson explains why, and what you must record yourself to get one.

Why this project, and what it proves to an employer

A voice system is hard because the latency adds up across parts that each look fast, so the budget and its levers are the real skill. Most candidates demo one component. A reviewer wants to see that you can explain where the time goes, what each stage costs, what happens when a stage is slow, and how the caller is protected from a wrong answer.

The plan names composition under a latency budget and evaluation of non-text systems, plus an India angle: Hindi and Hinglish speech and photographed forms. I cover the first two with code and tests, and the third with a labelled stand-in; I claim no Hindi recognition result. Background lessons: Case Study: A Voice Agent for Inbound Calls and TTFT, TPOT, and the Latency Budget. Retrieval reuses llmkit's embedder.

Scope, metric and non-goals (numbers first)

Before any code, I wrote the targets down, and I separated the ones I can test offline from the ones I cannot.

Non-goals. No real speech recogniser or synthesiser, no vision-language model, no telephony, no streaming partial hypotheses, no speaker separation. The offline tests cover plumbing and timing logic, and each section says which numbers are simulated. That scope set is English; Hinglish and Devanagari are tried separately and fail in part (Milestone 4).

Architecture and the seam: what is fake, what is real

The turn loop only depends on four interfaces, and each interface has a fake that returns scripted output. The real adapters for speech and vision plug in behind those interfaces later, and the timing logic does not change.

What is fake and what is real. Speech in is a WAV file generated from tones, with its true start and end recorded in an index, so endpointing has a known answer to test against. Speech out is FakeTTS, which returns bytes in proportion to characters; that is not a speech rate. The clock is simulated, and each stage adds an assumed duration.

The fixtures are in the project folder and total about 320 kilobytes. They were generated by evals/make_fixtures.py, printed in full below, so the boundaries can be re-derived. Stages and the FAQ are printed too, because the numbers and the answers depend on them.

# -*- coding: utf-8 -*-
"""Writes the WAV fixtures and their TRUE boundaries (in seconds). Synthetic tones, not recorded speech:
they test endpointing timing, nothing about real speech or recognisers."""
import json
import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "src"))
from voice.audio import quiet, tone, write_wav  # noqa: E402

OUT = os.path.join(HERE, "..", "fixtures", "audio")


def build(name, segments):
    """segments: list of ('t'|'q', seconds). Returns the sample list and the true speech intervals."""
    samples, t, speech = [], 0.0, []
    for kind, secs in segments:
        if kind == "t":
            samples += tone(secs)
            speech.append((round(t, 3), round(t + secs, 3)))
        else:
            samples += quiet(secs, seed=len(samples))
        t += secs
    write_wav(os.path.join(OUT, name + ".wav"), samples)
    return {"file": name + ".wav", "duration_s": round(t, 3), "speech_intervals_s": speech}


def main():
    os.makedirs(OUT, exist_ok=True)
    index = {
        "utt_single": build("utt_single", [("q", 0.3), ("t", 1.2), ("q", 2.0)]),
        "utt_pause_400ms": build("utt_pause_400ms", [("q", 0.3), ("t", 0.6), ("q", 0.4), ("t", 0.7), ("q", 1.0)]),
        "utt_pause_900ms": build("utt_pause_900ms", [("q", 0.3), ("t", 0.6), ("q", 0.9), ("t", 0.7), ("q", 1.0)]),
    }
    with open(os.path.join(OUT, "index.json"), "w", encoding="utf-8") as f:
        json.dump(index, f, indent=1, sort_keys=True)
    total = sum(os.path.getsize(os.path.join(OUT, n)) for n in os.listdir(OUT))
    print(json.dumps(index, indent=1, sort_keys=True))
    print("total bytes:", total)


if __name__ == "__main__":
    main()
    asr_ms: float = 250.0
    retrieval_ms: float = 80.0
    llm_first_ms: float = 500.0      # time to the first complete sentence
    llm_next_ms: float = 150.0       # time per later sentence
    tts_ms: float = 150.0            # time to the first audio chunk of a sentence


@dataclass
FAQ = [
    ("bill_due", "when is my utility bill due and is there a late fee", "Your utility bill is due on the 15th of each month. Late payment adds a small fee."),
    ("bill_pay", "how do I pay my utility bill online or by upi", "You can pay your bill online, at a bank counter, or through UPI with your consumer number."),
    ("clinic_hours", "what are the clinic opening hours and is it open on sunday", "The clinic is open from 9 am to 6 pm, Monday to Saturday. It is closed on Sunday."),
    ("clinic_book", "how do I book a clinic appointment for a date and time", "To book a clinic appointment, give your name and a preferred date and time."),
    ("meter_read", "how often is the meter read and how do I report a reading", "A meter reading is taken every two months. Share a photo of the meter to report a reading."),
    ("connection_new", "what documents do I need for a new electricity connection", "A new connection needs an identity proof and an address proof, and takes about seven days."),
]


STOP = set("a an the is are am be do does i my me we you your it to of for on in at by and or how what when where can "
           "please is there give tell with about from so that this will would should get".split())

What is printed and what is not. The lesson prints the functions where a decision lives, plus make_fixtures.py, Stages and the FAQ. The rest of the six files under src/voice/ (about 470 lines) is in the repository: the scripted ASR and TTS, lang_id, cer, tone and WAV helpers, the page renderer, redact, read_back. The package has no __init__.py: with src on the path, Python 3.9 treats voice as a namespace package. llmkit must be at <root>/aip1work/llmkit or in the folder named by AIP_SCRATCH.

Milestone 1 and 2: endpointing with a hangover

Endpointing decides when the caller has finished, and the only fair test is a clip whose true boundaries you wrote down yourself. The endpointer computes the energy of each 20 ms frame. Speech starts when a frame passes a threshold. The turn ends only after the energy has stayed below the threshold for the hangover, which is 450 ms here. The hangover is what stops a short pause from cutting the caller off.

class Endpointer:
    """Speech starts when a frame's RMS passes `threshold`. The turn ends only after `hangover_ms` of
    continuous quiet, so a short pause inside a sentence does not cut the user off."""

    def __init__(self, threshold: float = 300.0, hangover_ms: int = 450):
        self.threshold = threshold
        self.hangover_frames = hangover_ms // FRAME_MS
        self.in_speech = False
        self.quiet_run = 0
        self.speech_start: Optional[int] = None

    def feed(self, frame_index: int, samples: List[int]) -> Optional[int]:
        """Returns the frame index at which the turn is declared over, or None."""
        loud = frame_rms(samples) >= self.threshold
        if loud:
            if not self.in_speech:
                self.speech_start = frame_index
            self.in_speech, self.quiet_run = True, 0
            return None
        if self.in_speech:
            self.quiet_run += 1
            if self.quiet_run >= self.hangover_frames:
                self.in_speech = False
                return frame_index
        return None

The helper endpoint runs the endpointer over a whole clip and returns the start and end times. It is the function the tests call.

def endpoint(samples: List[int], threshold: float = 300.0, hangover_ms: int = 450) -> Tuple[Optional[float], Optional[float]]:
    """Returns (speech_start_s, endpoint_s) for a whole clip."""
    ep = Endpointer(threshold, hangover_ms)
    end_idx = None
    for i in range(0, len(samples) - FRAME + 1, FRAME):
        r = ep.feed(i // FRAME, samples[i:i + FRAME])
        if r is not None:
            end_idx = r
            break
    start = None if ep.speech_start is None else ep.speech_start * FRAME_MS / 1000.0
    end = None if end_idx is None else (end_idx + 1) * FRAME_MS / 1000.0
    return start, end

The three fixtures test three things. The single utterance ends about one hangover after the speech stops, which is what the tests assert. The 400 ms pause inside a sentence does not end the turn. The 900 ms pause does end it, and the test asserts that too, because that is the designed behaviour and a reader should not mistake it for a bug.

The plan asked for two things at once: an endpoint within 300 ms of the true end, and no endpoint inside a 400 ms pause. One hangover cannot do both. Below 400 ms it splits the pause; above it, every turn is delayed. I chose 450 ms: 450 divided by the 20 ms frame is 22 whole frames, so the endpoint lands 440 ms after speech stops on the fixture. That misses the 300 ms target, and the lesson reports the trade-off; a two-threshold design is the stretch goal. The fixed energy threshold suits synthetic tones; a deployment uses a trained voice activity detector, which I did not test.

Milestone 3: ASR evaluation, WER, and Hindi and Hinglish normalisation

A word error rate is only as good as the normalisation applied to both sides, and Hindi text has several ways to be the same word. The normalisation in this project does four things: Unicode composition, Devanagari digits to ASCII digits, lower-case Latin, and removal of the danda and punctuation. Scoring then counts substitutions, deletions and insertions with a Levenshtein alignment.

def normalise(text: str) -> str:
    """Hindi/Hinglish normalisation for scoring: NFC, Devanagari digits to ASCII, lower-case Latin,
    punctuation stripped (the danda and common marks), ZWJ/ZWNJ removed, whitespace collapsed."""
    t = unicodedata.normalize("NFC", text).translate(DEVA_DIGITS)
    t = t.replace(chr(0x200C), "").replace(chr(0x200D), "")   # joiners are drawing hints, not word breaks
    t = t.replace("।", " ").replace("॥", " ")          # danda and double danda
    t = re.sub(r"[^\w\sऀ-ॿ]", " ", t.lower())
    return " ".join(t.split())
def _edit(ref: Sequence[str], hyp: Sequence[str]) -> Tuple[int, int, int, int]:
    """Levenshtein alignment. Returns (substitutions, deletions, insertions, reference_length)."""
    n, m = len(ref), len(hyp)
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        d[i][0] = i
    for j in range(m + 1):
        d[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost)
    i, j, s, dl, ins = n, m, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + (0 if ref[i - 1] == hyp[j - 1] else 1):
            s += ref[i - 1] != hyp[j - 1]
            i, j = i - 1, j - 1
        elif i > 0 and d[i][j] == d[i - 1][j] + 1:
            dl += 1
            i -= 1
        else:
            ins += 1
            j -= 1
    return s, dl, ins, n
def wer(reference: str, hypothesis: str) -> float:
    """Word error rate on normalised text: (S + D + I) / N. Empty reference with output gives 1.0."""
    ref, hyp = normalise(reference).split(), normalise(hypothesis).split()
    if not ref:
        return 0.0 if not hyp else 1.0
    s, d, i, n = _edit(ref, hyp)
    return (s + d + i) / n

The tests check WER against hand-computed edits (one substitution in three words is 1/3, one deletion is 1/3) and that the Devanagari digit two scores the same as the Latin 2. evals/hinglish_probe.py adds Hindi checks (block in the next section): मेरा बिल २५०० रुपये का है। against मेरा बिल 2500 रुपये का है scores 0.00, because digits and danda are normalised; क्लिनिक कब खुलता है against क्लिनिक कब खुलेगा scores 0.50 (four reference words, one substitution, one deletion). My first normalisation turned the zero-width joiner in क्‍ष into a space and split one conjunct; it is fixed and tested. Romanised Hindi has no standard spelling: kab against kub scores 0.25, and nothing here folds spellings.

I did not report a WER for any speech. There are no recordings here, and a WER on tones would describe my normalisation, not a recogniser. The plan's golden set of 40 utterances must be recorded with consent and scored with these functions. Until then the claim is that the function is correct, not that any system is accurate.

Milestone 4: grounded answers, and the scope check

A voice assistant that answers from memory is a liability in a clinic, so every reply is grounded in one FAQ entry, and anything outside the FAQ is refused. The FAQ holds six entries. Each entry is indexed by a question a caller would ask, and answered with a fixed text. Retrieval embeds the caller's question and the FAQ questions with llmkit's hash embedder, after removing question and function words.

def content(text: str) -> str:
    """Drop question words and function words, so 'how do I bake a cake' shares nothing with 'how do I pay'."""
    return " ".join(w for w in text.lower().replace("?", " ").split() if w not in STOP)
class Grounder:
    def __init__(self, faq=FAQ, threshold: float = 0.2):
        """Entries are indexed by their QUESTION text (what a caller would ask), and answered with the ANSWER."""
        self.faq = list(faq)
        self.threshold = threshold
        self.embedder = HashEmbedder()
        self.vectors = self.embedder.embed([content(q) for _, q, _ in self.faq])
        self.keys = [k for k, _, _ in self.faq]

    def retrieve(self, question: str) -> Tuple[Optional[str], float]:
        """Returns (faq_id, score). faq_id is None when the best score is below the scope threshold."""
        q = self.embedder.embed_one(content(question))
        best_i, best = max(((i, cosine(q, v)) for i, v in enumerate(self.vectors)), key=lambda t: t[1])
        return (self.keys[best_i] if best >= self.threshold else None), best

    def answer(self, question: str) -> Optional[str]:
        fid, _ = self.retrieve(question)
        if fid is None:
            return None
        return next(a for k, _, a in self.faq if k == fid)

The first version indexed answers with raw words and failed both ways: "how do I bake a cake" scored 0.42 against a booking entry because "how do I" is most of the sentence. Removing function words and indexing questions fixed it. On 30 hand-written English questions (15 in scope, 15 out) at threshold 0.2, the check answers 14 of 15 in-scope questions and refuses 15 of 15 out-of-scope ones. The threshold was set on the same items, so this is a development result, as the eval output says.

Everything above is English, and the title promises Hindi and Hinglish. I wrote ten Hinglish and four Devanagari in-scope questions and six out-of-scope ones once, did not tune on them, and ran evals/hinglish_probe.py (block above). Hinglish: answered 9 of 10, routed correctly 7 of 10, helped by shared English nouns; clinic ka time kya hai went to the booking entry. Devanagari: answered 0 of 4, as it shares no token with the English FAQ, so its refusals are partly empty praise. The fix is Hinglish and Devanagari paraphrases for each entry, or a multilingual embedder, then a held-out measurement. This is the main gap in the project.

scope (hand-written, dev only): in-scope answered 14/15, correct FAQ 14/15; out-of-scope refused 15/15
Hinglish in-scope: answered 9/10, correct FAQ 7/10
Devanagari in-scope: answered 0/4, correct FAQ 0/4
Hindi and Hinglish out-of-scope: refused 6/6
wrong FAQ entry chosen: ['clinic ka time kya hai', 'naye connection ke liye kaunse documents chahiye']
normalise: 'मेरा बिल २५०० रुपये का है।' -> 'मेरा बिल 2500 रुपये का है'
normalise: 'क्\u200dष' -> 'क्ष'
WER, digits and danda only differ: 0.00
WER, one word differs (खुलता vs खुलेगा): 0.50
WER, Hinglish spelling variant (kab vs kub): 0.25
lang_id: hi, hinglish, en

Milestone 5: sentence streaming and time to first audio

The caller hears the first sentence while the model is still writing the second, so the time to first audio depends on the first sentence, not the whole answer. The chunker collects tokens and emits each sentence as soon as it is complete, which means at a full stop, a question mark or a danda, followed by whitespace. The last piece is emitted when the stream ends.

def sentences(stream: Iterable[str]):
    """Yields complete sentences from a token stream. A sentence is complete at . ? ! or the danda once the
    next piece of text starts, so the last one is emitted when the stream ends."""
    buf = ""
    for tok in stream:
        buf += tok
        parts = SENT_END.split(buf)
        for p in parts[:-1]:
            yield p.strip()
        buf = parts[-1]
    if buf.strip():
        yield buf.strip()
def run_turn(reply_tokens: Iterable[str], stages: Stages, budget_ms: float = 1200.0,
             asr_ms_observed: Optional[float] = None, barge_in_ms: Optional[float] = None,
             max_sentences: int = 3, degrade_llm_first_ms: float = 250.0) -> TurnResult:
    """asr_ms_observed lets a test inject a slow recogniser. If the time to first audio would exceed the budget
    once ASR has finished, the degrade path shortens the answer to one sentence and uses the smaller model
    (degrade_llm_first_ms), then recomputes TTFA."""
    asr = stages.asr_ms if asr_ms_observed is None else asr_ms_observed
    pre = stages.endpoint_ms + asr + stages.retrieval_ms
    degraded = False
    first = stages.llm_first_ms
    limit = max_sentences
    if pre + first + stages.tts_ms > budget_ms:
        degraded, first, limit = True, degrade_llm_first_ms, 1
    res = TurnResult(ttfa_ms=0.0, degraded=degraded,
                     stage_ms={"endpoint": stages.endpoint_ms, "asr": asr, "retrieval": stages.retrieval_ms,
                               "llm_first": first, "tts_first": stages.tts_ms})
    clock = pre + first            # LLM has produced its first sentence
    gen = sentences(reply_tokens)
    for n, sent in enumerate(gen):
        if n >= limit:
            break
        if n == 0:
            res.ttfa_ms = clock + stages.tts_ms
        else:
            clock += stages.llm_next_ms
        # the barge-in check happens on the playback tick grid
        if barge_in_ms is not None and clock + stages.tts_ms >= barge_in_ms:
            tick_edge = ((barge_in_ms // TICK_MS) + (1 if barge_in_ms % TICK_MS else 0)) * TICK_MS
            res.cancelled = True
            res.cancel_latency_ms = (tick_edge - barge_in_ms) + DETECT_MS
            gen.close()
            break
        res.spoken.append(sent)
        clock += stages.tts_ms
    return res

run_turn adds the stages in order: endpointing, recognition, retrieval, the first sentence from the model, and the first audio chunk. It records the time to first audio from the caller's last word. The durations are the assumptions in the Stages dataclass, not measurements. The first sentence is counted; later sentences add their own durations, and the loop stops after max_sentences.

Milestone 6: budget enforcement and the degrade path

A latency budget is only real if the system does something when it is exceeded, and the default configuration of this project exceeds its own budget. After recognition finishes, run_turn checks the remaining time. If the time to first audio would exceed the budget, the degrade path uses a smaller model for the first sentence and speaks only one sentence.

With the stage assumptions as written, the nominal time to first audio is 1,430 ms, above the 1,200 ms budget. That is simulated arithmetic: assumed stage times (endpoint 450, recognition 250, retrieval 80, first sentence 500, first audio chunk 150 ms) added in sequence, with no streaming recognition. The degrade path replaces the first sentence by 250 and gives 450 + 250 + 80 + 250 + 150 = 1,180, under the budget by 20 ms on assumed numbers. A 300 ms hangover saves 150 ms, a 250 ms first sentence saves 250 ms, and both give 1,030 ms. Whether a real model reaches 250 ms only a measurement can show.

simulated TTFA, both levers: 1030 ms
simulated TTFA, fast LLM first sentence (250 ms): 1180 ms
simulated TTFA, nominal (450 ms hangover): 1430 ms
simulated TTFA, short hangover (300 ms): 1280 ms
barge-in (simulated): 42 of 42 times cancelled, max cancel latency 109 ms

The test with an injected slow recogniser is the one that proves the mechanism: when the observed recognition time is 900 ms, the degrade flag is set, one sentence is spoken, and the time to first audio stays under the budget plus a margin.

Milestone 7: barge-in and turn-taking

When the caller speaks over the answer, the system must stop within a fraction of a second, or the caller stops trusting it. Playback checks for barge-in on a 50 ms tick, and voice activity detection needs about 60 ms, so in the simulation the worst-case cancellation is at most 109 ms. The loop cancels the LLM and TTS together and stops appending sentences.

The sweep places barge-in at 42 times between 0.5 and 1.2 seconds, and all 42 cancelled. The 109 is not a result of the sweep: it is the distance to the next tick (at most 49 ms) plus the assumed 60 ms detection delay, so any sweep returns it. The sweep checks that the cancel path runs, nothing is appended afterwards and the generator is closed. Real cancellation depends on the audio stack and echo cancellation, which the simulator lacks. The half-duplex fallback (mute the microphone during playback) is not implemented.

Milestone 8: the document-photo path and its quality gate

A photo that is blurred or unevenly lit gives wrong fields with full confidence, so the gate runs before any OCR and says why it refused. The image is a grid of grey levels produced by the project, so no imaging library is needed. The gate computes two numbers from the pixels: sharpness, which is the variance of a four-neighbour Laplacian, and evenness, which is the ratio of the dimmest to the brightest third of the page.

def sharpness(img: Image) -> float:
    """Variance of the 4-neighbour Laplacian over the interior. Sharp edges give high variance; blur gives low."""
    vals = []
    for y in range(1, H - 1):
        for x in range(1, W - 1):
            vals.append(4 * img[y][x] - img[y - 1][x] - img[y + 1][x] - img[y][x - 1] - img[y][x + 1])
    m = sum(vals) / len(vals)
    return sum((v - m) ** 2 for v in vals) / len(vals)
def quality_gate(img: Image) -> Tuple[bool, dict]:
    s, e = sharpness(img), evenness(img)
    ok = s >= SHARP_MIN and e >= EVEN_MIN
    return ok, {"sharpness": round(s, 1), "evenness": round(e, 3), "pass": ok}

The thresholds were chosen by hand on clean and degraded renders of one page, not on real photographs, and the lesson says so. On that page the clean render passes, blur at radius 1 and 2 passes, blur at radius 3 fails, and shadow at strength 0.3 fails on evenness. The gate is a real computation on real pixels, but a real phone photo will need its own thresholds, measured on photos you take.

Field extraction is a set of regular expressions over the recognised text, and recognition is a fake. fake_ocr replaces characters with question marks on a schedule that grows with the level. The summary shows all three fields survive clean text and none survive level 1. That is a fact about the script, not about any OCR engine's error rate.

image gate clean: pass=True sharpness=37913.4 evenness=0.960
image gate blur r=1: pass=True sharpness=2816.0 evenness=0.958
image gate blur r=2: pass=True sharpness=937.3 evenness=0.957
image gate blur r=3: pass=False sharpness=373.7 evenness=0.958
image gate shadow 0.3: pass=False sharpness=27784.7 evenness=0.780
image gate shadow 0.6: pass=False sharpness=19823.9 evenness=0.544
scripted OCR level 0: fields correct 3/3
scripted OCR level 1: fields correct 0/3
scripted OCR level 2: fields correct 0/3
scripted OCR level 3: fields correct 0/3
def fake_ocr(text: str, level: int) -> str:
    """SCRIPTED corruption: at level k, every step-th letter or digit is replaced by '?', where step is 4, 3, 2 for
    levels 1, 2, 3 (max(2, 5 - k)). Level 0 is clean.
    A real OCR engine is a different animal; this only lets the extraction and the scoring run offline."""
    if level <= 0:
        return text
    step = max(2, 4 - level + 1)
    return "".join("?" if (i % step == 0 and c.isalnum()) else c for i, c in enumerate(text))

Milestone 9: privacy and safety for a voice turn

The audio of a turn should exist only as long as the turn, and a log should never hold a full account number. AudioBuffer holds the samples of one turn. release overwrites each sample in place, empties the buffer and returns the count. The samples live in an array('h') of raw 16-bit values, so the overwrite replaces the bytes this object owns (a list of Python integers would only drop references). It cannot reach copies it does not own: the caller's list, swap files, operating-system buffers.

class AudioBuffer:
    """Holds the samples of one turn as raw 16-bit values in an array. release() overwrites every value with zero
    IN PLACE and then empties the buffer, so the bytes this object owns are scrubbed. It cannot scrub copies it
    does not own: the caller's own list, swap files, or operating-system buffers."""

    def __init__(self):
        self._samples = array("h")

    def add(self, samples) -> None:
        self._samples.extend(samples)

    def __len__(self) -> int:
        return len(self._samples)

    def release(self) -> int:
        n = len(self._samples)
        for i in range(n):
            self._samples[i] = 0
        del self._samples[:]
        return n

Transcripts are redacted for the log: any run of four or more digits keeps its last four and masks the rest. Sensitive numbers are read back digit by digit before the system acts on them, so the caller can correct a misheard number. Out-of-scope questions get one spoken refusal, and the refusal is the same sentence every time, so it cannot be used to probe the FAQ.

What is not in the code: a consent notice at the start of a call and a retention policy beyond the buffer. Both are product decisions a lawyer should review, so they are open items.

:::key Checked 28 October 2026 (check the current rules before you rely on them) India's Digital Personal Data Protection Act passed in 2023. According to law-firm summaries of the DPDP Rules notified in November 2025, they take effect in stages: Board provisions on notification, consent managers about 13 November 2026, the main obligations about 13 May 2027. I read summaries, not the Gazette, and they differ on details; check the MeitY and Data Protection Board pages. :::

Milestone 10: the end-to-end eval, the test log, and what it does not show

An end-to-end voice eval needs recorded callers, and this project has none, so the eval reports only what the offline pieces can support. The plan asked for 50 scripted dialogues, 40 recorded utterances and 30 photos. I built only the pieces that run offline: the scope counts, the latency arithmetic, the barge-in sweep and the image gate.

Twenty-five tests run in well under a second, which is what you want from the plumbing layer, because a timing bug is easier to find when the clock is simulated.

.........................
----------------------------------------------------------------------
Ran 25 tests in 0.102s

OK

What the eval does not show: word error rate on speech, the naturalness of a voice, the real delay of a model, telephone behaviour, or the field accuracy of a real OCR engine. Each needs a recording or a model, and each needs its own golden set. Every README number carries one label: measured on recordings, simulated from stated assumptions, or scripted for a plumbing test.

Cost, latency and deployment

The cost of a turn is dominated by the model, and the speech parts are free only when they run locally. The plan's figure of about $0.002 of model cost per turn, with ten turns in a three-minute call, is an assumption I did not verify; replace it with your model's price. Local options would be a Whisper-family recogniser and a local synthesiser; the faster-whisper repository page shows an MIT licence (opened on 28 October 2026).

:::key Checked 28 October 2026 (check before you choose a synthesiser) The original rhasspy/piper repository page says it was archived by its owner on 6 October 2025 and that development moved to OHF-Voice/piper1-gpl; it lists an MIT licence. The fork's page lists GPL-3.0 and says the Open Home Foundation is looking for maintainers. I did not read the licence texts or the release history, and a voice model has its own licence, separate from the code. If you ship a synthesiser under GPL terms, read the licence yourself before you choose it. :::

Deployment, if you do it: a Dockerfile with a small CPU recogniser, push-to-talk clips under 15 seconds, HTTPS (browsers will not open a microphone on plain HTTP), a rate limit, no audio stored. I did not deploy this project or check any host's limits.

Observability and failure analysis

A voice turn has many places to fail, so every turn writes one record: endpoint time, redacted transcript, language guess, retrieved FAQ id, first-sentence time, barge-in events and total. The record holds no audio, and redaction runs before the transcript is written.

Three failures are worth telling in the write-up, and each one has a before and an after.

The first is the default budget. My first defaults put the time to first audio at 1,430 ms against 1,200 ms, so every default turn degraded to one sentence. The degrade path works as designed, the table shows which lever moves the number, and a test asserts that the default exceeds the budget so nobody changes it quietly.

The second is the scope check. Raw word overlap answered "how do I bake a cake" from the booking entry; removing function words took out-of-scope refusals to 15 of 15 on the English set, a development result because the threshold was set on the same items, and the Hinglish probe above shows how little it carries over.

The third is the endpoint. A 300 ms hangover would have split the 400 ms pause in my fixture. I kept 450 ms, accepted the delay on every turn, and wrote the trade-off into the README.

How to present it: README numbers, the pitch, the resume bullet

Every number in the README needs one of three labels: measured on recordings, simulated from assumptions, or scripted for a plumbing test. The results table has the latency waterfall, marked simulated; the scope counts, marked dev only; the image gate, marked synthetic; and the WER line, which says "not measured, no recordings" until you record them.

The 90-second pitch (ILLUSTRATIVE: use only numbers your repository prints) has five beats. The problem: callers at a clinic who want an answer in seconds, in Hinglish. The decision that mattered: a hangover that protects the caller from being cut off, and the cost that comes with it. The measurement: the simulated budget and what each lever does. The failure: the default configuration exceeded its own budget, and the degrade path is what handles it. The open item: the WER on real Hinglish, which needs recordings.

The resume bullet follows the topic's formula, and it must only claim what a file in the repo shows. An ILLUSTRATIVE example, not a claim: built a voice FAQ assistant (Python standard library, offline endpointing and barge-in tests on a simulated clock) with grounded answers and refusal of out-of-scope questions (15 of 15 on a 30-item dev set); latency budget enforced with a degrade path, arithmetic shown per stage; WER functions checked by hand-computed edits, with no speech accuracy claimed.

The five interview questions, with outlines:

Pitfalls and stretch goals

The pitfalls are the ones this project avoided by testing, and the stretch goals need recordings or a real model. Testing only on clean tones. Waiting for the full answer before speaking. An endpoint so aggressive it cuts callers off. Quoting a vendor's WER as your own. A microphone the browser refuses on plain HTTP. Latency tests that use the real clock and flake on a busy machine.

Stretch goals, none done: a two-threshold endpointer, streaming partial recognition, telephony through a webhook, speaker separation, tool calls inside the turn (such as booking). I would do the endpointer next, because its trade-off is the one a caller feels.

How to say it in the interview

A voice assistant is three slow components in a row, so I measure each one and I budget the whole. I cannot test real speech offline, so I do not pretend to. What I can test offline is the timing logic, the endpointing, the cancellation, the privacy of the audio buffer, and the way the answer is cut into sentences. The scope check is tried on English, Hinglish and Devanagari questions, and it fails on the last. I say which numbers are simulated, which are the arithmetic of my own assumptions, and which need a recording.

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons