Cruxly › AI engineering › Inference Serving

TTFT, TPOT, and the Latency Budget

why an average latency number tells you nothing about a streaming endpoint

Prompt: define the SLO for a streaming chat endpoint, then say exactly what you would do — in order — when you start missing it during a traffic spike. You may not answer "add more GPUs" as the first step.

40 minutes first. The trap is answering in terms of "p95 latency," which for a streaming response is not a single number and hides the two failures users actually notice.

The four metrics, and which user complaint each one maps to

TTFT — time to first token. From request arrival to the first token reaching the client. It is queueing delay plus prefill time, so it scales with prompt length and with how loaded the server is. This is the metric that maps to "the app feels unresponsive," because it is the entire duration during which the user has no evidence anything is happening.

TPOT — time per output token (equivalently ITL, inter-token latency). The average gap between consecutive tokens once streaming has started. It is a pure decode-speed number and it maps to "it types too slowly." The target isn't reading speed — adults read only about 5–7 tokens/sec, and users skim streamed output anyway. It's perceived smoothness plus total wait: below about 40–50 ms/token the stream looks fluid and further gains are hard to notice; above about 80 ms/token it visibly crawls, and long outputs (code, agent steps that feed the next step) take noticeably longer to finish.

End-to-end latency = TTFT + TPOT × (output_tokens − 1). Worth computing but bad to alert on, because it is dominated by output length — a request that legitimately generated 2,000 tokens will always look "slow" next to one that generated 50, and an alert on total latency mostly detects verbosity.

Throughput, in output tokens/sec across the whole system, is the capacity and cost metric, and it moves opposite to per-user latency as the previous lesson established.

Metric	User says	Driven by
TTFT	"it's unresponsive"	queue depth + prompt length
TPOT / ITL	"it types too slowly"	batch size + bandwidth + model size
End-to-end	"it took forever"	mostly output length — don't alert on it
Tokens/sec	(nothing — it's your bill)	batch size, utilization

Work an example to show why the split matters. Two configurations both average 6 seconds end-to-end on a 300-token response. Config A: TTFT 400 ms, TPOT 19 ms. Config B: TTFT 3,900 ms, TPOT 7 ms. B has better decode speed and better throughput, and users will call it broken, because they stare at nothing for four seconds. An average latency dashboard rates these identically. That is the entire argument for separate SLOs, and it is the thing to say out loud first.

Percentiles, and the ones people misread

An average hides the shape of the distribution, and inference distributions are badly shaped. Prompt lengths are long-tailed, output lengths are long-tailed, queueing under load is bursty, and preemption from a full KV pool produces occasional multi-second outliers. A mean TTFT of 500 ms is perfectly compatible with 5% of users seeing 6 seconds.

Report p50, p95 and p99 separately, and know what each is telling you. p50 is the typical experience and what your demo shows. p95 is what your loudest users experience routinely — in a chat product with 20 turns per session, a user hits the p95 roughly every session. p99 is where preemption, cold starts and cross-AZ retries live.

Normalize TTFT by prompt length before comparing anything. A p99 TTFT of 3 seconds is alarming for 200-token prompts and excellent for 40,000-token ones, and an aggregate p99 across both buckets is a number about your traffic mix rather than about your server. Bucket by prompt length, or report TTFT-per-thousand-prompt-tokens alongside the raw figure.

Two measurement mistakes that make dashboards lie. First, measuring TTFT server-side from the start of prefill excludes queueing delay — which is exactly the part that degrades under load, so your dashboard stays green through the incident. Measure from request arrival, including time spent in the queue. Second, computing TPOT as (end_to_end − TTFT) / output_tokens averages away stalls: a stream that delivers 200 tokens smoothly and then freezes for 2 seconds mid-sentence has a fine average TPOT and a terrible user experience. Track max inter-token gap as its own metric to catch the chunked-prefill and preemption stalls from the previous lesson.

Queueing: where latency actually comes from under load

Past a utilization threshold, latency stops being about the model and becomes about the queue, and it goes non-linear. Standard queueing behaviour applies: as arrival rate approaches service rate, wait time grows toward infinity, not proportionally. At 50% utilization the queue is essentially empty; at 80% waits are noticeable; at 95% a small traffic increase produces a large latency increase. This is why capacity planning for an inference fleet targets something like 60-70% steady-state utilization rather than 90% — the headroom is not waste, it is the latency SLO.

Concretely: if a GPU serves 40 requests/sec and traffic rises from 32/sec (80%) to 38/sec (95%), throughput went up 19% and p99 TTFT can easily go up 5-10×. Nothing broke. That is just what a queue does, and being able to say so prevents a whole class of fruitless debugging.

Give the queue a maximum age and shed load rather than serving everything slowly. A request that has waited 20 seconds in the queue is one the user has almost certainly abandoned; computing its response burns a GPU slot that could have served someone still watching. An explicit queue timeout with a clear 503 and a Retry-After is better engineering — and better for aggregate latency — than an unbounded queue that degrades everyone uniformly.

Then shed by class, not uniformly. Interactive traffic gets priority; batch, background and retry traffic is the first to be shed or deferred. This is where the per-request-class pools from the batching lesson pay off a second time: a background summarization job queued behind interactive chat is a scheduling decision you already made.

Speculative decoding: when it pays and when it costs you

The idea exploits the fact that decode is bandwidth-bound, not compute-bound. A small, cheap draft model proposes k tokens ahead; the large target model then verifies all k in a single forward pass, because verifying k proposed tokens is one batched matrix multiply rather than k sequential ones. Accepted tokens are kept, and generation resumes from the first rejection. You are spending spare FLOPs — which decode has in abundance — to buy back memory bandwidth, which decode is starved of.

The arithmetic that decides whether it pays. With acceptance rate α and draft length k, the expected tokens per target-model pass is roughly (1 − α^(k+1)) / (1 − α). At α = 0.8 and k = 4 that is about 3.4 tokens per pass instead of 1 — call it a 2-3× TPOT improvement after subtracting draft overhead. At α = 0.4 it drops to about 1.6 tokens per pass, and once you subtract the draft model's own cost the whole thing is close to break-even or worse. Acceptance rate is the entire ballgame, and it depends on how well the draft model's distribution matches the target's on your traffic — not on a benchmark.

The critical caveat almost everyone misses: speculative decoding helps at low batch size and hurts at high batch size. Its whole premise is spare compute. At batch size 1-4 there is plenty, so it is nearly free throughput. At batch size 64 the GPU is no longer bandwidth-starved in the same way, the verification pass is genuinely expensive, and the rejected tokens are wasted compute that reduces total system throughput. So it is a latency optimization for lightly loaded or low-concurrency serving, and often an anti-optimization for a throughput-maximizing batch pool. Saying that distinction out loud is the single strongest signal available in this conversation.

And name the guarantee, because it is what makes the technique safe. Correctly implemented rejection sampling makes the output distribution identical to what the target model would have produced alone — this is not an approximation that trades quality for speed. If someone proposes "just use the small model," that is a quality trade; speculative decoding is not.

Variants worth naming — and which ones keep that guarantee:

EAGLE-style drafting (EAGLE-3 is current practice) — a small head that drafts from the target model's own hidden features; high acceptance rates, and verified with standard rejection sampling, so the output distribution is preserved.
Multi-token-prediction heads — some recent models are trained with extra heads that predict several tokens ahead, usable directly as a built-in draft.
n-gram / prompt-lookup drafting — no draft model at all; works surprisingly well when outputs quote the input heavily, as in summarization or code editing.
Watch the acceptance rule, not just the drafter. Medusa's "typical acceptance" scheme accepts tokens that are merely plausible rather than doing exact rejection sampling — it's faster, but it does not preserve the target model's distribution. If the guarantee matters, verify the engine uses exact rejection sampling.

Setting the SLO, and the ordered playbook when you miss it

Write the SLO as separate numbers with a prompt-length qualifier and a measurement point. Something you can actually hold someone to: "p95 TTFT under 800 ms for prompts under 2,000 tokens, measured from request arrival including queue time; p95 TPOT under 50 ms; max inter-token gap under 500 ms; 99.5% of requests served without a queue-timeout rejection." Note that every clause names what breaks if it is violated, and that "p95 latency under 5 seconds" appears nowhere.

Then the ordered response when you start missing it, cheapest and fastest first — this ordering is the actual answer to the prompt, and "add GPUs" is last for a reason:

Check which metric moved. TTFT degrading with TPOT flat means a queueing or prefill problem. TPOT degrading means batch size grew or preemption started. Both degrading means you are simply out of capacity.
Check KV utilization. Above ~90% means preemption is thrashing, and the fix is admission control or a lower max_model_len, not hardware.
Turn on or fix prefix caching. If TTFT is the problem and your traffic shares a system prompt, this is free and immediate — and verify no per-request value sits at the front of the template.
Enable chunked prefill if the symptom is stalls or spiky TTFT rather than a uniform rise.
Lower the batch cap if TPOT is the violated metric. This trades throughput for per-user latency deliberately.
Shed or defer non-interactive traffic by class, before degrading interactive traffic.
Quantize — 8-bit weights (FP8 on Hopper and newer) shrink memory and speed up bandwidth-bound decode; an 8-bit KV cache roughly doubles concurrency. Both gated on the eval suite.
Add speculative decoding only if you are at low batch size and can measure acceptance rate on real traffic.
Then scale out — which, as the autoscaling lesson, Autoscaling a GPU Fleet Without Burning the Budget, covers, is minutes away at best and may not be available at all.

Say the honest part too: some SLO misses are the SLO's fault. A p95 TTFT target of 300 ms on 32,000-token prompts is not achievable at any batch size, because prefill for 32K tokens simply takes longer than that. When the budget and the arithmetic disagree, the answer is renegotiating the SLO per prompt-length bucket, not tuning forever against physics.

How to say it in the interview

"For a streaming endpoint, end-to-end latency is the wrong primary metric, because a response that takes eight seconds total feels fast if the first token arrives in 300 milliseconds and then streams steadily — and feels broken if it arrives in four seconds and then dumps. So I'd set separate SLOs on time-to-first-token and on inter-token latency, and treat end-to-end as a derived number rather than the thing I alert on."

Checklist

Set separate SLOs for TTFT and TPOT instead of one end-to-end latency number
Refused to alert on end-to-end latency and said why
Measured TTFT from request arrival, including queue time
Tracked max inter-token gap, not just average TPOT
Normalized TTFT by prompt-length bucket before comparing or alerting
Explained rising latency at high utilization as queueing, not as a regression
Bounded the queue and shed by request class rather than degrading everyone
Stated that speculative decoding helps at low batch size and can hurt at high
Named acceptance rate as the metric that decides whether speculation pays
Put 'add more GPUs' last in the mitigation order

Practice this interactively