Cruxly › AI engineering › AI-Native Interfaces

Model Selection: Cost, Latency, and Capability Trade-offs

the frontend inherits every consequence of a choice it doesn't make

Prompt: your product has three AI-assisted features — an instant inline autocomplete, a document-summarization tool, and a multi-step research assistant. Choose a foundation model (or models) for each, and design how the frontend surfaces the consequences of that choice to the user.

40 minutes first. There is no single right model — the exercise is arguing the trade-off per feature and building an interface honest about what was traded away.

The three axes, and why they trade against each other

Cost, latency, and capability are not independent dials — pushing one usually costs one of the other two. A larger, more capable model is typically slower and more expensive per token. A smaller, cheaper, faster model makes more reasoning errors and has a shorter effective context window. There is no model that wins on all three axes simultaneously, which is why "just use the best model" is not an engineering answer — it's a refusal to make the trade-off explicit.

The right question per feature is not "which model is best" but "what does a wrong answer cost here, and what does a slow answer cost here." Inline autocomplete: a wrong suggestion costs one keystroke to dismiss, and a slow suggestion (anything over ~200ms) is actively worse than no suggestion, because it arrives after the user already typed past it. Document summarization: a wrong summary costs a re-read, and 2-4 seconds of latency is fully acceptable because the user expects to wait for a document-length operation. Multi-step research: a wrong intermediate step compounds into a wrong final answer that costs real time to notice, so capability dominates and 10-30 seconds of latency is acceptable if the interface sets that expectation honestly.

Reasoning models add a fourth dial inside a single model. Many current models expose an effort level per request (the portable concept; older APIs also had an explicit thinking-token budget) — the same model can answer quickly with little deliberation or spend many seconds and thousands of hidden tokens thinking. That turns part of model selection into a per-request choice: low effort for autocomplete-style features, high effort for the research feature, on the same model. The frontend has to budget for it too — high-effort requests have a long silent stretch before the first visible token, which needs the same progress narrative as a slow model.

Dated provider note (Anthropic effort docs, read 2026-10-22): on current Claude models output_config.effort (low to max) is the recommended control, defaults differ by model (medium on Opus 5.5 and Haiku 5.5, high on most others), and some models cannot switch thinking off (Opus 5.5 returns a 400 for thinking: disabled). Settings that older APIs exposed, such as an explicit thinking-token budget or sampling parameters, are restricted on some newer models, so read the current model's page rather than carrying settings over.

The frontend inherits the choice, whether or not it was told

A model swapped on the backend for cost reasons silently changes what the frontend needs to do. Moving the summarization feature from a high-capability model to a cheaper one to cut cost is a backend decision that increases the hallucination and error rate the frontend has to account for — if the interface's confidence-surfacing and source-citation design (built for a low-error-rate model) doesn't change alongside it, the UI keeps presenting the same unearned confidence for a now-less-reliable answer. Model selection and trust-surfacing design are coupled, not independent layers, and treating a model swap as invisible to the frontend is how a cost-cutting change becomes a trust incident.

Latency budget is a frontend design input, not an afterthought discovered in testing. If research-assistant answers take 15 seconds, that has to be designed for from the start — a progress narrative showing intermediate steps (the same "show the work" principle from the streaming lesson elsewhere in this track), not a spinner that makes 15 seconds feel like a hang. Choosing the model without simultaneously deciding how the wait is presented is scoping half the problem.

Say the trade-off out loud in the design, not just in a comment. "We use the cheaper model for autocomplete because latency dominates user experience there, and the fast, more capable model for research because correctness dominates there — accepting higher cost and latency where it buys down a more expensive failure mode." This sentence is the actual deliverable of a model-selection decision, not the model name itself.

Multi-model fallback within one feature, not just across providers

A single feature can use more than one model, escalating deliberately. A common pattern: try a fast, cheap model first, and escalate to a slower, more capable model only when the fast model's own output signals low confidence (a low logprob, a refusal, a response that fails a structural check) — getting the latency benefit for the common easy case without sacrificing capability for the harder cases that actually need it. This is a different mechanism from the provider-failover pattern elsewhere in this track (that escalates on provider health; this escalates on task difficulty), and conflating the two designs is a common mistake — provider failover switches because something is broken, capability escalation switches because the task turned out to be harder than expected.

The UI has to represent an escalation honestly, not hide it. If a query silently took 12 seconds because it escalated through two models before returning an answer, a user who expected the fast path's ~1-second latency has no way to know why this one was different — a lightweight "this one took a bit longer to get right" signals confidence without requiring the user to understand the internal architecture.

Follow-ups they will ask

"The product manager wants to switch the research assistant to a cheaper model to cut costs by 60%. What do you tell them?" Name the actual trade being proposed: a 60% cost cut on the feature where wrong answers compound and are hardest to notice is trading money for correctness in exactly the feature that can least afford it. The honest answer is not a flat no — it's naming what would need to change alongside the model swap (more aggressive fact-checking UI, explicit "verify this" prompts, narrower feature scope) to make the trade acceptable, rather than accepting or rejecting the request without surfacing its real cost.

"A/B test two models for the same feature — how does the frontend handle a user seeing different latency or behavior than another user?" Assign variants stickily per user (or per account), so each person gets one consistent experience for the whole test instead of flipping between models mid-conversation — that inconsistency is what users notice, and it also contaminates the measurement. Don't try to fake identical latency across variants; latency is part of what you're measuring. Where colleagues compare notes (a shared team workspace), assign by account so everyone in it sees the same variant.

"How would you detect that a model's real-world error rate is worse than expected, from the frontend?" Instrument user-correction signals (edits immediately after a suggestion, explicit "regenerate" clicks, copy-then-heavily-edit patterns) as an implicit quality signal fed back to whoever owns the model-selection decision — the frontend is often the first place a capability regression becomes visible, before it shows up in any backend eval, because users react to wrong answers before anyone runs a benchmark.

How to say it in the interview

"Autocomplete needs to feel instant, so I want the cheapest, fastest model even if it's less capable - a wrong suggestion costs the user one keystroke to ignore. The research assistant needs the opposite trade entirely, because a wrong multi-step answer costs real time to notice and undo. Picking one model for all three features means over-paying for two of them and under-serving the third."

Checklist

Chose a different model per feature based on the cost of a wrong answer versus a slow answer
Named cost, latency, and capability as trading against each other, not independent dials
Coupled a backend model swap to a corresponding change in the frontend's trust-surfacing design
Designed the latency budget and its presentation together, not the model choice alone
Stated the model-selection trade-off as an explicit sentence, not just a config value
Distinguished capability escalation (task is harder) from provider failover (something is broken)
Signaled an escalated, slower response to the user rather than letting an unexplained delay pass silently
Used the effort level as a per-request dial within one model
Assigned A/B variants stickily per user or account
Treated a proposed cost-cutting model swap as a trade to negotiate, not a request to silently accept or block

Practice this interactively