Cruxly › AI engineering › AI System Design Case Studies

Case Study: An IDE Code Assistant (Completion, Chat and Agent Mode)

decode is the cost, a 160 ms completion budget, and why acceptance rate alone lies

Prompt: "Design a code assistant inside an IDE with three features: inline completion as you type, a chat panel that knows the repository, and an agent mode that makes multi-file changes. The company has 5 million completion requests a day and monorepos of up to 2 million lines. Completion must feel instant. How do you build the context, keep it fast, measure quality and protect customer code?"

40 minutes first. Say that these are three products with three budgets (hundreds of milliseconds, seconds, minutes), then do the completion latency arithmetic on paper. The trap is to design one pipeline for all three, or to say "use a big model with RAG" without noticing that completion is decided by decoding speed, not by prefill.

Three products, three budgets

Completion, chat and agent mode share a brand and almost nothing else: their latency budgets differ by three orders of magnitude, and so do their models, contexts and failure costs.

	Inline completion	Chat	Agent mode
Budget	about 150-250 ms to show	first token about 1-2 s	minutes, background
Model	small, fill-in-the-middle	large, instruction-tuned	large, with tools
Context	1-2k tokens, current file	8-30k, retrieved	grows to 50-100k+
Failure	an ignored grey suggestion	wrong answer the user reads	wrong edits across files
Gate	latency, acceptance	helpfulness, groundedness	tests, sandbox, approval

Fill-in-the-middle matters for completion. The editor sends the code before the cursor (prefix) and after it (suffix); the model is trained to generate the missing middle given both, so a completion inside a function respects the lines below. A prompt that only sends the prefix wastes the suffix, which is half the information.

Requests per second. 5 million completion requests a day over an 8-hour working window is 5,000,000 / 28,800 = 174 per second, and about 350 per second at a 2x peak. Without debounce it would be far worse: a developer typing 25 keystrokes in a burst would trigger 25 requests, but with a short debounce (wait for a pause of about 75 ms) and cancellation we assume those 25 keystrokes produce 8 requests, a 1 - 8/25 = 68 percent reduction in traffic before any model is touched.

Which model, where. Completion runs on a small code model (a few billion parameters) because serving it is a latency and cost problem, as GPU Memory Math: Weights, KV Cache, and What Actually Fits and Continuous Batching and the Scheduler describe. Chat can route to a frontier model, since a second of delay is acceptable and its quality matters. Agent mode is the Coding Agents and Computer-Use Agents: Harness, Sandbox and Verification lesson with a product around it. Saying "I would not use one model for all three" is worth a point on its own.

State the product risk early. A wrong completion costs a keystroke to dismiss, but a wrong accepted completion that compiles is a latent bug, and a wrong agent edit across ten files costs a code review. The more autonomy, the more of the budget goes into verification rather than generation.

The completion latency budget: decode is the remaining cost

Do the latency arithmetic before choosing anything, because it shows that prefill is not the bottleneck for completion; decoding is. TTFT, TPOT, and the Latency Budget defined the two parts: time to first token is dominated by prefill, and each further token costs one decode step.

Start with the naive setup. A small model with a 1,500-token context: prefill 60 ms, decode at 150 tokens per second, output 40 tokens (a typical multi-line completion):

60 + 40 / 150 x 1000 = 60 + 267 = 327 ms of model time, before network. Too slow for a target near 200.

Cache the file prefix. Most of the 1,500 tokens (the earlier part of the open file, the imports) is identical to the previous keystroke's request. With prefix caching in the serving engine, only the new suffix tokens are processed, and prefill falls to about 10 ms: 10 + 267 = 277 ms. It helped, but 267 ms of the 277 is decode. This is the point of the exercise: after caching, decode is the cost.

Attack decode three ways.

A faster model: at 400 tokens per second, 40 tokens take 40 / 400 x 1000 = 100 ms, total 10 + 100 = 110 ms. A smaller or quantised model (see Quantization Methods Compared) gets there at some quality cost, which you measure.
Shorter output: stop at the end of the line or block. Most accepted completions are one line. At about 15 tokens and 150 tokens per second, decode is 15 / 150 x 1000 = 100 ms and total again 110 ms. Stopping early also improves acceptance, because short suggestions are right more often.
Speculative decoding: a small draft proposes several tokens and the main model verifies them in one pass. The original paper, "Fast Inference from Transformers via Speculative Decoding" (Leviathan et al., arXiv 2211.17192), reports 2x-3x speedups on T5-XXL with identical output distribution. Code is repetitive and predictable, so acceptance of drafted tokens is typically high; if we assume a 2x speedup, 40 tokens take about 133 ms and the total is 10 + 133 = 143 ms. The speedup depends on the model pair and the text, so measure it on your traffic; TTFT, TPOT, and the Latency Budget covers where it fits.

Build the whole budget, including what is not the model. Network round trip 30 ms, queue wait 10, prefill 10, decode 100, render 10: 160 ms. Within about 200 ms the suggestion feels attached to typing. Add the debounce as a separate human-perceived delay.

Cancel on keystroke. When the user types another character, the earlier request's result is stale. Cancel it in flight: the server abandons decoding and frees the batch slot. This saves decode work (not the prefill already spent), which matters at 350 requests per second. Also drop results that arrive after the cursor has moved.

Capacity, with an assumption. At the 350 requests per second peak, uncached prefill would be 350 x 1,500 = 525,000 tokens per second; with the file prefix cached and about 300 new tokens per request it is 350 x 300 = 105,000. On an assumed 25,000 prefill tokens per second per GPU that is 4.2 GPUs versus 21, so caching changes the fleet by a factor of five. Substitute your measured number.

Context building: what goes into the first 1,500 tokens

Context quality decides completion quality more than model size does, and it is a ranking problem under a token budget. Spend the tokens on the evidence most likely to determine the next lines.

Completion context (about 1,500 tokens), in priority order.

The code immediately before the cursor (about 600 tokens) and after it (about 300). Non-negotiable.
The imports and the signature of the enclosing function and class (about 150), so types and names are known.
Definitions of symbols used near the cursor (about 250): resolved by a language server or a symbol index, such as the signature of a function being called. These beat embedding search for exactness.
Snippets from other open tabs and recently edited files (about 200), chosen by similarity to the code around the cursor, because developers edit related files together.

Chat context (8k tokens or more) adds the repository. For a 2-million-line monorepo, count first: at an assumed 10 tokens per line that is 20 million tokens, which will never fit in a prompt. Chunked at 300 tokens it is about 66,700 chunks, so the index is small (66,700 x 1,024 dimensions x 4 bytes is 0.27 GB) and costs under a dollar to embed at $0.02 per million tokens. Retrieval, then, is cheap; what is hard is what to retrieve. Combine three signals, as in Hybrid Search, Rerankers and Late Interaction:

Lexical: identifiers and file names the user typed, matched by BM25. Code questions are full of exact names.
Semantic: embeddings of function-level chunks cut along syntax-tree boundaries (see Chunking and Document Parsing, since splitting mid-function ruins retrieval).
Structural: the symbol graph (callers, callees, imports, tests of this file), expanded one hop from the retrieved hits.

Then rerank and pack the best 6-10 snippets with their file paths and line numbers. Keep the index fresh per git commit by hashing files and re-embedding only changed ones (the incremental pattern from Ingestion Pipelines), and key the index by branch.

Agent mode uses tools instead of a prebuilt context. It searches, reads files and runs tests as it goes, so its context is built by the loop. That makes its bill a function of steps. Take 30 steps where the context starts at 10,000 tokens and grows by 2,000 each step to 68,000 at the end; the model is read sum = 1,170,000 input tokens over the run. At $2 per million uncached that is $2.34. With prompt caching, where each step reads the earlier context at $0.10 per million and writes only the 2,000 new tokens at $2.50 per million, input is about $0.28, and output at 500 tokens a step adds 30 x 500 x $10 / 1M = $0.15. A 30-step run is about $0.43 cached against $2.49 uncached; see Prompt Caching: Paying for the Same Tokens Once (prices are the October 2026 list prices quoted in the previous case study).

Evaluation: acceptance rate is necessary and misleading

Acceptance rate is the first metric every team ships and the easiest to game, so pair it with measures of what happened to the accepted text. If the assistant only suggests the end of a line it will have a high acceptance rate and save almost no typing.

Online metrics.

Shown rate and acceptance rate: of 100 suggestions shown, how many were accepted? Show only when confidence is high enough, and count suggestions that were requested but not shown, or the rate looks better than it is.
Persistence: how much of an accepted suggestion remains in the file after 5 or 30 minutes? Suppose 30 of 100 shown suggestions are accepted, but 12 of them are rewritten or deleted within 5 minutes. Persistence is 18 / 30 = 60 percent, and the "effective" accepted count is 18, not 30.
Characters or lines contributed per user-hour, and the dismissal and edit-immediately rates.
Latency percentiles: p50 and p95 time to show, because a p95 of 800 ms is a visibly broken product for one request in twenty.
Downstream: build failures after acceptance, and revert rates (for example whether the commit containing assisted code is reverted).

Offline metrics.

Exact match and edit similarity against held-out lines from the repository: cheap, regression-friendly, and blind to correct alternatives.
Execution-based: pass@1 on tasks drawn from the customer's own repo, with its tests as the oracle. This is the one that tracks correctness, and building it per repo (mask a function body, ask the model to rewrite it, run the tests) gives a private benchmark that cannot be in training data.
Regression set of real bugs: from the issue tracker, tasks with a failing test, to see whether chat and agent mode resolve them.

:::key Verified October 2026 SWE-bench (Jimenez et al., arXiv 2310.06770, retrieved October 2026) contains 2,294 tasks from 12 popular Python repositories, and the original paper reported that its best model, Claude 2, resolved 1.96 percent of issues. Scores on the benchmark have changed enormously since; this figure only shows how the benchmark started. Caveats to state in an interview: public benchmark repositories are widely present in training data (contamination), 12 Python repositories are not your monorepo, resolve rates depend on the agent harness as much as the model, and a patch passing the tests may not match what a maintainer would accept. Use public benchmarks to shortlist models and a private repo benchmark to decide. :::

Why a private benchmark. Run it nightly against every model and prompt change, as an Evaluation Gates That Actually Block a Bad Deploy gate. For 200 tasks, a pass@1 of 60 percent has a standard error of sqrt(0.6 x 0.4 / 200) = 3.5 points, so a 2-point change is noise; use the same tasks for both versions (paired) and repeat runs, because sampling makes pass/fail non-deterministic.

Privacy, agent-mode safety, and the half-the-cost answer

Enterprises buy a code assistant on trust before they buy it on quality, so state the data promises as design requirements. These are properties you build, not facts about any vendor.

No training on customer code by default, enforced contractually and technically (separate pipelines, no retention of prompts beyond a short debug window, an opt-in for any data use).
Secret filtering: scan the context for API keys, tokens and private-key blocks before the prompt leaves the machine, and exclude files like .env and anything matched by an ignore list. A secret leaked into a prompt is a secret in logs.
Data residency: a deployment option that keeps prompts and any index in the customer's region or VPC, which for Indian enterprises may mean in-country processing. Confirm the legal requirement with counsel.
Licence awareness: an option to detect when a suggestion matches public code verbatim over a threshold and to show the licence or suppress it.
Index privacy: the repo embedding index inherits the repo's own access control, so a developer never retrieves code from a repository they cannot read (the same permission problem as the enterprise case study).

Agent mode: sandbox, approvals, oracle. Coding Agents and Computer-Use Agents sets out the harness; the product decisions are these. Run edits and commands in an isolated sandbox with no production credentials and an egress allow-list, so a prompt injection hidden in a README or a dependency cannot exfiltrate code. Require approval for risky actions (installing packages, network access, deleting files, running unknown scripts) and show changes as a reviewable diff, never silently applied. Protect the test oracle: if the agent can edit the tests it is judged by, it can make them pass by deleting assertions, so the tests it must satisfy are read-only or checked by diff against the base. Put a step and token budget on every run, as in Cost Attribution and Runaway Prevention, and stop on repeated identical failures.

If asked to cut the cost in half. Order by risk:

Debounce and cancel harder: it removes requests the user never sees. If another 20 percent of requests are cancelled before decoding finishes, decode work falls by about that much.
Cache more: the file prefix for completion, the repo instructions for chat, and the whole conversation for agent mode (the 30-step run above falls from $2.49 to $0.43).
Shorter completions and stop sequences: fewer tokens decoded and higher acceptance.
Route chat: simple questions ("what does this function do") to a mid-size model, with the frontier model for design questions, tracking helpfulness per route.
Do not cut: the secret filter, the oracle protection, or the private benchmark.

How to say it in the interview

"These are three products. Completion has a budget of about 160 milliseconds, so it uses a small fill-in-the-middle model, a context of 1,500 tokens, a cached file prefix, debounce and cancel on keystroke, and a stop at the end of the line. The arithmetic: a 40-token completion at 150 tokens a second is 267 milliseconds of decoding alone, so prefill is not the problem; I fix decode with a faster model, shorter outputs and speculative decoding. Chat retrieves over a symbol-aware index of the repo, and agent mode runs tools in a sandbox with approvals and a test oracle it cannot edit. I measure acceptance together with persistence of accepted text, plus pass-at-one on the repo own tests, and I treat public benchmarks as caveated. Customer code is never used for training, secrets are filtered before the prompt leaves the machine, and enterprise data residency is a deployment option."

Checklist

Separated completion, chat and agent mode with a budget, model and gate for each
Did the completion latency arithmetic with prefill and decode separately
Used a fill-in-the-middle prompt with prefix and suffix
Cached the file prefix, debounced requests and cancelled stale requests on keystroke
Cut decode with a faster model, end-of-line stop or speculative decoding, and measured each on real traffic
Ranked context by priority within a token budget, using symbol resolution before embedding search
Combined lexical, semantic and structural retrieval over syntax-aware chunks and indexed per branch
Tracked persistence and latency percentiles next to acceptance rate
Built a private test-based benchmark and treated public leaderboards as shortlisting only
Filtered secrets client-side and committed to no training on customer code
Ran agent mode in a sandbox with egress limits, approvals, diff review and a read-only oracle
Put step and token budgets on agent runs and cached their growing context

Practice this interactively