Cruxly › System design › Design Exercises

LLM-Backed Query System

the model is the one component in this system whose behavior you cannot fully specify

Prompt: design a customer-support system where a user's question is answered by a large language model, grounded in your company's documentation, running as a chat interface. The system must handle high query volume, keep answers current as documentation changes, avoid confidently answering with wrong or fabricated information, and escalate to a human when it should.

45 minutes first. Every other system in this track has components whose behavior you can fully specify. This is the first one where the core component is fundamentally probabilistic, and the entire design exists to compensate for that.

Applying the four moves: Scope — answer support questions grounded in current documentation, at volume, without fabricating, escalating to a human when unsure; general-purpose chat out. Estimate — query volume × tokens per answer, which drives both latency and a per-query cost that has no equivalent in the other exercises. Sketch — retrieval over chunked, indexed documentation, an LLM call with the retrieved context, and an escalation path. Deep dive — the core component is probabilistic, so the design compensates around it: grounding with citations, a confidence or evaluation gate before an answer ships, re-indexing on doc changes so answers stay current, and a clear rule for when to hand off.

Scope, and why 'call the LLM' is not the design

In scope: retrieval-augmented generation (RAG) architecture, keeping the knowledge base current, confidence/escalation logic, evaluation and monitoring for answer quality. Out of scope (say it): training or fine-tuning a model (this system calls a foundation model as a service, the same "unreliable, rate-limited external dependency" framing as any third-party API), the support ticketing system's own workflow, multi-turn conversation memory beyond the current session.

Sending a user's raw question straight to a general-purpose LLM produces confident, fluent, and frequently wrong answers about your specific product — the model has no knowledge of your documentation, your recent changes, or facts specific to your company, and it will still generate a plausible-sounding answer rather than admitting it doesn't know. The system's actual job is retrieval-augmented generation: find the actually-relevant passages from your real documentation, and construct a prompt that gives the model those passages as grounding, then generate an answer constrained to what's in them — this reframing, from "prompt the model" to "retrieve, then ground, then generate," is the single most important idea in the whole exercise.

The model is the one component in this design whose exact output you cannot specify or guarantee — every other decision here exists to compensate for that fact, not to make the model itself more reliable, which is not something the exercise controls.

The pipeline: retrieval, grounding, generation, confidence

1. User question arrives
2. Retrieve top-K relevant documentation chunks (vector similarity search
   over pre-computed embeddings of the doc corpus)
3. Construct a prompt: the user's question + the retrieved chunks as
   explicit context, with an instruction to answer ONLY from that context
4. Generate a response from the LLM, constrained to the provided grounding
5. Score the response's confidence (does it cite the provided context, or
   does it look like it's answering from general knowledge?)
6. High confidence: return the answer, with sources cited.
   Low confidence: escalate to a human agent rather than guess.

Retrieval quality bounds answer quality — a perfect model given the wrong documentation chunks still produces a wrong answer, because the model can only ground itself in what it was actually given. This means the retrieval step (the same embedding-based similarity search covered in this track's AI-retrieval material) is not a minor preprocessing detail; it's the component most likely to be the actual source of a wrong answer, and debugging a bad response starts with checking what was retrieved, not assuming the model reasoned poorly.

The prompt must explicitly instruct the model to answer only from the provided context, and this instruction is a real, incompletely-effective mitigation, not a hard guarantee. A model can still ignore this instruction and answer from its own general training knowledge (which may be outdated or simply wrong for your specific product) — naming that this is a probabilistic mitigation, not a hard constraint, is the honest framing an interviewer is listening for, versus implying that a system prompt fully controls model behavior.

Keeping the knowledge base current, and evaluating what 'correct' means

Documentation changes, and stale embeddings silently produce confidently wrong answers about a feature that no longer works the way the retrieved chunk describes. The retrieval index needs the same durable-pipeline discipline as any other derived index in this track (the search-autocomplete exercise's cached top-K, the payment ledger's balance cache) — re-embed and re-index on every documentation change, ideally via an event-driven pipeline triggered by the doc source's own update mechanism, not a periodic full rebuild that leaves a stale window between changes.

"Is the answer correct" cannot be checked the way a traditional system's output can — there's no exact-match assertion for a natural-language answer. Evaluation for an LLM-backed system needs its own discipline: a held-out set of question/expected-answer pairs scored by a rubric (does it cite the right source, does it avoid stating anything not in the source, is it appropriately hedged when the retrieved context is thin), run continuously against new model versions and prompt changes before they ship — the same evaluation-gates-deployment principle covered in this track's AI-governance material, here applied at the level of a full system rather than one model call.

Confidence scoring and the escalation threshold are a product decision with a real cost on both sides, not a technical detail to tune once and forget. A threshold set too low escalates too often, defeating the purpose of automation and overwhelming human agents; set too high, it lets the system confidently answer questions it should have punted on. Naming that this threshold needs ongoing tuning against real escalation-quality data, not a one-time calibration, is what separates a system that was actually operated from one that was only launched.

Follow-ups they will ask

"How do you know if the model started hallucinating more after a provider updated it silently?" This is the same "the frontend is often the first place a capability regression becomes visible" principle from the AI-frontend material elsewhere in this track — instrument user-correction signals (explicit "this didn't help" feedback, immediate follow-up questions indicating the first answer failed) and run the evaluation suite continuously against production traffic samples, not just at initial launch, because a provider-side model update is a change you didn't make and won't see in your own deployment logs.

"The system must never give legal, medical, or financial advice, even if a user asks in a way that makes the retrieved documentation seem relevant." This needs an explicit guardrail layer separate from retrieval quality — a classifier or rule-based filter that detects the category of question being asked and refuses or escalates certain categories regardless of how good the retrieval match is, because "the retrieved chunk was relevant" and "this is a safe question to auto-answer" are different judgments that retrieval quality alone cannot make.

"Scale this to handle 10,000 concurrent conversations." The retrieval step (a vector similarity search) and the generation step (a call to an LLM provider) have very different scaling properties and failure modes — retrieval is typically fast and horizontally scalable with a proper vector index, while generation is the expensive, rate-limited, latency-variable step, which pushes toward the same multi-provider failover and cost-attribution discipline covered in this track's AI-backend material, treating the LLM call as the one component whose reliability this system does not control and must design around.

How to say it in the interview

"A traditional service has a spec: given this input, produce this output. An LLM has no such guarantee - it can hallucinate, and it can be right in a way that's hard to verify automatically. So the actual system design here is not 'call an LLM API' - it's retrieval to ground the model in real facts, evaluation to catch when it's wrong, and an honest escalation path for when the system should say 'I don't know' instead of guessing."

Checklist

Reframed the system from 'call an LLM' to 'retrieve, ground, then generate'
Named retrieval quality as the likely source of a wrong answer, not the model's reasoning
Described prompt-level grounding instructions as a probabilistic mitigation, not a hard guarantee
Re-indexed the retrieval corpus on documentation change via an event-driven pipeline, not a periodic full rebuild
Built a rubric-based evaluation suite instead of assuming natural-language answers can be exact-match checked
Named the escalation confidence threshold as an ongoing tuning decision, not a one-time calibration
Monitored production traffic continuously for silent model-behavior drift from a provider-side update
Added an explicit guardrail layer for sensitive question categories, separate from retrieval relevance
Distinguished retrieval's and generation's separate scaling properties and failure modes at high concurrency

Practice this interactively