Cruxly

Cruxly › AI engineering › AI-Native Interfaces

Multi-Turn Conversation State and Context-Window Limits

the conversation the user experiences is not the one the model sees

Open the interactive lesson

Prompt: design the conversation-state management for a multi-turn AI chat interface. The user experiences an unbroken, scrollable conversation; the model has a finite context window that the full history will eventually exceed, and the interface must keep working — coherently — once it does.

40 minutes first. This is the exercise where the gap between "what the user sees" and "what the model receives" stops being an implementation detail and starts being the whole design problem.

Scope, and the number that forces every other decision

In scope: turn history storage, context-window budget management, what happens when history exceeds the budget, and how the interface represents any resulting loss of context. Out of scope (say it): the model's actual reasoning quality, multi-user shared conversations, voice input.

With 200K–1M-token context windows, a plain 40-turn support chat fits easily — so the reason to manage context is no longer mainly the hard limit. Three things still force it during normal use:

So the interface needs a real answer for what happens as a session grows, not an implicit assumption that conversations stay short.

The frontend's naive mental model — send the whole history every turn — breaks at exactly the point a user is most invested in the conversation continuing. Silently truncating from the oldest messages when the budget is exceeded means the model gradually "forgets" earlier context with no signal to the user about what was lost or when, which is the single most common way multi-turn AI chat interfaces produce answers that contradict something the user said ten messages ago.

Truncation, summarization, or retrieval — three real strategies, not one default

Naive truncation (drop the oldest messages) is the cheapest option and the most honest about its own limitation, if it's surfaced. It's simple to implement and simple to reason about, but it means specific, possibly important early context (a stated constraint, a name, a decision) is gone with no warning. This is acceptable only if the interface tells the user when it happens — silent truncation is the failure mode, not truncation itself.

Rolling summarization compresses old turns into a shorter summary that's kept in the context window instead of the original messages. This preserves more effective history per token than raw truncation, at the cost of a summarization step that can itself introduce errors or drop nuance — a summary is a lossy compression, and a detail the user assumes is still "in the conversation" may have been paraphrased away or omitted by the summarization step itself.

Retrieval-based context (index the full conversation, retrieve only relevant past turns per query) scales to arbitrarily long conversations without the model needing all of it every turn — the same retrieval principle covered elsewhere in this track, applied to a conversation's own history instead of an external document set. This is the most capable option and the most complex: it needs the same relevance-ranking discipline as any RAG system, and a bad retrieval (missing the one old message that was actually relevant) is a harder failure to notice than truncation, because nothing visibly signals that anything was dropped.

What the user sees versus what the model receives — closing the gap honestly

The interface must represent the model's actual working context, not just the user's full scrollback, whenever they diverge. If turns 1-15 were summarized away and the model is now working from a compressed summary plus turns 16-40, a user scrolling back to turn 3 and referencing it assumes the model still has that detail verbatim — and if it doesn't, the resulting answer will feel like the assistant "forgot," which reads as a bug even though it's the predictable result of an undisclosed design choice.

A lightweight, non-intrusive signal at the point context was compressed is the fix — not a disruptive warning on every message, but a marker in the scrollback ("earlier messages summarized") that a user can expand if they want to see what the model currently has versus what they originally said. This is the same "make the invisible mechanism visible, calmly" principle as showing retrieval sources in a RAG interface elsewhere in this track, applied to the conversation's own history instead of external documents.

Editing or regenerating an earlier turn invalidates everything context- managed after it, and the interface has to decide what that means. If a user edits their message from ten turns ago, does the summary (built from the old version) get invalidated and rebuilt? Silently continuing with a stale summary built from text the user just changed is a correctness bug with no error message — deciding and stating this behavior (rebuild context from the edit point forward, at a real cost) is part of the design, not an implementation detail to defer.

The field's actual vocabulary: write, select, compress, isolate

Truncation, summarization, and retrieval are three instances of a broader framework the field now names explicitly, and knowing the names is not pedantry — it's what lets you reach for the fourth technique the three-strategies framing above doesn't cover. Context engineering in 2026 names four concrete moves: write (persist context outside the model's input window entirely — a scratchpad, an external memory store — so information survives without occupying a single token of the context sent on the next call), select (retrieve only what's relevant to the current turn, not everything available — the retrieval strategy from the previous section is an instance of select), compress (summarize to reclaim token budget — the summarization strategy above is an instance of compress), and isolate (give a sub-task its own clean context window, rather than letting it accumulate inside the main conversation's history). The three-strategies framing above is select and compress; write and isolate are the two moves a conversation-state design that stops at "truncate, summarize, or retrieve" is missing entirely.

Context rot is the measured phenomenon that makes all four moves necessary, not just a vague "the model gets confused when overloaded." Every major model family — Claude, GPT, Qwen, Gemini — shows a real, measured performance degradation as input token count grows, even on tasks that don't need more tokens to solve. A long conversation doesn't just risk exceeding the context window; it measurably degrades answer quality well before the window is full, purely from carrying tokens the current turn doesn't need. This reframes the whole lesson: managing context isn't only about fitting inside a hard limit, it's about keeping what's actually sent lean because a bloated-but-fitting context is also a correctness problem, not just a capacity one.

The numbers worth having ready — with their source and scope. In Anthropic's September 2025 internal evaluation on agentic search tasks:

Quote them as what they are: one vendor's internal eval, on agentic search, for clearing stale tool results specifically — not a general "compaction improves everything by 29%" law. They're still the concrete argument for treating write and compress as load-bearing design decisions rather than polish layered on after truncation ships.

One more design constraint: compaction interacts with prompt caching. Dropping the oldest message on every turn changes the start of the prompt every turn, which invalidates the provider's prefix cache each time. Compact in occasional batches — keep the prefix stable for many turns, then summarise a large chunk at once — so most turns still hit the cache.

Write and isolate in practice: scratchpads and sub-agent summaries

Structured note-taking is the third formalized strategy alongside compaction and sub-agent isolation, and it's the clearest instance of "write." Rather than relying on the raw conversation transcript as the only record of what happened, the agent writes its own working notes to persistent storage as it goes — a running scratchpad of what's been decided, what's pending, what a prior tool call returned. When a later compaction step runs, or when a lead agent reads back state, it's reading a curated summary the agent itself produced at the time, not asking a model to re-derive what happened by re-reading raw scrollback after the fact. This is a meaningfully different reliability property: re-deriving state from a long transcript is itself subject to context rot, while reading a note the agent wrote when the information was freshest is not.

Sub-agent isolation is "isolate" made concrete, and the lever that controls it is the size of what comes back, not just the fact that a sub-agent was used. Anthropic's own multi-agent research pattern has each sub-agent work inside a fully isolated context window — its own system prompt, its own restricted tool access, no visibility into the lead agent's full history — and return a condensed 1,000-2,000 token summary to the lead agent rather than its raw working transcript. Calling a sub-agent is not, by itself, isolation with a bounded cost — a sub-agent that dumps its entire 20,000-token transcript back to the lead agent has technically isolated the work but has not isolated the cost, and the lead agent's own context grows by the sub-agent's transcript size instead of by a controlled summary. The summary size is the actual design parameter: it's what determines how many delegated tasks the lead agent's context can absorb before it, too, starts accumulating the same rot the delegation was meant to avoid — a sub-agent's own window is isolated by construction, but the lead agent's context only stays cheap if what comes back is a bounded summary, not a raw transcript.

Follow-ups they will ask

"The user asks 'why did you forget what I said earlier?' — what does the product do?" This is the question the whole exercise has been building toward: the honest answer is a UI affordance (the expandable "context was compressed here" marker from the previous section), not a better prompt or a bigger model that pushes the problem further out without solving it. Naming that a sufficiently long conversation always eventually hits this, regardless of context-window size, is the mature answer — the exercise is designing for the inevitability, not trying to make it never happen.

"Support branching conversations — the user regenerates a response and now there are two possible continuations." This changes the data model from a linear list of turns to a tree, and context-window management now has to operate on one path through that tree, not the whole tree — the interface needs to make which branch is "active" unambiguous to both the user and whatever builds the model's context window, or a bug here silently mixes context from two different conversation paths into one model call.

"Estimate the cost impact of rolling summarization versus truncation versus retrieval at scale." Truncation costs nothing extra per turn. Summarization costs one additional model call per compression event (cheap relative to the main conversation, but non-zero at high volume). Retrieval costs an embedding call per turn plus a vector store, ongoing infrastructure rather than an occasional extra call — naming this cost shape difference, not just "retrieval is more expensive," shows the trade-off was actually reasoned through rather than recalled.

How to say it in the interview

"The user's mental model is one continuous conversation with unlimited memory. The model's actual input is a finite list of messages that has to fit in a token budget. Every decision in this exercise is about managing that gap without the user ever noticing it as a gap - until the interface has to be honest that it exists."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons