Cruxly

Cruxly › AI engineering › Inference Serving

Mixture-of-Experts and Long-Context Serving

671 GB of weights for a 37B-active model, 11 requests at 128K tokens, and what a 1M-token prompt costs before the first output token

Open the interactive lesson

Prompt: "We want to self-host a 671B-parameter mixture-of-experts model that the paper calls a 37B-active model, and product wants a 128K context window, with a stretch goal of 1M tokens. How many GPUs does it need, why is an MoE cheaper per token but not smaller in memory, what does a long-context request cost, and would you use a long context or RAG for a 300-page manual?"

40 minutes first. Compute weight memory and active compute for the MoE, the KV cache for one 128K request, and the prefill time for a long prompt, with every hardware figure labelled as an assumption. The trap is reading "37B active" as "fits like a 37B model".

MoE in one picture: total parameters set memory, active parameters set compute

A mixture-of-experts layer replaces one large feed-forward block with many smaller ones (the experts) and a router that picks a few per token. In a dense transformer every token uses every parameter. In an MoE, the router scores the experts for each token and sends it to the top k; the others are skipped for that token. Transformers and Attention, From the Whiteboard defines the block this replaces. Switch Transformers (Fedus, Zoph and Shazeer, January 2021) simplified routing and described the training instability and communication costs that had limited MoE; Mixtral (Jiang et al., January 2024) made the design familiar to practitioners.

Read the spec as two numbers. Total parameters are all the weights that must exist in memory. Active parameters are the ones used for a single token. Compute per token scales with active, memory with total.

:::key Verified October 2026 Mixtral 8x7B (arXiv 2401.04088, 8 January 2024): 8 experts per layer, 2 chosen per token, about 47B parameters accessible in total and about 13B active per token; the paper reports it matching or outperforming Llama 2 70B and GPT-3.5 on the benchmarks it evaluated (authors' own comparison). DeepSeek-V3 (arXiv 2412.19437, December 2024, revised February 2025): 671B total parameters with 37B active per token, using multi-head latent attention and DeepSeekMoE. Its released config (retrieved October 2026) shows 61 layers, 256 routed experts plus 1 shared expert, 8 routed experts chosen per token, and an FP8 block-quantised checkpoint. :::

The arithmetic for DeepSeek-V3 at 8-bit. Weights: 671B x 1 byte = 671 GB. Compute per token is about 2 x 37B = 74 GFLOP, against 2 x 671B = 1,342 GFLOP if every parameter were used: 18 times less compute per token than a dense model of the same total size. Memory is not reduced at all. Fitting: eight 80 GB GPUs are 640 GB, so the weights alone do not fit. Eight GPUs of assumed 141 GB each (H200-class) are 1,128 GB, leaving 1,128 - 671 = 457 GB for the KV cache, activations and workspace. Otherwise you need more GPUs, a 4-bit format (Quantization Methods Compared: INT8, GPTQ, AWQ, FP8, FP4 and KV-Cache Quantization; NVFP4 would put it near 380 GB at 4.5 bits, 671B x 4.5 / 8), or several nodes. Mixtral at bf16: 47B x 2 = 94 GB of weights for the compute of a 13B model; a dense 13B model needs 26 GB.

Why experts exist at all. For the same compute per token, more total parameters store more capacity; the price is memory, communication and a harder serving problem, covered next.

Load balancing is a training problem that becomes a serving problem. If the router sends most tokens to a few experts, the others learn little and the hot ones become bottlenecks. Training adds a loss term (the Switch paper describes an auxiliary balancing loss) to spread tokens. At serving time the distribution of experts used depends on the traffic, so a code-heavy workload can load different experts from a chat workload.

Serving an MoE: all experts resident, the batch-size effect, all-to-all and stragglers

An MoE is cheap per token in compute but not in memory traffic, and the difference depends on batch size. Decoding reads weights from memory every step. A token only needs its chosen experts, but a batch of tokens collectively chooses many.

How many experts does a batch touch? Assume, for a clean estimate, that routing is uniform and independent. The chance that a particular expert is chosen by none of B tokens is (1 - k/E)^B, so the fraction of experts touched is 1 - (1 - k/E)^B.

Real routing is not uniform, which makes the numbers an approximation (skew means fewer experts touched, but also hot spots). The consequence is the interview point: at batch 1 the step reads only the active slice (about 37 GB for V3 at 8-bit), so MoE decode is bandwidth-cheap; at batch 128 it reads almost all 671 GB. So the throughput gain from batching is smaller than for a dense model, where one weight read serves every token in the batch. MoE serving wants large batches across many GPUs so that each GPU's expert slice is used by many tokens per read.

Expert parallelism spreads experts across GPUs. Each GPU holds a subset of experts. After the router decides, tokens must travel to the GPUs that hold their experts, and the outputs must come back: two all-to-all exchanges per MoE layer. Compute the traffic for a Mixtral-shaped model (hidden size 4,096, 32 layers, top-2, bf16) at a batch of 256 tokens: 256 tokens x 32 layers x 2 experts x 2 directions x 4,096 x 2 B = 268 MB per step. At an assumed 400 GB/s of effective all-to-all bandwidth inside an NVLink-connected node that is 0.67 ms, small; across an assumed 25 GB/s network it is 10.7 ms, about three times the 3.6 ms that reading all of Mixtral's 94 GB of weights takes on 8 GPUs at an assumed 3.3 TB/s each (94 / 26.4). That is why large MoE deployments keep expert parallelism inside fast interconnect domains, and why Autoscaling a GPU Fleet Without Burning the Budget treats the replica shape as a unit that scales as a whole.

Load imbalance makes the slowest GPU the step time. Suppose 64 tokens each pick 2 of 8 experts, one expert per GPU: 128 assignments, ideal 16 per expert. If the hottest expert receives 32 (twice the average), that GPU takes twice as long and the layer waits for it, so utilisation in that layer is 16 / 32 = 50 percent. Mitigations: replicate hot experts, rebalance placement from observed routing statistics, and keep the batch large enough that averaging helps. Monitor per-expert token counts the way you monitor per-GPU utilisation.

Other serving consequences. Quantisation helps MoE twice, since expert weights are most of the model. KV cache is governed by the attention design, not the experts: DeepSeek-V3 uses multi-head latent attention, discussed next. Prefill is compute-heavy and can be run on separate resources from decode. The rest of the serving stack, such as continuous batching (Continuous Batching and the Scheduler), is unchanged in kind.

How to answer "how do you serve a 600B MoE". Size weights at your precision (671 GB at 8-bit), choose a GPU count that leaves KV headroom (8 x 141 GB assumed leaves 457 GB), put expert parallelism inside the fast interconnect domain, run large batches, and monitor expert load and all-to-all time. If it does not fit, quantise or add nodes, and say you would measure rather than assume.

Long context: what a 128K or 1M-token request costs

Long context is a cost in three places: KV memory that grows linearly, prefill compute that grows faster, and quality that does not improve with length. Reuse the 70B GQA model from GPU Memory Math: Weights, KV Cache, and What Actually Fits: 327,680 bytes of KV per token (80 layers, 8 KV heads, head dimension 128, fp16).

KV memory. One 128,000-token request holds 327,680 x 128,000 = 41.9 GB. With a 490 GB KV pool (8 x 80 GB minus about 150 GB of weights and overhead, from that lesson) that is 490 / 41.9 = 11.7, so 11 concurrent requests. At 8,000 tokens a request is 2.62 GB in decimal units (327,680 B x 8,000) and the pool holds about 187, the same figure as the memory lesson. Both lessons use decimal units; mixing in binary units (327,680 B is 0.3125 MiB, so 2.44 GiB per request) moves the answer by about 5 percent, so pick one unit and keep it. At 1M tokens one request needs 327,680 x 1,000,000 = 327.7 GB: the pool holds exactly one, and nothing else runs. Quantising the cache to 8 bits halves each figure; paging does not reduce it, it only removes waste. The architecture matters most: DeepSeek-V2 (May 2024) reports that multi-head latent attention cuts its KV cache by 93.3 percent relative to its predecessor. With V3's config values, if the cache stores the 512-dimensional latent plus the 64-dimensional rotary key per layer in bf16, that is (512 + 64) x 61 layers x 2 B = 70 KB per token, or 9.0 GB for 128,000 tokens, and the assumed 457 GB pool of the 8 x 141 GB configuration would hold about 50 such requests. That is about 4.7 times less KV per token than this 70B GQA model (327,680 B against 70,272 B). The per-token figure is our arithmetic from the published config (61 layers, kv_lora_rank 512, qk_rope_head_dim 64), an estimate rather than a reported number.

Prefill compute. Linear layers cost about 2 x parameters FLOPs per token. Attention costs about 4 x n x d_model per token per layer for full attention over n previous tokens (halved on average by the causal mask), so it grows with context. For the 70B model (d_model = 8,192, 80 layers):

FlashAttention and Why Attention Is a Memory-Traffic Problem is what makes these prompts runnable at all (memory stays linear), but it does not remove the quadratic FLOPs. Real long-context systems add sparse or windowed layers, chunked prefill (Continuous Batching and the Scheduler), context-parallel prefill across GPUs, and prefix caching so repeated prefixes are not recomputed.

The price of a request. With an assumed input price of $3 per million tokens (use your provider's current price), a 1M-token prompt is $3.00 per request. Twenty follow-up questions, each re-sending the document at an assumed 90 percent cache-read discount, cost 3.00 + 20 x 0.30 = $9.00. Without caching it is 21 x 3.00 = $63. Prompt Caching: Paying for the Same Tokens Once explains why the cached case needs a stable prefix and a warm cache.

Quality does not scale with window size. Liu et al., Lost in the Middle (July 2023), found that models used information unevenly across long inputs: performance was best when relevant content sat at the start or end and dropped when it was in the middle, even for models built for long contexts. Later models have improved on this, so treat it as a failure mode to test for, not a law. A claimed 1M window says what fits, not what the model uses well; test with your documents and put the question and key evidence where they are least likely to be lost.

Extending, managing and choosing: RoPE scaling, windows and sinks, long context versus RAG

A model trained on 4K-token sequences cannot simply be run at 16K. Rotary position embeddings encode position as rotation angles, and angles never seen in training behave unpredictably. Position interpolation rescales positions so the longer sequence maps back into the trained range: with a factor of 4, position 16,000 is treated as 4,000. The cost is resolution, since neighbouring tokens now differ by a quarter of the angle they did in training. YaRN (Peng et al., August 2023) refines this: it interpolates by frequency band ("NTK-by-parts": high-frequency dimensions, whose wavelength is short compared with the context, are left alone, low-frequency ones are interpolated like position interpolation, with a linear ramp between; the paper uses thresholds 1 and 32 for Llama) and scales the attention logits by a temperature, sqrt(1/t) = 0.1 ln(s) + 1 for scale factor s. The authors report needing about 10 times fewer tokens and 2.5 times fewer training steps than previous methods for context extension (their claim, from the abstract and training comparison). Practical rule: scaling changes the model's behaviour, so a long-context variant is a separate checkpoint to evaluate, short-context quality included.

Windows and sinks trade recall for bounded memory. A sliding window keeps only the last w tokens' KV, so memory is constant and attention is cheap, but information outside the window is gone. Xiao et al., Efficient Streaming Language Models with Attention Sinks (September 2023), found that simply dropping old KV fails once text exceeds the cache, and that keeping the first few tokens' KV (the "attention sinks") plus the recent window largely restores stability; they report stable language modelling up to 4 million tokens without fine-tuning (their evaluation). That stabilises streaming; it does not give recall of old content, so a streaming chat is not the same as a model that can look up a fact from the first page.

Management techniques that work with any model.

Long context or RAG for a 300-page manual? Do the arithmetic with assumed figures: 650 tokens per page gives 300 x 650 = 195,000 tokens; input at an assumed $3 per million.

RAG is about 65 times cheaper than uncached long context and 6.5 times cheaper than cached, and also faster to first token and easier to cite. Long context wins when the task needs the whole document at once (summarise, compare two contracts, find inconsistencies across chapters), when the question is hard to retrieve for, or when the corpus is small and static enough to cache. A common design uses both: retrieve to narrow, then give the model whole sections instead of fragments.

The interview answers in two sentences each. "An MoE is cheaper per token because compute follows active parameters, but all experts stay in memory, so it needs the memory of the total and large batches to be efficient." "A long window costs linear KV, quadratic-ish prefill and uneven recall, so I default to retrieval and use long context when the task needs the whole document."

How to say it in the interview

"In a mixture-of-experts model a router sends each token to a few expert networks, so total parameters set the memory I must hold and active parameters set the compute per token. A 671B model with 37B active needs about 671 GB at 8-bit, more than eight 80 GB GPUs, but fits on eight 141 GB GPUs (an assumed H200-class configuration) with about 450 GB left for KV cache. At small batch each step touches few experts and is cheap; at large batch nearly all experts are touched, so bandwidth per step approaches the full model, and expert parallelism adds all-to-all traffic and load imbalance. Long context is a separate cost: KV grows linearly, 41.9 GB for one 128K request on the 70B GQA model, so a 490 GB pool holds about 11, and naive prefill is quadratic. For a 300-page manual I would default to retrieval for cost and precision and use long context for tasks that need the whole document at once."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons