Cruxly › AI engineering › Inference Serving

GPU Memory Math: Weights, KV Cache, and What Actually Fits

why KV cache, not weights, caps your concurrency

Prompt: you have been handed 8×A100 80GB and told to serve a 70B parameter model to an internal product. How many concurrent requests can you support at an 8,000-token context? Show the arithmetic, then say what you would change to double it.

Do this on paper before reading on. Almost every serving interview opens with some version of this question, and the naive answer — "140 GB of weights fits in 640 GB of VRAM, so plenty of room" — is wrong in a way the interviewer is specifically waiting for.

Weights: the fixed cost, and what quantization actually buys

Start with the only number in this whole topic that does not move. Model weights occupy parameters × bytes_per_parameter, full stop. For a 70B model:

Add roughly 5-10% on top for activations, workspace buffers, CUDA context and fragmentation — so call fp16 70B about 150 GB in practice, not 140 GB exactly. That already rules out a single 80 GB A100 or H100 for fp16 70B, which is why 70B-class serving is a tensor-parallel deployment by default: shard each layer's weight matrices across N GPUs so each holds 140/N GB, at the cost of an all-reduce on every layer, which is why you want those GPUs on NVLink inside one node rather than across a network.

Quantization is the cheapest lever and the one with the most hand-waving around it. Going 16-bit → 8-bit halves weight memory and roughly halves the bytes the GPU must read per decode step, which — because decode is bandwidth-bound — usually makes it faster, not just smaller.

• On Hopper/Blackwell (H100, H200, B200), FP8 is the default. FP8 weights and activations (W8A8) are near-lossless on most workloads and run on native FP8 tensor cores. Blackwell adds native FP4 (NVFP4/MXFP4) for a further 2×.
• On the A100 in this prompt there are no FP8 tensor cores. The options are int8 (weight-only or W8A8 via SmoothQuant-style methods) or 4-bit weight-only (AWQ, GPTQ). Naming the hardware dependency is the signal.
• 4-bit is where quality starts to cost you. Weight-only 4-bit on a 70B is usually a small, measurable degradation — invisible on easy prompts, visible on long-chain reasoning and code. The answer is never "quantize, it's free"; it's "quantize, hold the eval suite as the gate, and go lower only if the numbers survive."

Say the trade-off direction out loud: quantizing weights does not just save weight memory, it frees VRAM for KV cache, which is what actually raises concurrency. That is the real reason to do it on a serving box, and connecting those two facts is the whole point of this lesson.

KV cache: the number that actually caps concurrency

Every token the model has already seen keeps a key vector and a value vector per layer, so it never has to be recomputed. That is the cache. Its size per token is:

bytes_per_token = 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes_per_dtype

Work it for a 70B-class model — 80 layers, 64 attention heads, head_dim 128, and (critically) 8 KV heads because it uses grouped-query attention — at fp16:

2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.33 MB per token

At 8,000 tokens of context that is 2.62 GB for one request. On 8×80 GB = 640 GB total, minus ~150 GB of weights and overhead, you have roughly 490 GB of usable KV space, so:

490 GB ÷ 2.62 GB ≈ 187 concurrent 8K-context requests

Now do the same model without grouped-query attention — 64 KV heads instead of 8 — and the per-token cost becomes 2.62 MB, so one 8K request costs 21 GB and your ceiling collapses from ~187 to about 23 concurrent requests on the same hardware. Nothing about the weights changed. GQA is worth naming by name for exactly this reason: it is an 8× concurrency multiplier hiding in an architecture diagram. Newer architectures push the same idea further: multi-head latent attention (MLA), used in DeepSeek's V2/V3 models, caches a single compressed latent vector per token instead of separate per-head keys and values, cutting KV memory several times below even GQA. When an interviewer asks how to raise concurrency for a given model, "what attention variant does it use?" is a legitimate first question.

The 7B case for contrast — 32 layers, 8 KV heads, head_dim 128, fp16: 2 × 32 × 8 × 128 × 2 = 131 KB per token. A single 80 GB GPU holds 14 GB of weights and ~62 GB of KV, which is roughly 473,000 tokens of cache — say 118 concurrent requests at 4K context each. A 7B model is not "concurrency-unlimited"; it just has a ceiling high enough that people forget it exists until a long-context feature ships.

The trap to name explicitly: KV cache scales with tokens, not requests. One request with a 128K context can consume the same VRAM as forty requests at 3K. Any capacity plan stated in "requests per GPU" without a context-length assumption attached is not a capacity plan.

Now answer the second half of the prompt: what doubles it? Rank the levers by how much KV pool they actually buy.

• KV-cache quantization is the direct 2× lever, and it's the one most candidates miss. It works on the A100 too (stored 8-bit, upcast for compute), with quality checked on the eval suite like any quantization.
• Weight quantization alone is the trap answer. It feels like "halve the memory," but weights are a fixed 140 GB slab; freeing 70 GB of a 490 GB pool is a 14% gain.
• Replica shape matters too. One TP=8 replica vs. two TP=4 replicas is a real throughput trade — each TP=4 replica fits (140 GB of weights in 320 GB), and two independent replicas avoid some all-reduce cost, but they hold two copies of the weights (~300 GB instead of ~150 GB), so the total KV pool shrinks from ~490 GB to ~340 GB unless the weights are quantized.

Fragmentation, and why PagedAttention was the fix

The pre-PagedAttention design reserved KV cache contiguously, for the maximum possible sequence length, at request admission time. If your server advertised an 8,192-token maximum, every admitted request got a 2.7 GB contiguous block reserved up front — even the request that turned out to generate 40 tokens and finish. Measured waste in that regime routinely ran 60-80% of the KV pool: internal fragmentation (reserved-but-unused tail of each block), external fragmentation (gaps between blocks too small for the next request), and reservation for speculative growth that never happened.

PagedAttention borrows the operating system's answer to exactly this problem. Split the KV cache into fixed-size blocks — 16 tokens is the typical choice — and keep a per-sequence block table mapping logical token positions to arbitrary physical blocks. A sequence now grows one small block at a time, on demand, and physical blocks need not be adjacent. Waste drops to at most one partially-filled block per sequence, which is under 4% instead of over 60%. Same hardware, 2-4× the throughput, purely from changing how memory is addressed.

The second, less-advertised win is copy-free sharing. Because a block table is just pointers, two sequences that share a prefix — the same 2,000-token system prompt, or the parallel branches of a beam search or an n-best sample — can point at the same physical blocks with a reference count, instead of each holding its own copy. That is the mechanism the prefix caching in the next lesson is built on; it is not a separate feature so much as a consequence of paging.

What to say when asked "so why not just allocate less?" Because the server cannot know in advance how many tokens a request will generate. That is the irreducible fact — output length is not knowable at admission — and every non-paged design has to resolve it by either over-reserving (wasting memory) or under-reserving (and evicting or failing mid-generation). Paging dissolves the dilemma rather than picking a side of it.

Admission control: what happens when the pool runs dry

A KV pool that fills up does not politely slow down; it forces a decision mid-generation, and you must have chosen which one in advance. The three real options, each with a failure mode worth naming:

Queue at the door. Refuse to admit a new request until enough blocks are free. Latency for the queued request goes up, but nothing already running is harmed. This is the right default and the one to state first.

Preempt and recompute. Evict a running sequence's blocks entirely and re-run its prefill from scratch when it is rescheduled. Cheap on memory, but you pay for that prompt's prefill twice — fine for short prompts, terrible for a 30K-token one.

Preempt and swap. Copy the evicted sequence's KV blocks to host RAM and copy them back later. Avoids recomputation, pays PCIe transfer time instead, and can be the worse deal than recompute for short prompts. Knowing that recompute-vs-swap is a prompt-length-dependent choice, not a global setting, is the senior detail here — and check what your engine version actually supports: vLLM's V1 engine dropped swap-based preemption and only recomputes.

Set max_model_len to what you actually serve, not to what the model supports. Advertising the model's full 128K context when 99% of your traffic is under 4K means either sizing the pool for a context nobody uses or having occasional 128K requests blow up capacity planning for everyone. Cap the served context at the real p99 of your traffic plus headroom, and reject above it with a clear error rather than degrading everyone silently.

Watch the one metric that predicts this failure before it happens: KV cache utilization percentage. GPU utilization will read near 100% and tell you nothing; KV utilization climbing past ~90% is the actual leading indicator that preemption and queueing are about to start. Alert on it, and make it the input to the autoscaling decision in the fleet lesson rather than CPU or GPU percent.

How to say it in the interview

"Weights are the easy part — they are a fixed, known number I can compute before booting anything. The number that actually decides my concurrency is KV cache, because it grows with every token of every active request, and it is the thing that runs out first. So let me size weights, then compute KV bytes per token, then divide the leftover VRAM by that to get my real concurrency ceiling."

Checklist

• Computed weight memory from parameters × bytes-per-param before anything else
• Derived KV bytes per token as 2 × layers × kv_heads × head_dim × dtype, with real numbers
• Named grouped-query attention as the reason kv_heads ≠ attention heads
• Stated concurrency as a function of context length, not as requests-per-GPU
• Named KV-cache quantization as the direct 2× concurrency lever, and weight quantization as the smaller one
• Explained fragmentation as the reason for PagedAttention, with a waste percentage
• Chose a policy for a full KV pool — queue, recompute, or swap — and named its cost
• Capped max_model_len at real p99 traffic rather than the model's advertised maximum
• Monitored KV cache utilization as the leading indicator, not GPU utilization

Related lessons

• Quantization Methods Compared: INT8, GPTQ, AWQ, FP8, FP4 and KV-Cache Quantization
• FlashAttention and Why Attention Is a Memory-Traffic Problem
• Continuous Batching and the Scheduler
• TTFT, TPOT, and the Latency Budget
• Mixture-of-Experts and Long-Context Serving
• Autoscaling a GPU Fleet Without Burning the Budget