Cruxly

Cruxly › AI engineering › Inference Serving

Quantization Methods Compared: INT8, GPTQ, AWQ, FP8, FP4 and KV-Cache Quantization

one outlier that turns a sub-1-percent error into 31 percent, a 4-bit hand example, and why INT4 gave no speedup at batch 128

Open the interactive lesson

Prompt: "We serve a 70B model on 8 GPUs and finance wants the bill halved. Explain how int8 quantisation works, compare AWQ and GPTQ, tell me which formats our hardware supports, say whether 4-bit will hurt quality, and explain why a team that moved to INT4 saw no speedup at batch size 128."

40 minutes first. Quantise four numbers by hand, show what one outlier does, and work out the time of one decode step at batch 128 with and without quantised weights. The trap is answering "4-bit is 4x faster and loses a little accuracy"; the real answer depends on which resource is the bottleneck at your batch size.

What quantisation does: scale, round, and the outlier that ruins a group

Quantisation maps a range of floating-point numbers onto a small set of integer codes, using one scale per group of numbers. For symmetric absmax int8, the scale is s = max(|w|) / 127, the code is q = round(w / s), and the dequantised value is q x s. GPU Memory Math: Weights, KV Cache, and What Actually Fits gave the memory table; this lesson is how the bits are chosen and how it fails.

Worked example. Weights [0.12, -0.50, 0.33, 0.90]:

Now add one outlier. Append 20.0 to the group. The scale becomes 20 / 127 = 0.1575, so the step between neighbouring codes is 0.1575:

One large value in a group sets the step size for everyone. That is the central fact behind every method in this lesson.

Granularity is the first defence. The scale can be shared across a whole tensor (cheap, fragile), one per output channel (a row of the weight matrix), or one per group of, say, 128 consecutive weights. Smaller groups confine an outlier's damage, at the cost of storing more scales. With a 16-bit scale per group of 128 weights the overhead is 16 / 128 = 0.125 bits per weight, so "4-bit" is really 4.125 bits: the 70B model is 70B x 4.125 / 8 = 36.1 GB, not 35 GB. With groups of 32 it is 4.5 bits, 39.4 GB.

Where the outliers live. In transformer language models, activations (not weights) develop a few large-magnitude feature dimensions. Dettmers et al., LLM.int8() (August 2022), describe systematic outlier features that strongly affect attention and predictions, and handle them by splitting the matrix multiplication: the outlier dimensions run in 16-bit while over 99.9 percent of values multiply in 8-bit. They report running models up to 175B parameters in int8 without loss of performance on their evaluations. The idea to retain: activations are harder to quantise than weights, and a few channels carry most of the trouble.

Three things a number can be quantised: weights, activations, and the KV cache. Weight-only quantisation shrinks the weights you store and read. Weight-and-activation (such as W8A8) lets the matrix multiply itself run in 8-bit on hardware that supports it, which also raises compute throughput. KV-cache quantisation stores keys and values in 8 bits, halving the cache that limits concurrency and that every decode step reads. Each relieves a different resource, which is the lens for the rest of the lesson.

The methods compared: LLM.int8, SmoothQuant, GPTQ, AWQ, NF4 and QAT

The methods differ in what they quantise, how they choose the rounding, and what they need to run.

MethodQuantisesHow it handles outliers or errorTypical use
LLM.int8()weights and activations (8-bit)outlier feature dimensions in 16-bit, rest in 8-bitmemory saving at 8-bit
SmoothQuantweights and activations (W8A8)migrates difficulty from activations to weights with an equivalent transformcompute-bound serving in 8-bit
GPTQweights only (3 or 4 bit)rounds columns one at a time and corrects the remaining weights using second-order information4-bit weight-only
AWQweights only (4 bit)scales up the salient channels, found from activations, before quantising4-bit weight-only
NF4 (QLoRA)weights, for traininga 4-bit data type fitted to normally distributed weights, plus double quantisationfine-tuning on one GPU

GPTQ (Frantar et al., October 2022) is post-training: it needs a small calibration set, no retraining, and reports quantising a 175B model in about four GPU hours with small accuracy loss at 3 or 4 bits, and inference speedups of about 3.25x on an A100 over fp16 in the authors' tests. AWQ (Lin et al., June 2023; MLSys 2024 best paper) rests on the observation that about 1 percent of weight channels matter disproportionately, and that these are found by looking at activation magnitudes, not weight magnitudes. Instead of keeping those channels in higher precision (awkward for hardware), it scales them up before quantisation. SmoothQuant (Xiao et al., November 2022) divides the large activation channels by a factor and multiplies the matching weight rows by the same factor, which leaves the layer's output unchanged but moves the quantisation difficulty into the weights; the authors report up to 1.56x speedup and 2x memory reduction with negligible accuracy loss.

An AWQ-style example by hand, in 4 bits. Use symmetric int4 with codes from -7 to 7, weights [0.20, -0.50, 0.33, 0.90], and an activation vector [10, 1, 1, 1]: the first channel is the salient one because its activation is ten times larger.

The rounding errors on other channels did not change; the gain came entirely from giving the salient channel finer effective resolution. The cost is that scaling up a channel can push its group's maximum higher, so real implementations search for the scale per channel. This is a toy, not a benchmark.

QAT versus PTQ. Post-training quantisation (GPTQ, AWQ, SmoothQuant) quantises a trained model using a calibration set; it is cheap and its failures are mild at 8 bits and noticeable below 4. Quantisation-aware training simulates the quantisation during training so the model adapts; it costs a training run and is what you reach for when PTQ quality is not acceptable. QLoRA (Dettmers et al., May 2023) is a different use: it keeps the base model frozen in 4-bit NF4 and trains small adapters, so a 65B model can be fine-tuned on a single 48 GB GPU, as LoRA and QLoRA: What Actually Changes During Fine-Tuning explains. That is a training-memory technique, and the adapters are trained in higher precision.

Calibration is a hidden dependency. GPTQ and AWQ need example inputs to estimate activation statistics or error. A calibration set of English web text will under-represent code, tool-call JSON or Hindi, so the quantised model can regress exactly there. Calibrate on text that looks like your traffic.

Hardware, formats and the roofline: why INT4 gave no speedup at batch 128

What a format buys depends on whether the hardware has native support for it and on whether the step is bound by memory or by compute.

:::key Verified October 2026 vLLM's quantisation documentation (retrieved October 2026) lists supported methods including AWQ, GPTQ, FP8 W8A8, INT8 W8A8, INT4 W4A16 through LLM Compressor, BitsAndBytes, quantised KV cache and others. Its hardware table marks FP8 W8A8 as supported on Ada, Hopper and AMD GPUs but not on Ampere; AWQ from Turing onward; GPTQ from Volta onward; INT8 W8A8 from Turing onward; and lists Marlin kernels (GPTQ, AWQ, FP8, FP4) for Turing and later NVIDIA generations, which are weight-only paths. NVIDIA's developer blog (24 June 2025) introduced NVFP4 with the Blackwell architecture: 4-bit values with one FP8 (E4M3) scale per 16 values plus a per-tensor FP32 scale, 4.5 bits per value, and approximately 3.5x smaller than FP16 and 1.8x smaller than FP8 (NVIDIA's figures). Tables like these change with each release; read the docs for your version. :::

Memory arithmetic first. For the 70B model: fp16 140 GB; 8-bit 70 GB; 4-bit with group scales about 36 GB; NVFP4 at 4.5 bits, 70B x 4.5 / 8 = 39.4 GB, a 3.56x reduction from fp16, matching the blog's roughly 3.5x. On Ampere (the A100 in GPU Memory Math) there are no FP8 tensor cores, so the 8-bit choice is INT8 (weight-only, or W8A8 via SmoothQuant-style methods) and the 4-bit choice is weight-only AWQ or GPTQ. On Hopper and later, FP8 is the natural 8-bit format; on Blackwell, NVFP4 or MXFP4 are the 4-bit formats.

Now the speed question, with the roofline. During decode, each step reads all the weights once, regardless of batch, and does 2 x parameters x batch FLOPs. With assumed round numbers (1,000 TFLOP/s of half-precision compute and 3.3 TB/s of HBM bandwidth per GPU, near a current data-centre GPU's datasheet; check yours), the ridge point is 1,000 / 3.3 = 303 FLOPs per byte. At fp16 each weight is 2 bytes and does 2 x batch FLOPs, so the intensity is batch FLOPs per byte and decode stays memory-bound until batch is about 300. At 4-bit weights (0.5 bytes) the intensity is 4 x batch, crossing the ridge near batch 76. Real kernels dequantise to fp16 inside the kernel, which costs extra instructions, so the practical crossover is lower.

Work one decode step for the 70B model at batch 128, 8 GPUs, 8,000-token contexts (aggregate bandwidth 8 x 3.3 = 26.4 TB/s, aggregate compute 8,000 TFLOP/s, all assumed):

The KV cache dwarfs the weights at this batch and context, which is the answer to the interview question "why no speedup from INT4 at batch 128?": weights were no longer the main bytes read (140 GB against 335.5 GB), and real 4-bit kernels also pay a dequantisation cost that eats part of the 1.28x, so a measured gain near 1.0 to 1.2x is unsurprising. Compute (2.2 ms) is still far below the memory time, so this step stays memory-bound. (The calculation is a lower bound on time; real systems add communication and scheduling.) Continuous Batching and the Scheduler is where batch size becomes a deliberate choice, and TTFT, TPOT, and the Latency Budget is where per-token time becomes a latency number.

Evaluating a quantised model properly, and the decision procedure

Perplexity says the model still predicts text; it does not say the product still works. Perplexity on a general corpus is cheap and catches gross damage, but quantisation errors concentrate where the model is already fragile: long chains of reasoning, code, strict JSON and tool calls, long contexts and lower-resource languages. Build the gate from the evaluation habits in Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks.

Expect the damage to grow as bits fall and as the model shrinks. 8-bit weights are usually near-lossless in published evaluations; 4-bit weight-only is typically a small measurable loss that is more visible on reasoning and code and on smaller models; below 4 bits needs QAT or special methods. That pattern is a rule of thumb from the papers and practice, not a guarantee, so the answer to "will 4-bit hurt quality?" is "measure it on our slices; here is the budget".

The decision procedure, in the order you would say it.

@@ 1 | Find the bottleneck Memory-bound or compute-bound at your real batch and context? Compute the step-time terms as above. If KV reads dominate, quantise the KV cache first; if weights dominate (small batch, short context), quantise weights.

@@ 2 | Pick the format the hardware accelerates FP8 on Ada, Hopper and later; NVFP4 on Blackwell; INT8 or 4-bit weight-only on Ampere. A format without native kernels may save memory and gain no speed.

@@ 3 | Calibrate on your traffic Use representative samples, including the slices that matter.

@@ 4 | Run the gate Task evals, slices, paired test, budget. Include the maximum context length you serve.

@@ 5 | Canary and keep the rollback Keep the previous precision artefact and the routing flag; roll back on the metric, not on the argument.

:::warn The most common production failure is a quantised checkpoint that passes the benchmark and regresses a rarely-sampled slice, such as long documents or code-mixed queries, found by customers. The gate must include the slices your traffic has, and the rollback must be one flag away. :::

How to say it in the interview

"Quantisation stores numbers in fewer bits using a scale and rounding: divide by the scale, round to an integer code, and multiply back. The scale is set by the largest value in the group, so one outlier wastes the resolution of everything else; that is why per-channel or per-group scales exist and why methods like SmoothQuant and AWQ move or protect the outlier channels. GPTQ and AWQ are weight-only: they shrink memory and the bytes read per decode step, which helps while decode is bandwidth-bound. At batch 128 with long contexts the KV cache reads dominate, so weight-only INT4 buys a modest gain and quantising the KV cache buys more. I would gate any change on task evaluations across slices, not perplexity, with a regression budget and a rollback path."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons