Track C · Document 09 · Shrinking the footprint
What the bits in a float actually buy, why the nominal bit width is never the real one, and which of five methods to reach for on which hardware.
Quantisation is usually explained without explaining what a float is, which turns the whole topic into memorisation. Two minutes here makes the rest obvious.
A floating-point number spends its bits on two things: range, how big or small the value can be, and precision, how finely it can be distinguished from its neighbours. Every format is a different split.
The two things bits buy are range and precision, and every format here is a different split between them. Quantisation is the same question asked once more: having given up precision, how do you spend what is left?
fp16 has five exponent bits, which caps the largest representable value at about 65,500. During training, activations and gradients routinely exceed that, so fp16 needs loss scaling and careful handling. bf16 spends the same sixteen bits differently: eight exponent bits, the same as fp32, so the same enormous range — and only seven mantissa bits, so much less precision.
That turns out to be the right trade, because neural networks tolerate imprecision far better than they tolerate overflow. Which is why almost every published checkpoint ships in bf16, and why bf16 is the baseline every quantisation result is measured against.
Two ways of writing down a measurement on a form with sixteen boxes. One scheme uses most of the boxes for decimal places — brilliant for anything between 0.001 and 100, and unable to write down a million at all. The other reserves boxes for an exponent — it can write anything from a trillionth to a trillion, but only to three significant figures. For measuring the same kind of thing over and over, the first is better. For measuring things whose magnitude you cannot predict, the second is the only one that works. Training is the second case.
Each weight is normally stored in sixteen bits. Quantisation stores it in eight, or four, or fewer. You obviously lose precision. The whole craft is in losing as little useful information as possible — and the mechanism that makes it work at all is the block scale.
Every block gets its own scale, which is why quantisation does not destroy the model. And every scale is stored, which is why the nominal bit width is never the real one.
The nominal bit width is never the real cost. A 4-bit format stores a scale — usually 16-bit — and often a zero-point, per block of 32 to 128 weights. And mixed-precision recipes keep sensitive tensors at higher precision.
So “13B at 4-bit is 6.5 GB” is wrong. AWQ or GPTQ at group size 128 is about 4.16 effective bits; GGUF Q4_K_M is about 4.9, which makes a 13B model roughly 7.9 GB rather than 6.5. That 20% gap is exactly the size of error that turns “it fits” into an out-of-memory failure on the first request.
Change the card. On a 24 GB A10 an 8B in bf16 leaves 1.8 GiB of cache — one user — and at 4-bit it leaves 14.4 GiB, which is fourteen. That is not a speed decision; it is the difference between a deployment and a demo.
Round-to-nearest has three weaknesses. Every named method is an attempt to fix one of them, and being able to say which is a far better answer than listing acronyms.
Four tabs, one question each. Round-to-nearest is the floor; GPTQ attacks compounding error; AWQ attacks importance and activation scale; k-quants attack it by spending bits unevenly across the model. FP8 sidesteps the whole discussion by having hardware support.
| Method | Calibration? | The idea, in one line | Runs on | Typical use |
|---|---|---|---|---|
| Round-to-nearest | No | Per-block scale from the largest magnitude, round everything to the nearest level | Anything | The baseline everywhere |
| GPTQ | Yes | Quantise weight by weight, adjusting the not-yet-quantised weights to compensate for each rounding error | GPU, hours | Older GPU servers |
| AWQ | Yes | Find the ~1% of channels carrying large activations and scale those weights up before rounding to protect them | GPU, faster than GPTQ | The pre-Hopper production default |
| k-quants (GGUF) | No | Mixed precision inside a layer — attention and output tensors keep more bits than the feed-forward | CPU, Apple silicon, GPU | Laptops and local use |
| FP8 | No | Cast to native 8-bit float; Hopper and later multiply it directly, with no unpacking | H100 and newer | The modern production answer |
Not all weights matter equally, and you find the ones that do by looking at the activation distribution rather than at the weights themselves. Protecting roughly one per cent of channels recovers most of the quantisation error, and it avoids the heavier second-order machinery GPTQ uses — which is why it became the common enterprise fallback on pre-Hopper cards.
This is misunderstood constantly, and getting the direction right is a genuine signal.
The reason 4-bit weights cannot be multiplied directly is simply that the tensor cores have no 4-bit multiply path. FP8 is where the silicon caught up, and fp4 on Blackwell-class hardware is the same story one step further.
| Weights | Activations | |
|---|---|---|
| Where from | Learned during training | Calculated from your input |
| Change? | Never | Every single request |
| Shared? | Yes — one copy serves everyone | No — yours alone |
| Quantised when? | Offline, once, producing a new file | At run time if at all, and it is much harder |
| Analogy | The recipe | The ingredients |
The KV cache is activations that you chose to keep. That is why quantising it is a runtime flag rather than an offline job, and why its error behaves differently: weight quantisation error is fixed and calibrated against once, but cache quantisation error accumulates over a generation, because a key quantised at token 10 is still being read at token 10,000.
Keeping those three straight — weights, activations, cache — makes half the confusing questions in this area evaporate. Document 10 covers the cache side.
Two ecosystems exist side by side and they do not mix: llama.cpp with GGUF (C++, self-contained, CPU-friendly) and the Python/CUDA world (vLLM, SGLang, AWQ, GPTQ, fp8). Choosing a format before choosing a server is the most common way to waste a week.
The honest answer to “does quantisation make the model dumber” is: a little, it depends on the method and the bit width, and you measure it on your own workload rather than trusting a general claim. The published picture is that fp8 is nearly indistinguishable, good 4-bit methods lose a small but measurable amount, and very low bit widths degrade noticeably — but those are statements about aggregate benchmarks, not about your task.
What to actually do: take a fixed set of a few hundred real requests, run them against the bf16 model and the quantised one, and compare on whatever metric your product cares about — exact-match on extraction, a judge score on chat, pass rate on code. Then decide whether the memory is worth it. That is an afternoon, and it converts an argument into a number.
It degrades unevenly. Aggregate scores can hold up while one capability falls off a cliff — frequently long-context retrieval, multilingual output, or exact formatting. Test the slices you care about separately, not just the mean.
Below about 4 effective bits, ask a different question. At that point the honest comparison is not “4-bit versus 3-bit of this model” but “3-bit of this model versus a smaller model at higher precision”. Very often the smaller model at 8-bit wins on both quality and speed, and nobody checked because the question was framed as a quantisation decision.
ArchitectWhat does quantisation actually do, and what does it cost?
It stores each weight in fewer bits — eight or four instead of sixteen — and the mechanism that makes it work is the block scale. You take a block of 32 to 128 weights that sit together, find their actual range, and fit the available levels across just that range. Without per-block scales you would be spreading sixteen levels across the whole dynamic range of the model and everything would round to the same value.
The cost is in two places. Quality, which is small for fp8, measurable for good 4-bit methods, and noticeable below that — and which you have to measure on your own workload rather than trust a general claim. And a bookkeeping cost that people forget: the scale is stored too. A 16-bit scale per 128 weights adds 0.16 bits to every weight, so “4-bit” is really 4.16, and a mixed recipe like Q4_K_M is 4.9. That is why a 13B Q4_K_M file is 7.9 GB and not the 6.5 GB the nominal arithmetic suggests.
ArchitectGPTQ or AWQ?
They attack different weaknesses of naive rounding. GPTQ attacks compounding error: it quantises weight by weight and nudges the not-yet-quantised weights to compensate for each rounding error it introduces — like shifting people in a group photo when one has to move. AWQ attacks importance: it watches the activations, finds the roughly one per cent of channels carrying unusually large signals, and scales those weights up before rounding so they land where the resolution is useful, scaling the inputs down to keep the maths equivalent.
In practice I would reach for AWQ on pre-Hopper hardware: it is simpler, faster to produce, more robust to the choice of calibration set, and slightly better in quality. GPTQ can overfit to its sample texts and uses heavier second-order machinery for the same job.
But the first question I would ask is what hardware we are on, because on H100 or later the answer is neither — it is fp8, which needs no calibration data at all, takes minutes, is near-lossless, and is multiplied natively by the silicon so it cuts operations as well as bytes.
ArchitectDoes a 4-bit model run four times faster?
No, and the reason is worth spelling out with the three-way test. Quantisation cuts bytes stored and bytes moved. It does not cut operations performed — except on hardware with native support for the format, which is fp8 on Hopper and fp4 on Blackwell.
Since decode is memory-bandwidth-bound, cutting bytes moved does speed it up, close to proportionally, at small batch. But two things blunt it. First, unpacking 4-bit weights to 16-bit for the multiply costs real work, so int4 kernels rarely hit the theoretical speedup. Second, and more importantly at scale, once the batch is large the KV cache is most of what you are reading — at 8k context and batch 64 the cache is 81% of the bytes — so shrinking the weights barely touches the total.
The honest framing: quantise the weights for capacity first, because freeing 7.5 GiB on an 8B directly buys concurrency. Treat the speed as a welcome secondary effect.
Eng managerCan we halve our GPU bill by quantising?
Possibly, and I would want to answer it with a measurement rather than a prediction, because the size of the win depends on which constraint we are actually under.
If we are memory-constrained — cache utilisation pinned, requests queueing for blocks — then yes, this is direct. Going from bf16 to fp8 on an 8B frees 7.5 GiB, which is 14% more KV budget and therefore 14% more users per card. Going to int4 frees 10.5 GiB. On a small card the effect is dramatic: an 8B in bf16 on a 24 GB A10 leaves room for one user; at int4 it leaves room for fourteen.
If we are compute-constrained or utilisation-constrained, much less. And I would put the alternative on the table at the same time, because it is often bigger: a half-idle expensive GPU costs the same as a busy one, so consolidating traffic onto fewer fuller machines frequently beats any quantisation gain. I would bring both numbers.
The cost side is a quality evaluation on real traffic and a rollback plan. fp8 is a low-risk change on Hopper; int4 is a real quality decision and I would want the evaluation before committing, not after.
ArchitectWhy can 4-bit weights not just be multiplied directly?
Because the tensor cores have no 4-bit multiply path. The weights stay 4-bit in HBM for the whole session — they are never expanded in memory, that would defeat the point — but when a slice is needed for a multiply it is unpacked to 16-bit inside on-chip memory, used, and discarded.
The direction matters and people get it backwards: it is the weights that get unpacked, not the activations. Activations were 16-bit all along.
fp8 is the exception on Hopper and later, where the silicon multiplies it natively with no unpacking step — which is exactly why fp8 is both near-lossless and genuinely fast, and why it became the enterprise default on new hardware. fp4 on Blackwell-class parts is the same story one step further.
ArchitectIs quantising the cache the same as quantising the weights?
Two separate things with different risk profiles. Quantising weights is an offline job producing a new file; quantising the KV cache is a runtime setting on the server. They stack — you can do both.
The risk differs because of where the error lives. Weight quantisation error is fixed: it is baked in once and you can calibrate against it. Cache quantisation error accumulates over a generation, because a key quantised at token 10 is still being read at token 10,000, and every subsequent token attends to it.
Practically that means 8-bit cache is generally safe and below that you should test on your own workload — and specifically test on long outputs, because a 200-token evaluation will not surface a problem that appears at 2,000.
Eng managerSomeone wants to run a 70B at 3-bit instead of an 8B at bf16. How do you evaluate that?
By insisting the comparison is run rather than argued, because both sides have a plausible story and the answer is empirical.
The memory arithmetic first: a 70B at roughly 3.9 effective bits is about 32 GiB of weights, which fits on one H100 with about 36 GiB left for cache — but the 70B costs 2.5 GiB per user at 8k against the 8B’s 1.0, so we would get roughly 14 concurrent users against 53. And single-stream decode would be about two and a half times slower, because the weight read is twice the size.
So the trade is quality against roughly 4× the concurrency and 2.5× the streaming speed. That is a big enough gap that I would want it settled by a head-to-head evaluation on a few hundred real requests, scored on whatever the product actually cares about — and I would want the per-slice breakdown, because heavy quantisation degrades unevenly and can hold up on aggregate while falling off a cliff on one capability. If the 70B at 3-bit does not clearly win on quality, the 8B wins on everything else.
Why is BF16 the format most weights ship in?
Same sixteen bits as fp16, split differently: eight exponent bits instead of five, so it has fp32’s dynamic range with less precision. Networks tolerate imprecision far better than overflow, so that trade is much more forgiving during training — and published checkpoints reflect what they were trained in.
Does quantisation make the model dumber?
A little, and how much depends on the method and the bit width. fp8 is nearly indistinguishable; good 4-bit methods lose a small measurable amount; very low widths degrade noticeably. The honest answer in an interview is that it is a trade you measure on your own workload rather than a general claim you quote.
Can I quantise a model myself?
Yes. Data-free methods like GGUF conversion or an fp8 cast run on an ordinary machine in minutes to hours. Calibration methods like GPTQ and AWQ need a GPU and can take many hours on a large model, because they have to run the model forward on sample text repeatedly.
Is quantisation the same as pruning or distillation?
No. Quantisation keeps every weight but stores each with fewer bits. Pruning removes weights entirely. Distillation trains a smaller, differently-shaped model to imitate a bigger one. Quantisation is the only one of the three that leaves the architecture untouched, which is why it is the only one that is an afternoon rather than a project.
What does “group size” control?
How many weights share one scale factor. Smaller groups track the local range more tightly, so quality is better, and cost proportionally more overhead — a group of 32 has four times the scale cost of a group of 128. 128 is the usual default and a reasonable place to leave it.
Why does loading a model take so long?
Most of it is reading tens of gigabytes off disk; disk is usually the bottleneck, not the GPU. Which is a real argument for quantisation that nobody makes: a 4-bit model is a quarter of the bytes to read, so cold start is roughly four times faster — and cold start is what makes autoscaling an LLM hard. That is document 15.
Can I mix formats — fp8 weights and an fp8 cache?
Yes, and you should think of them as independent levers that multiply. fp8 weights free memory that becomes KV budget; an fp8 cache halves what each user needs from that budget. Both are supported flags on modern servers. Just evaluate them separately, because their error modes differ.
What is SmoothQuant, and where does it fit?
It addresses the same activation-outlier problem AWQ does, but for activation quantisation rather than weight-only: it migrates the difficulty from activations into weights by rescaling both, so both can be quantised to 8 bits. Weight-only quantisation is far more common in serving because it is easier and captures most of the memory benefit; activation quantisation matters when you want the compute speedup too.
Why is my quantised model slower than expected?
Most likely the kernel. A quantised format only gets its speedup if the server has a fused kernel that reads the packed weights directly; a generic path that dequantises into a full-size buffer first gives you the memory saving and none of the speed. Check that the engine reports using the quantised kernel rather than a fallback — and be suspicious if throughput barely changed while memory did.
One sentence on when to quantise?
When memory is the binding constraint. If cache utilisation is pinned and requests are queueing for blocks, quantising the weights directly buys concurrency. If the card is half idle, quantisation is solving a problem you do not have and the win is in utilisation instead.
“Quantisation stores each weight in fewer bits, and the thing that makes it work is the per-block scale: you fit the available levels across the actual range of 32 to 128 neighbouring weights rather than across the whole model. The scale is stored too, so the nominal width is never the real one — 4-bit is really 4.16 with AWQ at group size 128, or 4.9 for a mixed GGUF recipe. The methods each attack a different weakness of naive rounding: GPTQ compensates for compounding error, AWQ protects the one per cent of channels carrying large activations, k-quants spend bits unevenly across the model. On Hopper and later the answer is usually fp8, because the silicon multiplies it natively so there is no unpacking step, it needs no calibration data, and it is near-lossless. At run time the weights stay packed in memory and are unpacked to 16-bit inside the chip for each multiply — it is the weights that unpack, not the activations. And it cuts bytes stored and bytes moved, not operations, so quantise for capacity first and treat the speed as a bonus.”
| Thread started here | Picked up in |
|---|---|
| The 70% of parameters that quantisation is mostly quantising | 02 · Inside the model |
| Why fp8 halves the bytes and doubles the peak, leaving the ridge batch unchanged | 04 · The GPU and the roofline |
| Why weight quantisation stops helping once the cache dominates the read | 05 · Prefill and decode |
| The memory freed, and what it becomes in the ladder | 06 · The KV cache |
| Quantising the cache, the runtime lever that stacks on top of this | 10 · Paging and prefix reuse |
| Which kernel actually gets used, and the flags that select it | 12 · Serving engines |
| Faster cold start as a consequence of a smaller file | 15 · Production |