Every practical guide to self-hosting an LLM — including ours — reaches for the same rule of thumb: run it at 4-bit, budget roughly 0.6 GB per billion parameters, get on with your life. It’s a good rule. It’s also a compression of an enormous amount of engineering, and it quietly hides a trade you are making on your users’ behalf.
A 70B model at 4-bit fits in 48 GB instead of 140 GB. You did not get that for free. Something was given up. This post is about what, exactly, and how to find out whether it matters for your workload.
What a quantized weight actually is
A model in FP16 stores each weight as a 16-bit floating-point number. Quantizing to 4 bits means each weight becomes one of sixteen possible values. That’s the whole idea, and stated that baldly it sounds like it shouldn’t work at all.
It works because you don’t quantize the entire tensor against a single scale. You break the weights into small blocks — commonly 32, 64 or 128 values — and each block gets its own scaling factor mapping its 16 integer levels onto the range that block actually occupies. A block whose weights all sit between −0.02 and 0.03 gets a fine-grained scale; a block with a wider spread gets a coarser one. Locality is what rescues the scheme: within a small block, weights tend to be similar in magnitude.
This also explains why 4-bit isn’t really 4 bits on disk. You store the quantized weights plus a scale per block, often plus a zero-point. At a block size of 64 with an FP16 scale, that’s an extra 16 bits per 64 weights — about 0.25 bits per weight of overhead, before any of the other bookkeeping. Hence “0.6 GB per billion parameters” rather than the 0.5 GB that a naive 4-bits-per-weight calculation gives you. The gap between those two numbers is the metadata that makes quantization work at all.
Smaller blocks mean better accuracy and more overhead. That single tension shows up again in every format decision below.
The outlier problem is the whole problem
Here is the fact that every quantization method is a different answer to.
Weight and activation distributions in transformers are not well-behaved. A small fraction of channels — often on the order of 1% — carry values with magnitudes far larger than the rest. And those channels matter disproportionately: they tend to be the ones carrying the most load in the layer.
Quantize naively and you have a bad choice. Stretch the scale to cover the outliers and every ordinary weight in the block collapses into two or three of your sixteen levels, destroying the precision where most of the information lives. Clip the outliers to keep resolution for the bulk, and you damage exactly the channels that mattered most.
Every method below is a different way of refusing that choice.
How the main methods differ
GPTQ works layer by layer, using second-order information — the Hessian of the layer’s reconstruction error — to decide how to round each weight. When it quantizes one weight, it adjusts the remaining un-quantized weights in that layer to compensate for the error just introduced. It’s a greedy error-minimisation procedure, and it was the method that first made 4-bit LLMs genuinely usable.
AWQ (Activation-aware Weight Quantization) starts from a different observation: not all weights are equally important, and importance is visible in the activations, not the weights. It identifies roughly the top 1% of salient weight channels using activation statistics, scales those channels up before quantizing, and then applies uniform low-bit quantization to everything. Scaling up the salient channels before the fact means the quantization error lands proportionally less on the channels that matter. On current model families — Llama 3 and later, Qwen 2 and later — AWQ typically beats GPTQ by roughly 0.5 to 1.0 percent perplexity at the same bit-width.
GGUF k-quants are the llama.cpp family, and they take a mixed approach: different tensors in the model get different bit allocations, with attention and embedding layers often kept at higher precision than feed-forward blocks. Names like Q4_K_M encode this — 4-bit, k-quant, medium. Built for CPU and Metal inference, which is why it’s the format you meet first on a laptop. (The local stack, explained.)
bitsandbytes NF4 quantizes on the fly at load time using a normal-float data type designed for the roughly Gaussian distribution of neural network weights. It’s the format underneath QLoRA, and its appeal is convenience — no separate quantization step — rather than peak quality.
| Method | Approach | Best for |
|---|---|---|
| GPTQ | Hessian-based error compensation, layer by layer | Established GPU serving, wide support |
| AWQ | Protects ~1% salient channels found via activations | Current best quality-per-bit on GPU |
| GGUF k-quants | Mixed bit allocation across tensor types | CPU and Apple Silicon inference |
| NF4 (bitsandbytes) | On-the-fly, normal-float data type | Fine-tuning workflows, QLoRA |
Post-training versus quantization-aware
The methods above are all post-training quantization (PTQ): take a finished model, run a few hundred calibration samples through it, produce a quantized version in minutes to hours. PTQ works well at 4 bits, can be pushed to 3, and degrades substantially below that.
Quantization-aware training (QAT) is the other family. Instead of compressing a finished model, you train with simulated quantization noise in the loop, so the model learns weights that survive being rounded. The results are markedly better at aggressive bit-widths — but QAT requires fine-tuning on billions of tokens, which puts it out of reach for most teams building on someone else’s base model.
This is precisely why Apple can ship a ~3B on-device model at 2-bit while your 2-bit PTQ experiment produces gibberish. They trained for it. (More on what that buys them.) One promising thread: research on QAT with synthetic data generated by the full-precision model itself, via knowledge distillation, removes the need for a labelled corpus — which lowers the barrier without removing the compute cost.
The practical rule: at 4 bits, PTQ is fine. Below 3 bits, you need QAT, which means you need to be the one training the model.
FP4 in hardware: NVFP4 and MXFP4
Until recently 4-bit was a software convention — you packed integers and unpacked them in the kernel. Current-generation datacentre GPUs implement 4-bit floating point natively, and two formats matter.
NVFP4 uses 16-value blocks with an FP8-E4M3 scaling factor per block, plus a second-level FP32 scale per tensor. MXFP4, from the open microscaling family, uses 32-value blocks with an E8M0 scale — a pure exponent, no mantissa bits.
That difference is the whole story. E4M3 has mantissa bits, so its scale can land closer to the block’s actual range; E8M0 can only represent powers of two, so it rounds the scale itself to the nearest power of two and wastes part of the range. Combined with the smaller 16-value block, NVFP4 generally comes out ahead at equivalent calibration quality.
MXFP4 isn’t beaten, though. Applying a block-wise Hadamard rotation before quantizing spreads the outliers across all channels in the block — turning a few extreme values into many moderate ones, which is exactly the distribution uniform quantization handles well. Work along these lines (MR-GPTQ, ICLR 2026) closes most of the gap.
One caution that survives all of this: quantizing weights to 4 bits is well understood, but quantizing activations to 4 bits as well — W4A4 — still shows sharp accuracy drops. When a vendor advertises “4-bit,” check which side they mean.
What actually degrades
This is the section to read twice, because the standard benchmarks will actively mislead you here.
Quantized models typically hold MMLU within 1–3 points of full precision. Teams see that, conclude quantization is nearly free, and ship. Then specific things break in production, and the aggregate benchmark never showed it. What degrades, in rough order of how sharply:
- Long context. Sequences beyond about 4k tokens are markedly more sensitive than short ones. Worse, KV-cache quantization hurts long context more than weight quantization at the same bit-width — so if you’re compressing the KV cache to fit more concurrent users, that’s where your quality is going.
- Multilingual, especially non-Latin scripts. Quantization degrades multilingual capability disproportionately, and the effect is worst for low-resource languages and non-Latin scripts. The cross-lingual routing inside the model appears to be genuinely fragile under precision loss. If you are running Hindi, Tamil, Bengali or code-mixed speech — as any Indian voice deployment is — this is not a footnote, it’s the main risk.
- Multi-step reasoning. Chained arithmetic and multi-hop logic degrade measurably; specialised math workloads have shown degradation past 5% at common 4-bit k-quant settings, while general benchmarks moved barely at all.
- In-context learning. The ability to pick up a pattern from examples in the prompt weakens as bit-width drops — which matters most for exactly the few-shot prompting that small deployments lean on.
The pattern: quantization preserves what the model knows far better than what the model does with what’s in front of it. Recall survives; the delicate machinery degrades.
Calibration data is a decision, not a default
PTQ needs a calibration set — a few hundred samples run through the model to measure activation ranges. Almost every default pipeline uses a generic English web corpus.
If you then deploy in Hindi, or on legal contracts, or on medical transcripts, you have calibrated the quantization to a distribution your traffic doesn’t match. The scales are tuned for activations your model won’t see. Research on calibrating with language-diverse data shows meaningful recovery on multilingual performance from nothing more than changing what you calibrate on.
This is the cheapest quality win in the entire stack — a few hundred samples of your own domain, in your own languages — and it is very widely skipped.
A decision procedure
- Establish an FP16 baseline on your own eval set. Not MMLU. The tasks you actually run. Without this number, every subsequent comparison is guesswork — and if you don’t have that eval set yet, building it is the prerequisite for this whole exercise.
- Start at 4-bit AWQ for GPU serving, or
Q4_K_Mfor CPU and Apple Silicon. This is the default that’s right most of the time. - Measure the specific things that break. Long-context retrieval, your non-English languages, any multi-step reasoning. Aggregate scores hide precisely these.
- Keep the KV cache at higher precision than the weights if long context matters. It’s the more sensitive of the two.
- Calibrate on your domain and your languages. Cheap, and it addresses the failure mode most likely to hit you.
- If quality doesn’t hold, step up before you step sideways. 6-bit or 8-bit is a smaller change than swapping model families. On phone-class hardware, 6-bit costs roughly 20–27% throughput and about 40% more memory than 4-bit — a real price, but a knowable one.
- Don’t go below 3-bit with PTQ. That’s QAT territory, and QAT means training.
The honest summary
Four-bit quantization is one of the best trades available in machine learning: roughly 92–95% of full-precision quality for a quarter of the memory, which is what makes self-hosted inference practical at all. Nothing here argues against using it.
What’s worth arguing against is treating it as free. The compression is not uniform across capabilities — it takes more from long context, from languages that were underrepresented to begin with, and from multi-step reasoning than it takes from factual recall. If your product happens to live in one of those areas, the default that works fine for everyone else will quietly underperform for you, and the benchmark you were watching will not tell you.
Measure the thing you actually sell. Everything else is a proxy.
Sizing an on-prem deployment and want the precision decision made on evidence rather than convention? That’s the kind of problem we like.