What is quantization? (and why it lets you run big models on small GPUs)
On this page
- The one-sentence version
- How much you actually save
- Why "Q4" isn't exactly 4 bits
- Does quality actually suffer?
- The rule that matters most
Quantization is the single trick that turns "you need a data-center" into "this runs on your gaming card." Here's what it actually does to a model — and what it costs you.
Open-weight models ship in FP16 — every weight stored as a 16-bit number. That's accurate, but heavy: a 70-billion-parameter model in FP16 is roughly 140 GB of weights. No consumer GPU comes close. Quantization is how you shrink that without retraining anything.
1The one-sentence version
Quantization stores each weight using fewer bits — say 4 instead of 16 — trading a little numerical precision for a large drop in memory. The model keeps the same parameters, the same architecture, the same context window. It just represents each number more coarsely.
2How much you actually save
Weight size scales almost linearly with bits-per-weight, so the math is simple: 4-bit is roughly a quarter the size of 16-bit. If you want the exact numbers for any model, that's precisely what the VRAM calculator computes — and the methodology shows the formula:
Halve the bits-per-weight, halve the weight memory. That's the whole lever.
3Why "Q4" isn't exactly 4 bits
Modern quantization (the GGUF K-quants used by llama.cpp, Ollama and LM Studio) isn't uniform. It stores the parts of the model that matter most at higher precision and the rest at lower precision. That's why the formats have effective bits-per-weight that aren't round numbers — Q4_K_M is about 4.85 bits, not 4.0. The extra fraction is spent protecting quality where it counts.
4Does quality actually suffer?
Less than most people fear. For everyday chat and coding:
- Q8 is effectively indistinguishable from FP16.
- Q6 / Q5 — imperceptible to a human in normal use.
- Q4_K_M — the popular sweet spot; a tiny, rarely-noticed loss.
- Below Q4 — this is where degradation becomes real: repetition, weaker reasoning, more mistakes.
The full side-by-side is in FP16 vs Q8 vs Q6 vs Q4.
5The rule that matters most
A 14B model at Q4_K_M will typically out-think a 7B model at Q8, for the same VRAM. So when you're memory-limited, drop precision before you drop model size — down to about Q4. Below that, shrink the model instead.
6What quantization does not do
- It doesn't reduce the parameter count — a quantized 8B is still an 8B.
- It doesn't shrink the KV cache (the context memory). That's a separate cost that grows with how long a conversation gets — see the methodology.
- It doesn't change the model's training or knowledge — only how the weights are stored.
See also: why training needs more.
7Common questions
What is quantization in an LLM?
Storing the model's weights at lower numeric precision than the FP16 they ship in — for example about 4.85 bits per weight for Q4_K_M instead of 16. VRAM required falls roughly in proportion to the bits used.
Does quantization make a model worse?
Below about 4 bits, measurably. Between Q4_K_M and FP16 the difference is small for most tasks, which is why Q4_K_M is the common default. The practical rule is that a larger model at Q4 usually beats a smaller model at FP16 on the same card.
Why is Q4 not exactly 4 bits per weight?
K-quants are mixtures — different tensors in the same file are stored at different precisions. The effective size lands near 4.85–4.89 bits per weight depending on architecture, so sizing a 4-bit build at 4 bits underestimates it by roughly 20%.