FP16 vs Q8 vs Q6 vs Q4: which quantization should you use?
On this page
- The formats at a glance
- What that means in real VRAM
- Which one — by what you're doing
- Bonus: lower quant is also faster
You've picked a model. Now which build do you download? This is the practical decision — and for most people the answer is one line, with a few exceptions worth knowing.
Every quantization is a point on one line: more bits = more quality and more VRAM; fewer bits = smaller and faster. New here? Start with what quantization is. Otherwise, here's how the common formats actually compare.
1The formats at a glance
| Format | Bits / weight | Quality | Use when |
|---|---|---|---|
| FP16 | 16.0 | Reference | You have VRAM to burn or need exact reproducibility |
| Q8_0 | 8.5 | Near-lossless | You want maximum quality that still halves the size |
| Q6_K | 6.56 | Excellent | Quality-first with real VRAM savings |
| Q5_K_M | 5.69 | Very good | A safe margin above the sweet spot |
| Q4_K_M | 4.85 | The sweet spot | Best model that fits — the default for most people |
| Q4_0 | 4.5 | Good | Older/simpler; a touch more loss than Q4_K_M |
2What that means in real VRAM
Same model, six builds. Here's Qwen2.5 14B (≈14.8B params), weights only — overhead and KV cache come on top (how that's calculated):
| Format | Weights | Comfortable card |
|---|---|---|
| FP16 | ≈ 27.6 GiB | A100 / dual 24 GB |
| Q8_0 | ≈ 14.6 GiB | RTX 4090 24 GB |
| Q6_K | ≈ 11.3 GiB | RTX 4070 Ti SUPER 16 GB |
| Q5_K_M | ≈ 9.8 GiB | 16 GB card |
| Q4_K_M | ≈ 8.4 GiB | 12 GB card (RTX 3060/4070) |
| Q4_0 | ≈ 7.8 GiB | 12 GB card |
Cutting from FP16 to Q4_K_M drops the same model from ~27.6 GiB to ~8.4 GiB — a 14B model that needed a 24 GB card now fits on a 12 GB one.
3Which one — by what you're doing
Want the best model that fits → Q4_K_M
This is the right default. The VRAM you save lets you step up to a bigger, smarter model, and that almost always wins: a 14B at Q4_K_M beats a 7B at Q8 for the same memory. Optimize for model size first, precision second — down to Q4.
Want maximum quality and have the headroom → Q8_0
Effectively indistinguishable from FP16 at half the size. Reach for it when you're precision-sensitive (structured output, code that must compile, evals) and the model already fits comfortably.
Somewhere in between → Q5_K_M or Q6_K
A sensible margin when Q4 makes you nervous but Q8 won't fit. Little to fault here.
Tight on VRAM → stay at Q4_K_M, shrink the model
Don't drop below Q4 to squeeze a model in — the quality falls off a cliff there. Pick a smaller model at Q4_K_M instead. Use the model finder to see what fits your card.
4Bonus: lower quant is also faster
Decoding is memory-bandwidth bound — fewer bytes per weight means fewer bytes to read per token, so lower quants generate faster on the same card. The decode-speed ceiling is literally bandwidth ÷ weight-bytes, so Q4 has a higher ceiling than Q8 on identical hardware.
Related: what VRAM is.
5Common questions
Should I use FP16 or Q4 for local inference?
Q4_K_M for almost all local single-user inference: it cuts weights by roughly 3.3× against FP16 for a small quality cost. FP16 is worth it when you have memory to spare and need maximum fidelity, or when a runtime requires it.
How much VRAM does quantization save?
Roughly in proportion to bits per weight. Moving from FP16 (16 bits) to Q4_K_M (about 4.85) cuts the weight term by about 3.3× — usually the single largest lever available.
Is a lower quantization also faster?
Yes, for a memory-bound workload. Decode speed is bounded by memory bandwidth divided by the model's size in memory, so a smaller model is read faster per token.