Learn · Concepts
Training vs inference memory (where the extra GPU memory goes)
In short
- Inference stores weights. Training stores weights, gradients and optimizer state — roughly 16 bytes per parameter for the standard mixed-precision AdamW recipe.
- For Llama 3.1 8B Instruct: about 5.5 GB to run at Q4_K_M, against roughly 120 GB to full fine-tune — about 22× more. Against FP16 inference (16.5 GB) the gap is nearer 7.3× — the multiple depends on what you compare against.
- 12 of 12 cards in our set can run it; 1 can full fine-tune it.
- LoRA and QLoRA collapse the gap by training a small adapter instead of every weight. That is why they exist.
Every VRAM figure elsewhere on this site is an inference number: what it costs to run a model that already exists. Training the same model costs several times more, for reasons that have nothing to do with the weights themselves. Here is where it goes.
1Where training memory goes
Inference needs the weights resident and little else. Training additionally needs a gradient for every trainable parameter and the optimizer’s running state. For the standard mixed-precision AdamW recipe that is sixteen bytes per parameter: an fp16 weight, an fp16 gradient, and three fp32 tensors: master copy, momentum and variance.
| Component | Per parameter | For Llama 3.1 8B Instruct | Why |
|---|---|---|---|
| Weights (fp16) | 2 bytes | 15.0 GB | The model itself, in mixed precision. |
| Gradients (fp16) | 2 bytes | 15.0 GB | One per trainable parameter, every step. |
| Optimizer — master weights (fp32) | 4 bytes | 29.9 GB | AdamW keeps a high-precision copy. |
| Optimizer — momentum (fp32) | 4 bytes | 29.9 GB | First moment estimate. |
| Optimizer — variance (fp32) | 4 bytes | 29.9 GB | Second moment estimate. |
| Total | 16 bytes | 120 GB | Before activations, which depend on batch size and sequence length. |
Activations are on top and scale with batch size and sequence length — which is why gradient checkpointing exists, trading recomputation for memory. The fine-tuning calculator models all of it.
2Why LoRA and QLoRA exist
The table above is the argument for adapters. If gradients and optimizer state are charged per trainable parameter, then freezing the base model and training a small low-rank adapter removes almost all of that cost — you pay full price for a few million parameters instead of 8.0 billion.
QLoRA goes further by holding the frozen base in 4-bit. The base is never updated, so quantization error never compounds through training, and memory is dominated by the compact base plus a small adapter. That is how fine-tuning a model this size becomes possible on a single consumer card.
3What it means for choosing a GPU
A card that comfortably runs a model may be nowhere near able to train it. On our set, 12 of 12 cards run Llama 3.1 8B Instruct at Q4_K_M, but only 1 can hold a full fine-tune of it. If training is the goal, size for the training figure or plan on adapters — and check what else is competing for that VRAM.
4Common questions
Why does training need so much more VRAM than inference?
Inference only needs the weights resident. Training also stores a gradient per trainable parameter and the optimizer's state — for mixed-precision AdamW that is about 16 bytes per parameter, against roughly 0.6 bytes for a Q4_K_M weight. For Llama 3.1 8B Instruct that is about 120 GB versus 5.5 GB.
Can I fine-tune on a consumer GPU?
With QLoRA, often yes. It freezes the base model in 4-bit and trains a small adapter, so gradients and optimizer state are charged on a few million parameters rather than billions. Full fine-tuning of a 7-8B model needs well over 100 GB and is a multi-GPU job.
Does quantization help with training memory?
Only via QLoRA, where the frozen base is held in 4-bit. You cannot simply train a quantized model the way you run one — gradients need higher precision, which is exactly the problem adapters sidestep.