Learn

Learn · Concepts

What is VRAM? (and why it decides what you can run)

By Bitpute · Published 24 July 2026 · Updated 24 July 2026 · How we estimate · Sources · Editorial policy · Version history · Report an error

VRAM — video RAM — is the memory attached directly to a GPU. For language-model inference it is the constraint that decides what you can run at all, before any question of speed. What occupies it, why the usable figure is lower than the number on the box, and how to reason about the limit.

1What actually occupies VRAM

Three things share the budget, in this order of size for most setups:

Model weights. The dominant term. At Q4_K_M a parameter costs about 4.85 bits, so Llama 3.1 8B Instruct needs roughly 4.5 GB of weights.

Runtime overhead. CUDA context, activation buffers and allocator slack. We model this as 0.75 GB plus 5% of weight size. The derivation is in the methodology.

KV cache. Grows with context length and can exceed the weights on a small model with a long window. It has its own page because it is the term people forget.

2Usable VRAM is not nameplate VRAM

A 24 GB card does not give you 24 GB. The display framebuffer, the driver, the CUDA context and memory fragmentation all take a share before your model loads. We size against 95% of nameplate, which is deliberately conservative. A model sized to the full figure is the most common way an apparently-fitting setup throws an out-of-memory error.

3VRAM versus system RAM

They are separate budgets. System RAM is larger and far cheaper, but an order of magnitude slower to reach from the GPU. Runtimes can offload layers to system RAM when VRAM runs out; the model then runs, but decode speed collapses because every offloaded layer crosses PCIe on each token. Offloading is a fallback, not a plan. How much system RAM you actually need has its own page.

4How to need less of it

In order of how much they buy you:

Quantize harder. Moving from FP16 to Q4_K_M cuts weights by roughly 3.3×. This is the single largest lever and usually the first one to reach for. See which quantization to run.

Shorten the context. KV cache scales linearly with it. Halving context halves that term.

Pick a smaller model. Obvious, but worth stating: a well-chosen 8B often beats a badly-quantized 70B that barely fits.

Exact figures for your combination come from the GPU Memory Calculator.

5Common questions

How much VRAM do I need to run an LLM?

It depends on parameter count and quantization. Llama 3.1 8B Instruct needs about 5.5 GB at Q4_K_M, which fits 12 of the 12 cards in our set. Larger models scale roughly linearly with parameter count.

Is VRAM the same as system RAM?

No. VRAM is on the graphics card and is what the GPU reads model weights from. System RAM is separate, larger, cheaper and much slower to reach. A model that does not fit in VRAM can spill into system RAM, but decode speed drops sharply.

Why can't I use all of my card's VRAM?

The display buffer, driver, CUDA context and allocator fragmentation consume part of it before your model loads. We size against 95% of nameplate capacity: about 22.8 GB on a 24 GB card.

Related: how memory bandwidth sets decode speed and what KV cache costs.

See also: how the specs relate.