Why VRAM matters more than anything else for local AI
On this page
- Capacity decides what you can run
- Bandwidth decides how fast
- What happens when you run out
- The practical VRAM tiers
For running LLMs on your own machine, VRAM is the hard wall — it decides what you can run at all, and its bandwidth decides how fast. Raw compute barely enters into it. Here's why.
People shopping for an AI GPU instinctively look at teraflops. For inference — actually running a model — that's the wrong number. Two things about your VRAM decide almost everything: its capacity and its bandwidth.
1Capacity decides what you can run
A model has to fit — weights + overhead + KV cache — inside VRAM. If it doesn't, you have three options, all bad: spill part of it to system RAM (a large slowdown), run on CPU (much slower still), or simply can't load it. So capacity is a yes/no gate: a 24 GB card runs models a faster 12 GB card physically cannot, no matter how quick the 12 GB card's cores are.
2Bandwidth decides how fast
Token generation is memory-bandwidth bound: to produce each token, the GPU reads the active weights from VRAM once. So the speed ceiling is simply how fast it can read memory:
That's why memory bandwidth (GB/s), not core count, is the spec that predicts tokens/second. It also explains why lower quantization is faster — fewer bytes to read per token. (Full derivation in the methodology.)
3What happens when you run out
- Offload to system RAM. Layers that don't fit run over the PCIe bus from RAM — often a 5–20× slowdown for the offloaded portion.
- CPU inference. Possible for small models with enough RAM, but slow.
- Out of memory. The load just fails.
None are a substitute for fitting in VRAM. That's why sizing first — before you buy or download — saves the most pain. Start with the VRAM calculator.
4The practical VRAM tiers
| VRAM | What it comfortably runs (Q4_K_M) |
|---|---|
| 12 GB | Up to ~14B models — great for chat & coding |
| 16 GB | 14B comfortably, 32B tight |
| 24 GB | 32B models, or 70B with offload |
| 48 GB | 70B-class models on a single card |
Pick your card by the model size you actually want to run, then check bandwidth for speed — in that order.
Related: what VRAM is.
5Common questions
Why does VRAM matter more than any other GPU spec for LLMs?
Because it is a hard limit. If a model does not fit, it either fails to load or falls back to system RAM at a large speed penalty. Every other specification only matters once the model fits.
What happens when you run out of VRAM?
Either an out-of-memory error, or the runtime offloads layers to system RAM. The second keeps the model running but collapses decode speed, because offloaded layers cross PCIe on every token.
How much VRAM do I actually need?
It depends on parameter count and quantization. Usable capacity is about 95% of nameplate, so a 24 GB card gives roughly 22.8 GB — and KV cache for your context comes out of the same budget.