Learn · Concepts
GPU concepts for running LLMs locally
The short version
- VRAM decides what runs. Not enough and the model will not load.
- Memory bandwidth decides how fast it runs, once it fits.
- Quantization decides how much VRAM you need in the first place.
- Everything else — CUDA, Tensor Cores, PCIe, precision formats — matters, but only after those three.
GPU specifications are a long list, and for running language models locally most of it does not matter. What follows maps how the concepts relate: which decides what, in what order, and where each is documented in detail.
1The order that actually matters
Three questions, in this sequence:
1. Does it fit? Determined by VRAM and by the quantization you choose. Nothing else matters until this is yes.
2. How fast will it generate? Determined by memory bandwidth, because each token requires reading every weight. The H200 141GB at 4800 GB/s sits at the top of our set.
3. How much context can you hold? Determined by KV cache, which grows with conversation length and comes out of the same VRAM budget.
2The supporting cast
These appear on every spec sheet and are worth understanding, but none of them changes the answer to the three questions above.
CUDA is NVIDIA’s compute platform. Every runtime worth using — PyTorch, TensorRT-LLM, vLLM, llama.cpp, Ollama — targets it, so on NVIDIA hardware compatibility is rarely the issue. What differs between cards is capacity and speed, not whether software runs.
Tensor Cores are dedicated matrix-multiply units. Their generation determines which numeric formats the silicon accelerates: Ampere added BF16, Ada and Hopper added FP8, Blackwell added FP4. INT8 has been available since Turing. Each GPU page states which its card supports.
FP16, BF16 and INT8 are numeric formats. FP16 and BF16 are both 16-bit but spend their bits differently — BF16 trades precision for the same exponent range as FP32, which makes it more forgiving in training. For local inference, weights are usually held in 4-6 bit GGUF quants instead, which is a different mechanism from Tensor Core formats.
PCIe is how the card talks to the rest of the machine. It matters when a model is split across cards or offloaded to system RAM; for a model that fits on one GPU it is close to irrelevant, because the weights are already resident.
3Training versus running
Everything above concerns inference — running a model that already exists. Training the same model costs several times more, because gradients and optimizer state are stored per trainable parameter. A card that runs a model comfortably may not be able to train it at all.
4Where to go next
By what you are trying to do:
Work out if a specific model fits: the GPU Memory Calculator, or the model database for a pre-computed answer on 506 models.
Choose between two cards: GPU Compare lines them up on the specs that decide it.
Understand the arithmetic: the methodology gives every formula, and common mistakes lists the ways a correct-looking calculation goes wrong.
5Common questions
What GPU specification matters most for running LLMs?
VRAM capacity, because it decides whether a model loads at all. Memory bandwidth is second, because it sets how fast tokens are generated once the model fits. Core count and clock speed matter far less for single-stream inference, which is memory-bound rather than compute-bound.
Do I need to understand CUDA to run a model locally?
No. Runtimes such as Ollama, llama.cpp and vLLM handle it. CUDA matters for compatibility — on NVIDIA hardware it is broadly a solved problem — not for deciding which card to buy.
What is the difference between FP16 and BF16?
Both are 16-bit. BF16 keeps the same exponent range as FP32 at the cost of mantissa precision, which makes it more numerically forgiving during training. FP16 has more precision but a narrower range. For local inference, weights are typically stored in 4-6 bit GGUF quantization instead of either.