Learn · Concepts

Memory bandwidth (the number that sets your tokens per second)

By Bitpute · Published 24 July 2026 · Updated 24 July 2026 · How we estimate · Sources · Editorial policy · Version history · Report an error

In short

Memory bandwidth sets decode speed once a model fits. Generating each token requires reading every weight, so tokens per second is bounded by bandwidth divided by weight size.
Core count is not the limit for single-stream inference — the GPU spends most of its time waiting on memory.
Cards in our set span 288 GB/s (RTX 4060 Ti 16GB) to 4800 GB/s (H200 141GB) — a 16.7× spread.
More VRAM does not imply more bandwidth. The RTX 4060 Ti 16GB holds larger models than the RTX 3060 12GB but reads them more slowly.

Memory bandwidth is how fast a GPU can move data between its VRAM and its compute units, measured in gigabytes per second. Once a model fits in memory, this is the number that decides how fast it generates text — not core count, not clock speed.

1Why bandwidth sets decode speed

Generating one token requires reading every weight in the model once. That makes single-stream decoding memory-bound: the arithmetic is trivial next to the cost of fetching the weights. So the ceiling is simply:

tokens/sec ≤ memory bandwidth ÷ model size in memory

For Llama 3.1 8B Instruct at Q4_K_M (4.5 GB of weights) on a H200 141GB at 4800 GB/s, that ceiling is about 1059 tokens/sec. Real throughput is lower — this is an upper bound from bandwidth alone, not a benchmark.

2The bandwidth ladder

Every GPU in our set, fastest first, with the theoretical ceiling for Llama 3.1 8B Instruct at Q4_K_M:

Card	Bandwidth	Ceiling for a 4.5 GB model
H200 141GB	4800 GB/s	1059 tok/s
H100 80GB	3350 GB/s	739 tok/s
A100 80GB	2039 GB/s	450 tok/s
RTX 5090	1792 GB/s	395 tok/s
A100 40GB	1555 GB/s	343 tok/s
RTX 4090	1008 GB/s	222 tok/s
RTX 3090	936 GB/s	206 tok/s
L40S	864 GB/s	191 tok/s
RTX A6000	768 GB/s	169 tok/s
RTX 4070 Ti SUPER	672 GB/s	148 tok/s
RTX 3060 12GB	360 GB/s	79 tok/s
RTX 4060 Ti 16GB	288 GB/s	64 tok/s

Note the spread is larger than the VRAM spread. Two cards can hold the same model and differ by more than 3× in how fast they run it.

3HBM versus GDDR

Datacentre cards use HBM — memory stacked vertically beside the die on a very wide bus. Consumer cards use GDDR on a conventional bus. HBM delivers far more bandwidth per watt, which is why an H100 reads memory several times faster than a flagship gaming card despite similar generation. It is also why the datacentre parts cost what they do. Which memory each card uses is stated on its GPU page.

4When bandwidth stops being the limit

Two cases. Batching: serving many requests at once amortises each weight read across several tokens, so throughput becomes compute-bound rather than memory-bound. Prompt processing: reading a long prompt is parallel and compute-heavy, unlike token-by-token generation. Our figures assume single-stream decoding at batch 1, which is the local-inference case — stated in the methodology.

5Common questions

Does more VRAM mean a faster GPU?

No. VRAM decides what fits; bandwidth decides how fast it runs. The RTX 4060 Ti 16GB holds larger models than the RTX 3060 12GB but has lower bandwidth (288 GB/s against 360 GB/s), so on models both can run it is roughly 20% slower.

How do I estimate tokens per second?

Divide memory bandwidth by the model's size in memory. A 4.5 GB model on a 4800 GB/s card gives an upper bound near 1059 tokens/sec. Real output is lower; this is a ceiling, not a measurement.

Why is memory bandwidth the bottleneck and not compute?

Generating each token requires reading every weight once, while the arithmetic per weight is tiny. The GPU finishes the maths long before the next weights arrive, so it waits on memory.