GPU Memory Calculator
Exact VRAM requirements for LLM inference: weights, a GQA-aware KV cache, and runtime overhead — for 13 quantization formats and multi-GPU tensor parallelism. Every input updates the estimate instantly.
Most popular quality/size balance
Minimum VRAM
6.51 GB
Recommended (10% headroom)
7.16 GB
Total footprint
6.51 GB
Memory breakdown
Will it fit?
- FitsRTX 3060 12GBconsumer5.09 GB free
- FitsRTX 4060 Ti 16GBconsumer9.09 GB free
- FitsRTX 4070 Ti SUPERconsumer9.09 GB free
- FitsRTX 3090consumer17.1 GB free
- FitsRTX 4090consumer17.1 GB free
- FitsRTX 5090consumer25.1 GB free
- FitsA100 40GBdatacenter33.3 GB free
- FitsRTX A6000workstation41.1 GB free
- FitsL40Sdatacenter41.3 GB free
- FitsA100 80GBdatacenter73.3 GB free
- FitsH100 80GBdatacenter73.3 GB free
- FitsH200 141GBdatacenter134 GB free
Methodology
How the numbers are computed
Weights. Parameters × effective bits per weight ÷ 8. Full-precision rates (FP32 = 32, FP16/BF16 = 16, INT8/FP8 = 8) are exact by definition. GGUF effective rates come from the llama.cpp quantization documentation and are verifiable against real file sizes — Q4_K_M stores about 4.85 bits per weight once block scales are included. GPTQ and AWQ 4-bit assume group size 128, which adds roughly 0.25 bpw of scale and zero-point metadata per the original papers' packing schemes.
KV cache. 2 × layers × KV heads × head dimension × context length × batch × bytes per element. This is the standard formula used by vLLM (PagedAttention paper, §2) and the Hugging Face transformers documentation. Architecture values for each preset are read from the model's public config.json — using KV heads, not attention heads, so grouped-query attention is handled correctly.
Overhead. The CUDA context, framework buffers and allocator fragmentation cannot be derived from first principles. We model them as 0.75 GB per GPU plus 5% of the weight footprint — an engineering estimate, labeled as such, informed by published memory-footprint numbers from llama.cpp and vLLM deployments. Consumer cards driving a display lose a further ~0.4 GB, which the fit check accounts for.
Units. 1 GB on this page means 1 GiB (1024³ bytes), matching what nvidia-smi reports.
FAQ
About this calculator
- What is the KV cache and why does it matter?
- During generation, a transformer stores the key and value tensors of every previous token so it does not recompute them. That store — the KV cache — grows linearly with context length and batch size. At long contexts it can exceed the size of the model weights, which is why a calculator that ignores it is wrong exactly when you need it most.
- Why is GGUF Q4_K_M listed as 4.85 bits per weight, not 4?
- K-quant formats store per-block scale factors alongside the 4-bit values. Counting that metadata, the real storage cost is about 4.85 bits per weight — you can verify this from any Q4_K_M file size on Hugging Face. Using the nominal 4 bits would underestimate weights by roughly 20%.
- How does tensor parallelism change the requirement?
- Weights and KV cache shard roughly evenly across the GPUs, so per-GPU memory drops close to 1/N — but each GPU still pays its own fixed runtime overhead, and NCCL communication buffers add a small extra cost. The calculator models all three effects.
- Should I use an FP8 KV cache?
- If your serving stack supports it (vLLM and TensorRT-LLM do), an 8-bit KV cache halves cache memory with minimal quality impact for most workloads. Toggle it in the calculator to see the difference at your context length.