Six formulas power every tool on the site.
Model weight memory
verifiedweights_bytes = parameters × effective_bpw ÷ 8
Storing N parameters at b bits each takes N × b ÷ 8 bytes. The only subtlety is that quantized formats store metadata (block scales, zero-points) alongside the quantized values, so the effective bits-per-weight is higher than the nominal bit count.
Assumptions
- ·Effective bits-per-weight is used, not nominal — GGUF Q4_K_M is ~4.85 bpw, not 4.0, once block scales are counted.
- ·GPTQ and AWQ 4-bit assume group size 128, adding ~0.25 bpw of scale/zero metadata.
- ·1 GB means 1 GiB (1024³ bytes), matching nvidia-smi.
Edge cases
- ·Mixed-precision checkpoints (e.g. some layers FP16, some INT8) are approximated by their dominant format.
- ·Embedding and LM-head tensors are included in the published parameter count.
llama.cpp projectFrantar et al.Lin et al.IEEE / Google Brain
KV cache memory
verifiedkv_bytes = 2 × layers × kv_heads × head_dim × context × batch × bytes_per_element
During generation, the model stores the key and value tensors of every previous token so it need not recompute them. The leading 2 covers K and V. Using kv_heads (not attention_heads) makes this correct for grouped-query attention, where several query heads share one KV head.
Assumptions
- ·Grouped-query attention is handled by using the model's kv_heads from its public config.json.
- ·bytes_per_element is 2 for an FP16 cache, 1 for an FP8/INT8 cache.
- ·Architecture values (layers, kv_heads, head_dim) are read from each model's published config.json.
Edge cases
- ·Sliding-window attention (e.g. Mistral) can cap the effective context below the requested length; we compute the full-context upper bound.
- ·Paged-attention serving stacks (vLLM, TGI) allocate the cache in blocks, so real usage is slightly lower than this contiguous estimate.
Hugging FaceKwon et al. (vLLM)
Runtime overhead
estimatedoverhead_bytes = 0.75 GB × gpus + 0.05 × weights_bytes
The CUDA context, framework buffers, dequantization scratch and allocator fragmentation cannot be derived from first principles. We model them as a fixed per-GPU cost plus a small fraction of the weight footprint, informed by published memory-footprint numbers from llama.cpp and vLLM deployments.
Assumptions
- ·0.75 GB per GPU covers the CUDA context and framework buffers.
- ·5% of weights covers dequantization and activation scratch during inference.
- ·This term is an engineering estimate, not exact arithmetic — labeled as such throughout the UI.
Edge cases
- ·Training (not just inference) adds optimizer state and gradients that this inference-focused term does not model.
- ·Consumer cards driving a display lose a further ~0.4 GB, accounted for separately in the GPU fit check.
NVIDIA
Tensor parallelism
verifiedper_gpu = (weights + kv) ÷ N + 0.75 GB + (0.05 × weights) ÷ N + comm ; comm = 0.02 × (weights + kv)
Splitting a model across N GPUs shards the weights and KV cache roughly evenly, so per-GPU memory drops close to 1/N. Each GPU still pays its own fixed context cost, and NCCL communication buffers add a small overhead.
Assumptions
- ·Weights and KV cache shard evenly across GPUs.
- ·Communication buffers are ~2% of the sharded footprint.
- ·per_gpu × N reconciles exactly with the reported total (verified by test).
Edge cases
- ·When layer count is not divisible by N, pipeline splits are uneven — the UI warns about this.
- ·Real tensor-parallel speed depends on interconnect (NVLink vs PCIe); we model memory, not that speed effect.
Kwon et al. (vLLM)NVIDIA
Inference speed (decode)
estimatedtokens_per_sec ≈ (memory_bandwidth × gpus) ÷ model_bytes × MBU ; realistic MBU band 0.28–0.55
Generating one token requires reading every weight from memory once, so decode speed is bounded by memory bandwidth divided by model size. No kernel hits 100% bandwidth utilisation, and attention plus launch overhead take a larger share on big models, so we report a realistic band that brackets observed llama.cpp and vLLM single-stream numbers rather than the theoretical ceiling.
Assumptions
- ·Single-stream, decode-phase (not prefill) throughput.
- ·The pure roofline is a ceiling; the reported 0.28–0.55 band is the realistic range.
- ·Batching many requests raises aggregate throughput well above this per-stream figure.
Edge cases
- ·Very small models on very fast GPUs become compute- or launch-bound, where this memory-bound estimate over-predicts.
- ·Multi-GPU speed depends on interconnect bandwidth, folded into the band rather than modelled explicitly.
NVIDIAKwon et al. (vLLM)
Cloud cost & local electricity
estimatedcloud/mo = $/gpu-hr × gpus × 730 ; electricity/mo = (TDP × gpus × util ÷ 1000) × 730 × rate
Cloud rental is the per-GPU hourly rate times GPU count times hours in a month. Local electricity is the board power drawn under load (a fraction of TDP) converted to kilowatt-hours and multiplied by the user's rate. Hardware purchase price is not included.
Assumptions
- ·730 hours per month (24 × 365 ÷ 12).
- ·Cloud rates are indicative on-demand market ranges with an explicit 'as of' date; GPU prices move weekly.
- ·Utilisation defaults to 70% of TDP; the user can adjust it.
Edge cases
- ·Spot/preemptible pricing is far lower than the on-demand range shown.
- ·Storage, egress and idle time are not included in the hourly figure.
NVIDIA
Full fine-tuning memory (mixed-precision Adam)
verifiedbytes ≈ 16 × trainable_params + activations (2 fp16 weight + 2 fp16 grad + 4 fp32 master + 4 fp32 m + 4 fp32 v)
Full fine-tuning trains every parameter. Mixed-precision AdamW keeps, for each trainable parameter, an fp16 weight, an fp16 gradient, and three fp32 tensors (master copy, momentum, variance) — sixteen bytes in total. This is why full fine-tuning a 7B model needs well over 100 GB and usually spans multiple GPUs.
Assumptions
- ·AdamW optimizer with the standard fp32 master-weight + m + v state.
- ·Weights and gradients held in fp16/bf16 (2 bytes each).
- ·Activation memory is estimated separately and is the least precise term.
Edge cases
- ·8-bit optimizers (bitsandbytes) cut the 12 bytes of fp32 state substantially.
- ·ZeRO/FSDP sharding splits weights, gradients and optimizer state across GPUs, lowering per-GPU memory.
Rajbhandari et al. (Microsoft)Hugging Face
LoRA / QLoRA fine-tuning memory
verifiedbytes ≈ base_params × base_bpw/8 + 16 × adapter_params + activations (QLoRA base_bpw ≈ 4.5)
LoRA freezes the base model (weights only, no gradients or optimizer state) and trains small low-rank adapters — typically well under 1% of the parameters. Only those adapters carry the 16-byte Adam state, so the base weights dominate memory. QLoRA additionally stores the frozen base in 4-bit (NF4 ≈ 4.5 bits/weight), which is what lets a 70B model fine-tune on a single 48 GB GPU.
Assumptions
- ·Adapter parameter count approximated from architecture (rank × targeted linear-layer dimensions).
- ·Adapters trained in bf16 with fp32 Adam state (16 bytes each).
- ·QLoRA base stored at ~4.5 effective bits/weight (NF4 with double quantization).
Edge cases
- ·Targeting all linear layers (not just attention) multiplies adapter count several-fold.
- ·Intermediate (MLP) size is approximated as 3.5× hidden when it is not in the model record, so the trainable-parameter figure is indicative.
Hu et al. (Microsoft)Dettmers et al.