Skip to content
Bitpute
← Back to workspace

Bitpute engineering report

Llama 3.1 8B · GGUF Q4_K_M

RTX 4090 · 8,192 ctx

Medium confidence

Minimum VRAM

6.51 GB

Recommended

7.16 GB

Decode speed

58.0–114

tok/s (est.)

Memory breakdown

Requirement vs. rounded capacity6.5 / 8 GB
Weights 4.5 GBKV cache 1.0 GBOverhead (est.) 1.0 GB

Monthly cost

Cloud rental (on-demand)
$248 – $540
Local electricity
$28

Cloud figures are indicative on-demand ranges as of July 2026; hardware purchase price is not included.

Analysis

  • Llama 3.1 8B at GGUF Q4_K_M needs about 6.51 GB per GPU, and your RTX 4090 has room to spare. This is a comfortable fit.
  • Expect roughly 58.0–114 tokens per second for single-stream generation, bounded by the RTX 4090's 1,008 GB/s memory bandwidth. Batching multiple requests raises total throughput well above this.
  • Renting this in the cloud runs about $0.34–$0.74 per hour on-demand ($248–$540 a month at full utilisation). Running your own card instead costs about $28 a month in electricity at your rate — before the hardware itself.
  • RTX 3060 12GBComfortable fit, ~5.1 GB free · from $0.10/GPU-hr cloud
  • RTX 4060 Ti 16GBComfortable fit, ~9.1 GB free · from $0.12/GPU-hr cloud
  • RTX 4070 Ti SUPERComfortable fit, ~9.1 GB free · from $0.18/GPU-hr cloud
  • RTX 3090Comfortable fit, ~17.1 GB free · from $0.20/GPU-hr cloud

Confidence

Medium confidence. Memory is computed from exact formulas; runtime overhead, inference speed (decode), cloud cost & local electricity are engineering estimates and are labeled as such.

References

  • llama.cpp project — Quantization formats and effective bits-per-weight (Model weight memory)
  • Frantar et al. — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Model weight memory)
  • Lin et al. — AWQ: Activation-aware Weight Quantization for LLM Compression (Model weight memory)
  • IEEE / Google Brain — IEEE 754 floating-point & bfloat16 storage widths (Model weight memory)
  • Hugging Face — KV cache & attention memory in the Transformers library (KV cache memory)
  • Kwon et al. (vLLM) — Efficient Memory Management for LLM Serving with PagedAttention (KV cache memory)
  • NVIDIA — GPU product datasheets (memory capacity, bandwidth, TDP) (Runtime overhead)
  • Kwon et al. (vLLM) — Efficient Memory Management for LLM Serving with PagedAttention (Tensor parallelism)
  • NVIDIA — GPU product datasheets (memory capacity, bandwidth, TDP) (Tensor parallelism)
  • NVIDIA — GPU product datasheets (memory capacity, bandwidth, TDP) (Inference speed (decode))
  • Kwon et al. (vLLM) — Efficient Memory Management for LLM Serving with PagedAttention (Inference speed (decode))
  • NVIDIA — GPU product datasheets (memory capacity, bandwidth, TDP) (Cloud cost & local electricity)