GPU guide

RTX 5090 for AI: what it actually runs

By Bitpute · Published 12 July 2026 · Updated 24 July 2026 · Specs verified against manufacturer datasheets 24 July 2026 · How we estimate · Sources · Editorial policy · Version history · Report an error

AI suitability

Here is the honest subtlety: 32 GB does not unlock a bigger model tier than 24 GB does — the next size up, Llama 3.3 70B, needs about 45 GB and stays out of reach. What the extra 8 GB buys is quality and context on the 32B class: Qwen2.5 32B fits at Q6_K (about 29.1 GB) instead of being pinned to Q4, and there is real room for longer contexts. Add bandwidth nearly double a 4090's, and the 5090 is the best single-card experience below the workstation tier — just not a bigger-model ticket.

For LLM inference the number that matters most is memory bandwidth, because generating each token means reading the whole model. On the RTX 5090 that puts the theoretical ceiling around 368 tokens/second on Llama 3.1 8B at Q4_K_M, and about 90 tokens/second on the biggest model it comfortably holds, Qwen2.5 32B — real-world throughput lands below these ceilings. At Q4_K_M with an 8K context this card comfortably fits 10 of the 12 models in our database; at the near-lossless Q8_0, 7.

Which models fit the RTX 5090?

Computed at an 8K context (or the model's own cap). ✓ fits comfortably (≤95% of 32 GB) · ~ tight · ✗ doesn't fit. Every model links to its own guide.

Electricity

The board is rated at 575 W. Run it under sustained load 8 hours a day and that's about 138 kWh a month — roughly $17/month at an example rate of $0.12/kWh (set your own tariff in the workspace). Idle and light chat draw far less; the figure above is the worst case, not the typical bill.

Renting instead of buying

Major clouds don't rack consumer GeForce cards, but GPU marketplaces rent the RTX 5090 directly by the hour — often the cheapest way to test before buying. If you want a datacenter instance that covers the same model set, the smallest step is the RTX A6000 (48 GB). Price the rent-vs-buy question properly in the workspace, which models cloud rental cost and electricity side by side.

Alternatives

Compare any two of these head-to-head — speed on the same model, cost, power — in GPU Compare.

Why bandwidth matters more than core count for token generation: memory bandwidth explained.

Capacity against speed

The two axes are independent. Position on the horizontal decides which models fit; position on the vertical decides how fast they decode once they do. Cards up and to the left of it hold less but read faster; A100 40GB sits lower-right — more memory, less bandwidth.

If a model will not load on this card

In rough order of likelihood. For reference, the largest model in our catalogue that fits this card at Q4_K_M is Nemotron Super 49B Instruct at 29.8 GB.

1. The model is larger than usable VRAM, not nameplate VRAM. This card reports 32 GB but roughly 30.4 GB is available to a model after the display buffer, driver and CUDA context. A model sized against the nameplate figure will appear to fit and then fail.
2. KV cache grew past the headroom. Weights are fixed; the cache is not. A load that succeeds at 2K context can fail at 32K on the same card, because the cache comes out of the same budget. If it loaded yesterday and fails today, context length is the first thing to check. See KV cache.
3. The quantization is heavier than assumed. Q4_K_M is roughly 4.85 bits per weight, not 4. On a large model that difference is gigabytes. Confirm which file you actually downloaded — see which quantization to run.
4. Something else is already holding VRAM. A browser with hardware acceleration, another model still resident, or a previous process that did not release memory. On Linux nvidia-smi shows what is allocated.
5. It loaded but generation is very slow. That usually means layers were offloaded to system RAM rather than the load failing outright. The model runs, but every offloaded layer crosses PCIe on each token. Reduce context, drop a quantization level, or use a card with more VRAM.

Common questions

Is the RTX 5090 good for AI and local LLMs?

Yes, within its tier: with 32 GB of VRAM and 1792 GB/s of memory bandwidth it comfortably runs 10 of the 12 models in our database at Q4_K_M with an 8K context, the largest being Qwen2.5 32B. Memory bandwidth caps generation speed at roughly 368 tokens/second on an 8B model in theory, with real-world results lower.

What is the biggest model an RTX 5090 can run?

Qwen2.5 32B at Q4_K_M with an 8K context is the largest comfortable fit (22.2 GB of 32 GB). Larger models need a bigger card or a multi-GPU split.

How much electricity does an RTX 5090 use?

The board is rated at 575 W. Under sustained load for 8 hours a day that is about 138 kWh a month — roughly $17/month at an example rate of $0.12/kWh. Idle draw is far lower, so light interactive use costs much less.

Consumer cards, ranked by VRAM

Gaming cards — the cheapest route to VRAM. No ECC memory, and on NVIDIA no NVLink from Ada onward. Every card in this tier, smallest memory first — the one you are reading is highlighted. Cards shown without a memory figure are not in our calculation engine yet.

Where it sits in the stack

The RTX 5090 is a Blackwell-generation part with GDDR7 memory and 5th-generation Tensor Cores. GDDR keeps the card affordable but caps bandwidth well below the HBM used in datacentre cards — and bandwidth, not core count, is what sets decode speed once a model fits.

It has no NVLink. Multiple cards talk over PCIe 5.0 instead, routed through the CPU, so splitting one model across two of these costs noticeably more than it would on an NVLink pair. Two cards each holding their own model is the friendlier pattern here.

Its Tensor Cores handle FP16, BF16, INT8, FP8 and FP4. FP4 is new to Blackwell and is not available on the Hopper datacentre cards, which stop at FP8. Most local inference runs weights at 4-6 bits via GGUF quantization rather than a native Tensor Core format, so these matter most for training and for server runtimes such as TensorRT-LLM and vLLM.

Blackwell Tensor Cores add FP4 — a precision the Hopper cards above it do not support. None of this changes the VRAM arithmetic on this page — weights and KV cache still have to fit.

On the software side it is a CUDA device like any other NVIDIA card, so PyTorch, TensorRT, vLLM, llama.cpp and Ollama all run on it unchanged. What differs between cards is not compatibility but how much fits and how fast it decodes — which is what the numbers above measure. For the quantization formats these runtimes expect, and how much room the GPU Memory Calculator says you have left, start there.

Head to head

Pre-computed comparisons against the cards it is usually weighed against:

RTX 3090 vs RTX 5090RTX 4090 vs RTX 5090RTX 5090 vs H100 80GBRTX 5090 vs L40SRTX 5090 vs RTX A6000