Local LLM on NVIDIA RTX 5090

Is the NVIDIA RTX 5090 the right card for this?

32 GB is the capacity most local-LLM tooling is written around, and this card sits in that bracket at 1792 GB/s. Released in 2025, it is 1 year old in 2026 and sells for about $2500, or roughly $78 per gigabyte of VRAM. It draws up to 575 W, so the power supply and the case airflow are part of the budget, not an afterthought. Catalogue throughput class: Q4 7B: 90-120, Q4 32B: 35-45, Q4 70B: 15-22.

At 2025 this is among the newest entries here, which cuts both ways. The memory technology is current, and the runtime support is the least settled of anything in this guide: check that the backend you intend to use has shipped a release naming this generation before buying for it.

What the price gap buys

No other card in this guide carries 32 GB, so there is no like-for-like price comparison to make. The nearest capacities are listed at the foot of this page, and moving to any of them changes which models fit, not just how fast they run.

At 1 year old this is still a current retail part, so the price above is a shop price rather than a listing price and stock is the usual constraint rather than condition. Buying new also means the 575 W figure is the number your power supply has to meet from day one.

Where the NVIDIA RTX 5090 sits in this guide

Across the 20 cards catalogued here, the NVIDIA RTX 5090 ranks 4th on memory bandwidth at 1792 GB/s, 14th cheapest at about $2500, and 11th on bandwidth per watt at 3.1 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 32 GB parts.

GDDR7 is the current generation of board-mounted graphics memory, and it is the reason this card reaches 1792 GB/s on a 32 GB bus where the previous generation needed a wider one. For token generation, which reads every weight once per token, that bandwidth translates almost linearly into speed.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA RTX 5090?

Of the 46 models in our catalogue with per-size memory figures, 39 have at least one size that fits in 32 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
LLaVA	34B	22 GB
Command R	35B	22 GB
Qwen 2.5	32B	20 GB
Qwen 2.5-Coder	32B	20 GB
Qwen 3	32B	20 GB
DeepSeek R1	32B	20 GB
DeepSeek Coder	33B	20 GB
Gemma 4	31b	20 GB

A further 31 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: Llama 3.1 70B at 42 GB, Llama 3.3 70B at 42 GB, DeepSeek R1 70B at 42 GB.

Expected tokens per second on NVIDIA RTX 5090

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA RTX 5090's 1792 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~323 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~108 tok/s
Llama 3.1 8B	Q8_0	Fits in VRAM	~81 tok/s
Mistral Nemo 12B	Q4_K_M	Fits in VRAM	~72 tok/s
Qwen 2.5 14B	Q4_K_M	Fits in VRAM	~63 tok/s
Qwen 3 32B	Q4_K_M	Fits in VRAM	~32 tok/s
DeepSeek R1 distilled 32B	Q4_K_M	Fits in VRAM	~32 tok/s
Llama 3.1 70B	Q4_K_M	Needs CPU offload	~5 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA RTX 5090

Best model to download first

Llama 3.1 8B for everyday chat and coding; a 32B at Q4_K_M for top quality within 32 GB. Start with ollama pull llama3.1:8b.

Recommended inference backend

Use Ollama for general use or Mullama if you need a drop-in Ollama alternative with native bindings for 6 languages. For production serving on multi-GPU setups, see vLLM.

Sources

[1] nvidia.com/rtx-5090 · [2] community benchmarks · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).