Local LLM on NVIDIA RTX 6000 Ada

Is the NVIDIA RTX 6000 Ada the right card for this?

A workstation card trades peak bandwidth for capacity and a power budget a desk can live with: 48 GB at 960 GB/s inside 300 W, which is 3.2 GB/s per watt. That matters because it can sit under a desk running all day without a server chassis or a dedicated circuit. At about $6800 the memory works out to roughly $142 per gigabyte, well above consumer pricing; what you buy with the premium is a single card that holds a model two consumer cards would have to split. Catalogue throughput class: Q4 70B: 25-35, Q4 32B: 50-70, FP16 70B: OOM at 48GB.

A 2023 part sits in the middle of this guide's range: old enough that backend support has stopped moving and the community has written up the quirks, new enough that nothing treats it as legacy. That is the least risky place on the calendar to buy from if you want the setup to work the first time.

What the price gap buys

No other card in this guide carries 48 GB, so there is no like-for-like price comparison to make. The nearest capacities are listed at the foot of this page, and moving to any of them changes which models fit, not just how fast they run.

Released in 2023, this sits between the shelf and the second-hand market: new stock still turns up, and used examples are common enough that the price above spans both. That makes it worth checking the two markets against each other before committing, because the gap can be wider than the gap to the next card up.

Where the NVIDIA RTX 6000 Ada sits in this guide

Across the 20 cards catalogued here, the NVIDIA RTX 6000 Ada ranks 7th on memory bandwidth at 960 GB/s, 17th cheapest at about $6800, and 10th on bandwidth per watt at 3.2 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 48 GB parts.

GDDR6 is the conservative choice on this card, and at 960 GB/s it sets the ceiling on tokens per second more firmly than the core clock does. When comparing this against a card with the same 48 GB but newer memory, the capacity is identical and the throughput is not.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA RTX 6000 Ada?

Of the 46 models in our catalogue with per-size memory figures, 40 have at least one size that fits in 48 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
Llama 3.1	70B	42 GB
Llama 3.3	70B	42 GB
DeepSeek R1	70B	42 GB
Code Llama	70B	42 GB
Hermes 3	70B	42 GB
Cogito	70B	42 GB
LLaVA	34B	22 GB
Command R	35B	22 GB

A further 32 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: Qwen 2.5 72B at 44 GB, Llama 3.2 Vision 90B at 55 GB, Mistral Large 123B at 70 GB.

Expected tokens per second on NVIDIA RTX 6000 Ada

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA RTX 6000 Ada's 960 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~173 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~58 tok/s
Llama 3.1 8B	Q8_0	Fits in VRAM	~43 tok/s
Mistral Nemo 12B	Q4_K_M	Fits in VRAM	~38 tok/s
Qwen 2.5 14B	Q4_K_M	Fits in VRAM	~34 tok/s
Qwen 3 32B	Q4_K_M	Fits in VRAM	~17 tok/s
DeepSeek R1 distilled 32B	Q4_K_M	Fits in VRAM	~17 tok/s
Llama 3.1 70B	Q4_K_M	Fits in VRAM	~10 tok/s
Llama 4 Scout 109B (MoE)	Q4_K_M	Needs CPU offload	~2 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA RTX 6000 Ada

Best model to download first

Llama 3.1 8B to check the stack works, then the largest model in the table above that you actually need. At 48 GB the constraint stops being capacity. Start with ollama pull llama3.1:8b.

Recommended inference backend

Use Ollama for general use or Mullama if you need a drop-in Ollama alternative with native bindings for 6 languages. For production serving on multi-GPU setups, see vLLM.

Sources

[1] nvidia.com/rtx-6000-ada · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).