Local LLM on NVIDIA RTX 3060 12GB

Is the NVIDIA RTX 3060 12GB the right card for this?

At 12 GB this is an entry point rather than a destination: small and mid-sized models run comfortably, and anything above the 14B class does not fit at Q4 without offloading to system RAM. 360 GB/s of GDDR6 and a 170 W draw make it easy to slot into an existing machine. At about $250 the memory costs roughly $21 per gigabyte, which is the best ratio in this guide's consumer range. Catalogue throughput class: Q4 7B: 14-20, Q4 3B: 30-45.

One other card here carries 12 GB, so the two load the same weights and differ only in how fast they read them. At 360 GB/s this is not the quickest of that group, so its argument is price rather than speed.

Released in 2021, this is one of the older entries in the guide, and age here is mostly an advantage: the current generation of local-AI tooling was written while this hardware was already widespread, so backend support is settled and the failure modes are documented. What it cannot have is any of the memory technology that arrived after it, and memory is what sets generation speed.

What the price gap buys

Nothing else in this guide holds 12 GB for less. The fastest alternative at this capacity is the NVIDIA RTX 5070, which costs $300 more for 87% more bandwidth and not one extra model. If the models you want already fit here, the money buys speed and nothing else.

Released in 2021, this sits between the shelf and the second-hand market: new stock still turns up, and used examples are common enough that the price above spans both. That makes it worth checking the two markets against each other before committing, because the gap can be wider than the gap to the next card up.

Where the NVIDIA RTX 3060 12GB sits in this guide

Across the 20 cards catalogued here, the NVIDIA RTX 3060 12GB ranks 16th on memory bandwidth at 360 GB/s, 3rd cheapest at about $250, and 18th on bandwidth per watt at 2.1 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 12 GB parts.

GDDR6 is the conservative choice on this card, and at 360 GB/s it sets the ceiling on tokens per second more firmly than the core clock does. When comparing this against a card with the same 12 GB but newer memory, the capacity is identical and the throughput is not.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA RTX 3060 12GB?

Of the 46 models in our catalogue with per-size memory figures, 31 have at least one size that fits in 12 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
StarCoder2	15B	10 GB
LLaVA	13B	10 GB
DeepSeek Coder V2	16B	10 GB
Qwen 2.5	14B	9 GB
Qwen 2.5-Coder	14B	9 GB
Qwen 3	14B	9 GB
DeepSeek R1	14B	9 GB
Phi-4	14B	9 GB

A further 23 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: GPT-OSS 20B at 14 GB, Mistral Small 24B at 15 GB, Magistral 24B at 15 GB.

Expected tokens per second on NVIDIA RTX 3060 12GB

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA RTX 3060 12GB's 360 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~65 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~22 tok/s
Llama 3.1 8B	Q8_0	Fits in VRAM	~16 tok/s
Mistral Nemo 12B	Q4_K_M	Fits in VRAM	~14 tok/s
Qwen 2.5 14B	Q4_K_M	Fits in VRAM	~13 tok/s
Qwen 3 32B	Q4_K_M	Needs CPU offload	~2 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA RTX 3060 12GB

Best model to download first

Llama 3.1 8B or Mistral Nemo 12B. Start with ollama pull llama3.1:8b.

Recommended inference backend

Use Ollama for general use or Mullama if you need a drop-in Ollama alternative with native bindings for 6 languages. For production serving on multi-GPU setups, see vLLM.

Sources

[1] nvidia.com/rtx-3060 · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).