Local LLM on NVIDIA A100 80GB

Is the NVIDIA A100 80GB the right card for this?

This is an accelerator you rent or requisition rather than one you drop into a desktop. HBM2e stacks put 2039 GB/s behind 80 GB, which is the combination that makes batched serving worthwhile: the same weights stream fast enough to answer many concurrent requests, not just one. At roughly $15000 it costs about $188 per gigabyte of memory, so nobody buys one to chat with a 7B model. Its case is throughput per rack slot, and the figure our catalogue records for it is FP16 70B: 25-35, FP16 405B: OOM (offload needed), Q4 405B: 12-18.

One other card here carries 80 GB, so the two load the same weights and differ only in how fast they read them. At 2039 GB/s this is not the quickest of that group, so its argument is price rather than speed.

Released in 2020, this is one of the older entries in the guide, and age here is mostly an advantage: the current generation of local-AI tooling was written while this hardware was already widespread, so backend support is settled and the failure modes are documented. What it cannot have is any of the memory technology that arrived after it, and memory is what sets generation speed.

What the price gap buys

Nothing else in this guide holds 80 GB for less. The fastest alternative at this capacity is the NVIDIA H100 80GB, which costs $15000 more for 64% more bandwidth and not one extra model. If the models you want already fit here, the money buys speed and nothing else.

A part at about $15000 is not bought the way a graphics card is bought. Most people meet this hardware by the hour from a cloud provider, and the figure above is useful mainly as the number to compare a rental bill against. If you are specifying one, the 400 W draw and the cooling it implies decide the chassis before the accelerator does.

Where the NVIDIA A100 80GB sits in this guide

Across the 20 cards catalogued here, the NVIDIA A100 80GB ranks 3rd on memory bandwidth at 2039 GB/s, 18th cheapest at about $15000, and 6th on bandwidth per watt at 5.1 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 80 GB parts.

HBM2e was the previous stacked-memory generation, and at 2039 GB/s it still outruns every consumer card in this guide. Second-hand and cloud availability is the reason it stays relevant in 2026 rather than any advantage over newer HBM.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA A100 80GB?

Of the 46 models in our catalogue with per-size memory figures, 41 have at least one size that fits in 80 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
Mistral Large	123B	70 GB
GPT-OSS	120B	70 GB
Llama 3.2 Vision	90B	55 GB
Qwen 2.5	72B	44 GB
Llama 3.1	70B	42 GB
Llama 3.3	70B	42 GB
DeepSeek R1	70B	42 GB
Code Llama	70B	42 GB

A further 33 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: Qwen 3 235B at 130 GB, minimax M2 230B at 130 GB, minimax M2.1 230B at 130 GB.

Expected tokens per second on NVIDIA A100 80GB

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA A100 80GB's 2039 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~367 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~122 tok/s
Llama 3.1 8B	Q8_0	Fits in VRAM	~92 tok/s
Mistral Nemo 12B	Q4_K_M	Fits in VRAM	~82 tok/s
Qwen 2.5 14B	Q4_K_M	Fits in VRAM	~71 tok/s
Qwen 3 32B	Q4_K_M	Fits in VRAM	~37 tok/s
DeepSeek R1 distilled 32B	Q4_K_M	Fits in VRAM	~37 tok/s
Llama 3.1 70B	Q4_K_M	Fits in VRAM	~20 tok/s
Llama 4 Scout 109B (MoE)	Q4_K_M	Fits in VRAM	~16 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA A100 80GB

Best model to download first

Llama 3.1 8B to check the stack works, then the largest model in the table above that you actually need. At 80 GB the constraint stops being capacity. Start with ollama pull llama3.1:8b.

Recommended inference backend

At this capacity the point is serving several people at once, so use vLLM or TensorRT-LLM for continuous batching rather than a single-stream runner.

Sources

[1] nvidia.com/a100 · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).