Local LLM on NVIDIA B200

Is the NVIDIA B200 the right card for this?

This is an accelerator you rent or requisition rather than one you drop into a desktop. HBM3e stacks put 8000 GB/s behind 192 GB, which is the combination that makes batched serving worthwhile: the same weights stream fast enough to answer many concurrent requests, not just one. At roughly $40000 it costs about $208 per gigabyte of memory, so nobody buys one to chat with a 7B model. Its case is throughput per rack slot, and the figure our catalogue records for it is FP16 405B: 35-50, FP8 405B: 70+.

One other card here carries 192 GB, so the two load the same weights and differ only in how fast they read them. At 8000 GB/s this is the quickest of that group, so what you gain over the others is tokens per second, not a model you could not otherwise open.

A 2024 part sits in the middle of this guide's range: old enough that backend support has stopped moving and the community has written up the quirks, new enough that nothing treats it as legacy. That is the least risky place on the calendar to buy from if you want the setup to work the first time.

What the price gap buys

The cheapest 192 GB option in this guide is the Apple M3 Ultra at about $4000. This card costs $36000 more and reads memory 900% faster, which is roughly the gain in tokens per second. It buys no additional model: both load exactly the same weights, so the question is only whether you are waiting on the output.

A part at about $40000 is not bought the way a graphics card is bought. Most people meet this hardware by the hour from a cloud provider, and the figure above is useful mainly as the number to compare a rental bill against. If you are specifying one, the 1000 W draw and the cooling it implies decide the chassis before the accelerator does.

Where the NVIDIA B200 sits in this guide

Across the 20 cards catalogued here, the NVIDIA B200 ranks 1st on memory bandwidth at 8000 GB/s, 20th cheapest at about $40000, and 4th on bandwidth per watt at 8.0 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 192 GB parts.

HBM3e is the newest generation of stacked on-package memory: the DRAM dies sit on the same substrate as the processor rather than on the board, which is how a single part reaches 8000 GB/s. Bandwidth on this scale stops being the bottleneck for one conversation and starts mattering for how many conversations run at once.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA B200?

Of the 46 models in our catalogue with per-size memory figures, 45 have at least one size that fits in 192 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
DeepSeek Coder V2	236B	140 GB
Qwen 3	235B	130 GB
minimax M2	230B	130 GB
minimax M2.1	230B	130 GB
minimax M2.5	230B	130 GB
minimax M3	230B	130 GB
Qwen 3 VL	235B	130 GB
Mistral Large	123B	70 GB

A further 37 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: Llama 3.1 405B at 240 GB, Hermes 3 405B at 240 GB, Qwen 3-Coder 480B at 280 GB.

Expected tokens per second on NVIDIA B200

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA B200's 8000 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~1440 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~480 tok/s
Llama 3.1 8B	Q8_0	Fits in VRAM	~360 tok/s
Mistral Nemo 12B	Q4_K_M	Fits in VRAM	~320 tok/s
Qwen 2.5 14B	Q4_K_M	Fits in VRAM	~280 tok/s
Qwen 3 32B	Q4_K_M	Fits in VRAM	~144 tok/s
DeepSeek R1 distilled 32B	Q4_K_M	Fits in VRAM	~144 tok/s
Llama 3.1 70B	Q4_K_M	Fits in VRAM	~80 tok/s
Llama 4 Scout 109B (MoE)	Q4_K_M	Fits in VRAM	~64 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA B200

Best model to download first

Llama 3.1 8B to check the stack works, then the largest model in the table above that you actually need. At 192 GB the constraint stops being capacity. Start with ollama pull llama3.1:8b.

Recommended inference backend

At this capacity the point is serving several people at once, so use vLLM or TensorRT-LLM for continuous batching rather than a single-stream runner.

Sources

[1] nvidia.com/b200 · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).