Local LLM on NVIDIA Jetson Orin Nano 8GB

Is the NVIDIA Jetson Orin Nano 8GB the right card for this?

A 15 W handheld running a language model is a demonstration more than a workstation, and it is a real one: 8 GB of Unified LPDDR5 is shared with the operating system and the game you were playing, and 68 GB/s is the constraint on every token. Expect small models, short contexts and warm hands. At about $199 it is not bought for inference, but it can do it. Catalogue throughput class: Q4 3B: 8-12, Q4 7B: 3-5.

A 2023 part sits in the middle of this guide's range: old enough that backend support has stopped moving and the community has written up the quirks, new enough that nothing treats it as legacy. That is the least risky place on the calendar to buy from if you want the setup to work the first time.

What the price gap buys

No other card in this guide carries 8 GB, so there is no like-for-like price comparison to make. The nearest capacities are listed at the foot of this page, and moving to any of them changes which models fit, not just how fast they run.

At about $199 this is an impulse purchase next to every other entry in this guide, and it is sold as a complete device rather than as a component. That changes the question from "will it fit in my machine" to "is what it can run worth having", which the model table below answers directly.

Where the NVIDIA Jetson Orin Nano 8GB sits in this guide

Across the 20 cards catalogued here, the NVIDIA Jetson Orin Nano 8GB ranks 19th on memory bandwidth at 68 GB/s, 2nd cheapest at about $199, and 9th on bandwidth per watt at 4.5 GB/s per watt. Those three positions, not the capacity figure, are what separate it from other 8 GB parts.

Unified LPDDR5 is shared between the CPU and the GPU on this part, which is what makes 8 GB usable by a model at all on a machine this size. The trade is 68 GB/s: enough to hold very large models, not enough to run them at the speed a discrete card manages on smaller ones.

This is a CUDA card, which in practice means every local runner in our tool directory supports it first. Ollama and llama.cpp pick it up without configuration; vLLM and TensorRT-LLM target it specifically. Driver and CUDA toolkit versions are the usual source of trouble, not the card.

Which catalogue models fit on NVIDIA Jetson Orin Nano 8GB?

Of the 46 models in our catalogue with per-size memory figures, 26 have at least one size that fits in 8 GB at Q4_K_M with room for context. The largest of each are below, biggest first.

Model	Largest size that fits	Weights at Q4_K_M
Llama 3.2 Vision	11B	7 GB
Falcon 3	10B	7 GB
Gemma 2	9B	6 GB
LLaVA	7B	6 GB
Granite 3.3	8B	6 GB
Hermes 3	8B	6 GB
Qwen 3 VL	8B	6 GB
Cogito	8B	6 GB

A further 18 smaller models also fit; see the model catalogue.

The next sizes up are out of reach without offloading: Mistral Nemo 12B at 8 GB, Gemma 3 12B at 8 GB, Gemma 4 12b at 8 GB.

Expected tokens per second on NVIDIA Jetson Orin Nano 8GB

Memory bandwidth, not compute, is the bottleneck for single-stream token generation. The NVIDIA Jetson Orin Nano 8GB's 68 GB/s puts the following ceiling on a batch of one at 2048 tokens of context.

Model	Quantization	Status	Approx tokens/sec
Llama 3.2 1B	Q4_K_M	Fits in VRAM	~12 tok/s
Llama 3.1 8B	Q4_K_M	Fits in VRAM	~4 tok/s
Llama 3.1 8B	Q8_0	Needs CPU offload	~1 tok/s

These come from a bandwidth heuristic (tokens/sec ≈ bandwidth in GB/s × a per-model efficiency factor), not from a test rig in our office. Real numbers move with model architecture, batch size and KV-cache size. Treat them as an order of magnitude, and treat the "reported throughput class" row in the table above as the community-measured figure.

Build recommendations for NVIDIA Jetson Orin Nano 8GB

Best model to download first

Llama 3.1 8B at Q4_K_M. Start with ollama pull llama3.1:8b.

Recommended inference backend

Use Ollama for general use or Mullama if you need a drop-in Ollama alternative with native bindings for 6 languages. For production serving on multi-GPU setups, see vLLM.

Sources

[1] nvidia.com/jetson-orin · VRAM figures for catalogue models from src/data/models.json (2026-06-29); card specifications from src/data/gpus.json (2026-06-29).