EXL2 vs GGUF vs GPTQ vs AWQ vs BnB in 2026

Quantization is what makes a 70B model fit in 24GB of VRAM. In 2026 there are five major formats to know, each with different tradeoffs. This page helps you pick.

Format comparison

	EXL2	GGUF	GPTQ	AWQ	BnB
Year	2023	2023	2022	2023	2022
Owner	turboderp (ExLlamaV2)	ggerganov (llama.cpp)	Frantar et al.	MIT (Lin et al.)	bitsandbytes team
Primary use	NVIDIA inference	universal	NVIDIA inference	NVIDIA inference	training-time quant
Quantization levels	2.0 – 8.0 bpw	Q2_K – Q8_0 + i-quants	2/3/4/8-bit	2/3/4/16-bit	4/8-bit (NF4)
Calibration	required	not required	required	required	not required
Perplexity loss at 4-bit	<2%	<2%	2-3%	<2%	3-5% (training)
Hardware	NVIDIA RTX	CPU, NVIDIA, AMD, Apple	NVIDIA	NVIDIA	NVIDIA (training)
Tools	ExLlamaV2	llama.cpp, Ollama, Mullama	AutoGPTQ	AutoAWQ	transformers + bitsandbytes
Conversions	from GPTQ/AWQ/HF	universal target	from HF	from HF	from HF

Quality vs size (Llama 3.1 8B)

Format	Bits per weight	File size	MMLU	Tokens/sec on 3090
FP16 (reference)	16.0	16 GB	69.0	28
Q8_0 GGUF	8.5	8.5 GB	68.7	35
Q5_K_M GGUF	5.7	5.7 GB	68.4	45
Q4_K_M GGUF	4.9	4.9 GB	67.9	52
AWQ 4-bit	4.3	4.3 GB	67.8	58
EXL2 4.0 bpw	4.0	4.5 GB	67.7	62
GPTQ 4-bit	4.3	4.3 GB	67.3	50
Q3_K_M GGUF	3.9	3.9 GB	66.0	60
Q2_K GGUF	2.7	2.7 GB	60.5	65

EXL2 has the best tokens/sec on NVIDIA because it is optimized for that exact hardware. GGUF is the most flexible. AWQ has the best quality-per-byte for very small quants. GPTQ is the legacy format, superseded by AWQ.

When to use each

Choose EXL2 when…

• You are on NVIDIA consumer GPUs (RTX 3090, 4090, 5090) and want the fastest inference.
• You have the original FP16 model and can run a calibration pass to produce the EXL2 file.
• You use ExLlamaV2 as your inference engine (Tabby, lmdeploy, and some others support it too).

Choose GGUF when…

• You want the widest hardware support (CPU, NVIDIA, AMD, Apple Silicon, mobile).
• You use Ollama, Mullama, LM Studio, or llama.cpp as your daily driver.
• You want k-quants (Q4_K_M, Q5_K_M, etc.) which are the best quality-per-byte at most sizes.
• You need i-quants (IQ1, IQ2, IQ3) for very low VRAM.

Choose GPTQ when…

• You have an older model that only exists in GPTQ (some Hugging Face uploads are GPTQ-only).
• You need compatibility with vLLM, TGI, or TensorRT-LLM which all read GPTQ.
• Otherwise, prefer AWQ (newer, slightly better quality at the same size).

Choose AWQ when…

• You are deploying on NVIDIA GPUs for production serving.
• You need activation-aware quantization for the best quality at 3-4 bit.
• You use vLLM, TGI, or TensorRT-LLM (all three have native AWQ support).

Choose BnB (bitsandbytes) when…

• You are doing QLoRA fine-tuning (4-bit base + LoRA adapters).
• You do not need a separate quantized file — BnB quantizes on-the-fly during training.
• For inference, prefer one of the other four.

Conversion paths

From	To	Tool
HF safetensors	GGUF	llama.cpp/convert_hf_to_gguf.py
HF safetensors	GPTQ	AutoGPTQ
HF safetensors	AWQ	AutoAWQ
HF safetensors	EXL2	ExLlamaV2 convert.py
GPTQ	EXL2	ExLlamaV2 convert_gptq_to_exl2.py
AWQ	EXL2	ExLlamaV2 convert_awq_to_exl2.py
GGUF	(universal)	load with llama.cpp

Setup examples

GGUF (Ollama)

# Automatic — Ollama picks the right GGUF quant
ollama run llama3.1:8b
# Uses Q4_K_M by default

EXL2 (Tabby)

# Convert a HF model to EXL2
python convert.py -i ./Llama-3.1-8B-Instruct -o ./llama-3.1-8b-exl2 -c ./calibration-data

# Serve with Tabby (or ExLlamaV2 directly)
tabby serve --model ./llama-3.1-8b-exl2

AWQ (vLLM)

# AutoAWQ produces a quantized model
autoawq-cli quantize --model meta-llama/Llama-3.1-8B-Instruct \
  --output-dir ./llama-3.1-8b-awq --bits 4

# Serve with vLLM
vllm serve ./llama-3.1-8b-awq --quantization awq

Decision matrix

Your hardware	Best format
NVIDIA RTX 3090/4090/5090	EXL2 (fastest) or AWQ (best quality)
NVIDIA A100/H100 (production)	AWQ (served via vLLM)
AMD GPU	GGUF (only choice)
Apple Silicon	GGUF (universal) or MLX (native)
CPU only	GGUF (only choice)
Mobile (Android/iOS)	GGUF (Q4_K_M) or MLX (iOS)
Raspberry Pi / Jetson	GGUF (Q4_0 or Q5_0)
24GB GPU, want max quality	AWQ 4-bit
12GB GPU, want max speed	EXL2 4-bit
8GB GPU, max model	Q3_K_M GGUF or Q2_K GGUF
6GB GPU, smallest model	Q2_K GGUF or Q4_0 GGUF
Fine-tune a quantized model	BnB (QLoRA)

See also

• Quantization explained — what quantization actually does
• Hardware requirements — what model fits your hardware
• GGUF vs GPTQ vs AWQ vs EXL2 (2026 edition) — pair-format comparison
• Local LLM on RTX 3090 — concrete build with quant recommendations