SmolLM2

What is SmolLM2?

SmolLM2 is HuggingFace's open-weights language model. Released in 2024-11, it ships in 3 sizes (135M, 360M, 1.7B) and is licensed Apache 2.0. The most popular use case is small chat and tool use on any hardware.

VRAM and hardware

The smallest 135M size needs at least CPU VRAM; the largest 1.7B needs around CPU.

Size	Min VRAM (Q4_K_M)	Recommended VRAM	Tokens/sec on 3090
135M	CPU	CPU	~40 tok/s
360M	CPU	CPU	~40 tok/s
1.7B	1 GB	2 GB	~34 tok/s

How to run SmolLM2 locally

Option 1: Ollama (simplest)

ollama pull smollm2
ollama run smollm2

Option 2: Mullama (production)

mullama pull smollm2
mullama run smollm2

Option 3: llama.cpp (CLI)

# Download a GGUF from Hugging Face (search "smollm2 gguf")
./llama-cli -m smollm2.Q4_K_M.gguf -p "Hello, AI!"

Option 4: Python with Mullama or llama-cpp-python

from mullama import Model, Context
model = Model.load("smollm2.Q4_K_M.gguf", n_gpu_layers=99)
ctx = Context(model, n_ctx=4096)
print(ctx.generate("Hello, AI!", 256))

What you can build with SmolLM2

• chat
• edge
• IoT
• Raspberry Pi

Hardware it fits on

• Apple Silicon (8GB): 135M in MLX
• Apple Silicon (16GB): 360M in MLX
• Apple Silicon (32GB): 1.7B in MLX or GGUF
• Apple Silicon (64GB+): 1.7B in MLX
• RTX 3090 (24GB): 1.7B at Q4_K_M
• RTX 4090 (24GB): same as 3090, ~30% faster
• RTX 5090 (32GB): 1.7B at Q8_0
• 2× RTX 6000 Ada (96GB total): 1.7B at Q4_K_M

Related models in the SmolLM family

This is the only SmolLM-family model in the catalog right now.

Other models you might consider

• Llama 3.1 (Meta) — 8B, general-purpose chat, agentic workflows, code
• Llama 3.2 (Meta) — 1B, low-end hardware, mobile, edge deployment
• Llama 3.2 Vision (Meta) — 11B, vision + chat on desktop GPUs
• Llama 3.3 (Meta) — 70B, best-in-class 70B for desktops
• Llama 4 (Meta) — Scout 17B (109B MoE), vision + long context (10M tokens)
• Qwen 2.5 (Alibaba) — 0.5B, strongest 7B-32B on consumer hardware