KoboldCpp

KoboldCpp is a single-binary LLM inference server built on llama.cpp with a focus on CPU performance and zero-friction deployment. There is no Python, no venv, no pip install — you download one executable, drop a GGUF next to it, and run. That alone makes it the default on machines without a CUDA GPU: office laptops, old desktops, ARM boards, and Android phones. It also ships a built-in KoboldAI-style story UI, an OpenAI-compatible /v1/chat/completions endpoint, and a KoboldHorde integration that lets your instance contribute compute to a distributed network. SillyTavern, KoboldAI Lite, and many roleplay frontends drive it directly, which is why it dominates the interactive-fiction corner of the local-LLM world.

Key Features

• One file, no dependencies. The release is a single binary with llama.cpp, a CPU BLAS library, and a small UI baked in. Run it on a fresh Windows box or a Termux session on Android without installing a toolchain.
• Best-in-class CPU throughput. KoboldCpp’s maintainers track and tune the fastest llama.cpp backends (CLBlast, OpenBLAS, Apple Accelerate) and ship pre-built variants per CPU feature set. On a mid-range CPU it consistently outperforms a generic llama.cpp build by 20-40%.
• Multiple API surfaces. Exposes KoboldAI, OpenAI-compatible, and text-generation-webui APIs simultaneously, so almost any frontend works without an adapter. SillyTavern connects on port 5001 out of the box.
• Built-in story UI and KoboldHorde. A browser text-adventure interface is included for users who want to skip a separate frontend, and the Horde toggle lets you share or borrow compute across the public network.

When to Use KoboldCpp

Pick KoboldCpp when your target hardware has no usable GPU (laptops, older PCs, ARM SBCs, phones), when you want one binary you can ship to a non-technical user, or when you are building a SillyTavern / roleplay setup and want the frontend KoboldCpp was designed to pair with. For GPU-first serving on a dedicated box, Ollama or Mullama are usually simpler. For raw CPU benchmark parity, llama.cpp itself is equivalent — KoboldCpp’s value is the packaging and the story-UI ecosystem, not the kernel.

Setup

# Download the right binary for your platform from the GitHub Releases page:
#   Windows: koboldcpp.exe          (or koboldcpp_noavx2.exe for old CPUs)
#   macOS:   koboldcpp-mac          (Accelerate build)
#   Linux:   koboldcpp-linux        (or -rocm / -cuda variants)
#   Android: install the Termux APK, then run the linux-arm binary

# Run with a GGUF:
./koboldcpp --model ./models/llama-3-8b-instruct.Q4_K_M.gguf \
            --port 5001 --smartcontext --usecublas 0  # 0 = auto-detect GPU

Open http://localhost:5001 for the built-in UI. For SillyTavern, add a “Custom (KoboldCpp)” endpoint pointing at the same port. Common flags: --threads N, --gpulayers N (CUDA/ROCm), --contextsize 8192.

See also

• Ollama — the GPU-first alternative with model management
• SillyTavern — the frontend most often paired with KoboldCpp
• All inference engines — full catalog