Inference Engine Built by Cognisoc pre-alpha MIT

UniLLM

A modular LLM inference runtime written in Rust. 47 architecture families, format-agnostic weight loading (SafeTensors, GGUF, PyTorch), and a clean three-layer abstraction (TensorCore, ModelCore, WeightLoaderCore).

GitHub
Platforms: windowsmacoslinux

Status: Pre-alpha. APIs may change. Suitable for research, prototyping, and teams evaluating Rust LLM runtimes for production work. Track on GitHub: cognisoc/unillm — file issues, watch releases.

What is UniLLM

UniLLM is a modular, type-safe Rust runtime for local language model inference. It provides a unified interface for running LLMs across 47 architecture families and three weight formats, organized around three composable abstractions:

  1. TensorCore — device-agnostic tensor operations. CPU, CUDA, Metal. All ops go through ops_fn::operation().
  2. ModelCore — universal Model trait with forward() and generate(). Configuration via the model_config! macro.
  3. WeightLoaderCore — format-agnostic weight loading for SafeTensors, GGUF, and PyTorch files.

The goal is to make it cheap to add a new model architecture or a new weight format without rewriting the rest of the runtime.

Supported architectures (47)

UniLLM covers most of the architectures a Rust LLM team will encounter in 2026:

  • Core LLMs — LLaMA, Qwen, Gemma, Phi, DeepSeek, Mistral, Mixtral
  • GPT family — GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT
  • Code — StarCoder, CodeLlama
  • MoE — DeepSeek-MoE, DBRX, Grok, Arctic, Jamba
  • RWKV / linear attention — RWKV-4, RWKV-6, RecurrentGemma
  • Vision-language — Qwen2-VL, Phi-3-Vision, InternVL, CogVLM, Idefics, Florence, LLaVA, CLIP
  • Audio / speech — Wav2Vec2, HuBERT, MusicGen, Encodec, Whisper
  • Encoder — BERT, T5
  • Specialized — Mamba, MiniCPM, OLMo, Granite
  • Additional — Yi, Falcon, Baichuan, InternLM, ChatGLM

All models share the same Model trait and are configured through the model_config! macro.

Key Features

Three-layer modular design. TensorCore, ModelCore, and WeightLoaderCore are independent abstractions. Add a new architecture by writing a model_config! block; add a new weight format by writing a new WeightLoader; add a new device by writing a new ops_fn::operation(). None of the three layers needs to change to extend the others.

Format-agnostic weights. Load from SafeTensors (HuggingFace default), GGUF (llama.cpp / Ollama default), or PyTorch checkpoints (.bin / .pt). Convert between them at load time without external scripts.

Type-safe configuration. The model_config! macro generates compile-time-checked config structs, so a missing field or wrong type is a compile error rather than a runtime crash.

Device-agnostic. The same model definition runs on CPU, CUDA, or Metal. Backends are swappable without code changes.

Research-friendly. Adding a new architecture is well-documented and the abstraction layers make it clear what to write. Several research groups have used UniLLM to implement newly-published architectures within days of paper release.

When to Use UniLLM

UniLLM is the right tool when you are:

  • Building Rust infrastructure for LLM inference and want a runtime that doesn’t lock you into one weight format or one device.
  • Researching new architectures or training methods and need a clean place to add and benchmark them.
  • Migrating from candle, burn, or tch-rs and want a more modular abstraction.
  • Working across multiple weight formats in a single product (e.g., serving GGUF for end users and SafeTensors for research).

UniLLM is not the right tool when:

  • You want a polished production daemon right now. Use Ollama, vLLM, or TGI.
  • You need every architecture with maximum optimization. llama.cpp and vLLM are ahead here for the specific architectures they target.
  • You want a CLI. UniLLM is library-first.

How UniLLM compares

UniLLMllama.cppcandleburn
Architecture count47~30~20~10
Weight formats3 (SafeTensors, GGUF, PyTorch)1 (GGUF)1 (SafeTensors)1 (SafeTensors)
Modularity3 layers1 (monolithic)2 (ops + models)3 (tensor + autodiff + module)
GPU supportCPU/CUDA/MetalCPU/CUDA/Metal/Vulkan/…CPU/CUDA/Metal/WGPUCPU/CUDA/Metal/WGPU
Type safetycompile-time model configruntimeruntimecompile-time
Native bindingsplanned (C FFI)✓ (C/C++)✓ (C/Python)✓ (C/Python)

Quick start

git clone https://github.com/cognisoc/unillm.git
cd unillm
cargo test --workspace

# Run inference (downloads TinyLlama on first run, ~600MB)
cargo run --bin unillm -p unillm-runtime -- generate --prompt "Explain gravity"

# Use a different model
cargo run --bin unillm -p unillm-runtime -- generate --model llama2:7b --prompt "Hello"

# List cached models
cargo run --bin unillm -p unillm-runtime -- models

Adding a model

model_config!(MyModelConfig {
    vocab_size: usize = 32000,
    hidden_size: usize = 4096,
    num_layers: usize = 32,
    num_heads: usize = 32,
    // ...
});

impl Model for MyModel {
    type Config = MyModelConfig;
    fn forward(&self, input: &Tensor) -> Result<Tensor> { /* ... */ }
}

See also

  • Mullama — production-ready Rust LLM runtime with 6 native bindings; UniLLM is more research-oriented
  • llama.cpp — the upstream C++ engine UniLLM is designed to complement, not replace
  • vLLM — Python production serving stack for the architectures UniLLM can also serve

Comparisons