llmdot

Status: Pre-alpha. APIs may change. Best for early adopters and contributors comfortable with breaking changes between minor versions. Track on GitHub: cognisoc/llmdot — file issues, watch releases.

What is llmdot

llmdot is a .NET-native runtime for local language model inference built around the GGUF model format. It executes major decoder-only transformer and hybrid architectures in the 1–8B parameter range — including multimodal variants — through architecture-agnostic execution templates resolved from GGUF metadata at load time.

The project is designed around a single opinionated goal: make local LLM execution in .NET as simple as adding a NuGet package, loading a GGUF file, and streaming tokens. The default path is pure managed code with zero native runtime dependencies, focused on CPU-first execution. Optional packages provide GPU acceleration through thin backend adapters.

Key Features

Single NuGet package, opinionated defaults. Llmdot.Model.LoadAsync(path) is the entire API surface for most apps. No native binaries to ship, no per-platform configuration, no third-party download scripts. Add the package, copy a .gguf file, run.

Architecture-agnostic execution templates. When you load a GGUF file, llmdot reads the model architecture from the file’s metadata and dispatches to the right execution template (LLaMA, Qwen, Phi, Gemma, Mistral, and others). You don’t have to write a parser per architecture.

Pure managed code, CPU-first. The default execution path is C# with no native dependencies beyond what .NET ships. This makes it trimming-safe and NativeAOT-friendly — you can compile a single self-contained executable that includes llmdot and runs on any x64 / ARM64 machine.

AOT and trimming safe. llmdot is built with the same constraints as modern .NET libraries: no reflection on hot paths, no dynamic loading, no JIT-only constructs. Ship a 12MB single-file binary that runs offline.

GGUF-native. All major quantizations are supported (Q2_K through Q8_0, including the i-quants). Pick the size/quality tradeoff that fits your hardware.

Optional GPU acceleration. When you need it, a separate package provides CUDA, Metal, and Vulkan backends through thin adapters. The default path stays CPU-first; GPU is opt-in.

When to Use llmdot

llmdot is the right tool when you are:

• Building a .NET application that needs on-device LLM inference — desktop apps (WPF, WinUI, Avalonia), services (ASP.NET, Worker Service), or libraries consumed by other .NET code.
• Shipping a single-file executable that runs LLM inference offline — no internet, no daemon, no Python sidecar.
• Working in a NativeAOT or trimmed deployment and need a runtime that does not break the build.
• Choosing between HTTP and embedded — llmdot lets you embed inference directly in-process (no HTTP, no daemon, no FFI) and also expose an OpenAI-compatible server when you want one.

llmdot is not the right tool when:

• You need to serve models larger than ~8B parameters at production throughput. Use vLLM, TensorRT-LLM, or TGI for that.
• You need Python. Use llama-cpp-python or Ollama.
• You need JavaScript, Go, PHP, or another non-.NET language. Use Mullama (which has native bindings for 6 languages).

How llmdot compares

	llmdot	llama-cpp Python	ONNX Runtime	Semantic Kernel + Ollama
GGUF models	✓	✓	partial	✗
Managed code	✓	✗	partial	partial
NativeAOT safe	✓	✗	partial	partial
Single-file deploy	✓	✗	partial	✗
OpenAI-compatible server	planned	partial	partial	partial
GPU acceleration	optional	optional	✓	depends
.NET-native API	✓	✗	✓	✓

Code sample

using Llmdot;

await using var model = await LlmModel.LoadAsync("phi-3-mini-q4_k_m.gguf");
await using var session = model.CreateChatSession();

await foreach (var token in session.StreamAsync("Explain GGUF in one paragraph."))
    Console.Write(token);

The code sample above reflects the target API shape. See roadmap for the current state of the implementation.

Ecosystem Role

llmdot fills a specific gap in the .NET ecosystem: an inference runtime that treats GGUF as the canonical format and is built for the deployment constraints modern .NET teams care about (AOT, trimming, single-file, cross-platform). Existing options in the .NET space either wrap the Python ecosystem (slow, fragile), rely on ONNX (limited model support), or use Semantic Kernel + Ollama (requires a separate process).

See also

• Mullama — language-agnostic runtime in Rust with 6 native bindings, including a C/C++ FFI binding
• llama.cpp — the upstream C++ inference engine that powers most other local LLM runtimes
• Ollama — the most popular HTTP-based local LLM daemon; llmdot is a good fit when you need in-process inference in .NET
• Llama-cpp-python vs llmdot (see Mullama vs llama-cpp-python for a similar comparison)