Local AI Glossary
What is Inference engine?
Source: local-llm.net/glossary
Software that loads a model into memory and executes inference. Examples include llama.cpp, Ollama, vLLM, ExLlamaV2, MLX, and TGI. The inference engine handles tokenization, attention computation, KV-cache management, and token sampling.
How Inference engine fits in the local AI stack
Inference engine is part of the broader local AI ecosystem. See the related guides, comparisons, and tools below to see how it applies in practice.
See also
- Full Local AI Glossary
- All learn topics
- What is Local AI?
- Quantization explained
- Hardware requirements
- How to choose a local LLM