Local AI Glossary
131 terms covering model architecture, quantization, inference, training, RAG, and hardware — the definitive reference for anyone running AI locally.
A
Activation The output produced by a neuron or layer in a neural network after applying a non-linear function to its inputs. Activat… Adapter A small trainable module inserted into a pre-trained model to modify its behavior without changing the original weights.… AGI (Artificial General Intelligence) A hypothetical AI system capable of understanding and learning any intellectual task that a human can. Current LLMs, inc… Alignment The process of training a model to behave in accordance with human intentions, values, and instructions. Alignment techn… Attention mechanism The core innovation of the Transformer architecture. Attention allows each token in a sequence to "attend to" (consider … AWQ (Activation-Aware Weight Quantization) A quantization method that identifies and preserves the most important weights based on their impact on model activation… AWQ 4-bit A 4-bit variant of Activation-Aware Weight Quantization, optimized for fast inference on NVIDIA Ampere and newer GPUs. U…
B
Batch size The number of input sequences processed simultaneously during inference or training. Larger batch sizes increase through… Beam search A text generation strategy that explores multiple candidate sequences simultaneously, keeping the top-k most probable se… BF16 (BFloat16) A 16-bit floating-point format developed by Google Brain with the same exponent range as FP32 but fewer mantissa bits th… Bits per weight (bpw) A measure of quantization precision indicating the average number of bits used to store each model weight. FP16 uses 16 … BnB (bitsandbytes) A Python library for 4-bit and 8-bit quantization with a focus on training-time use. Used in QLoRA for fine-tuning on co… BM25 A traditional keyword-based ranking function used in search engines. The de facto baseline for keyword search. Often com…
C
Calibration data A representative dataset used during quantization (in GPTQ, AWQ, and EXL2) to measure the impact of weight compression o… Chat template A formatting specification that defines how user messages, assistant responses, and system prompts are structured for a … Checkpoint A saved snapshot of a model's weights, optimizer state, and training progress at a particular point during training. Che… Chunking The process of splitting documents into smaller segments (chunks) for use in RAG pipelines. Effective chunking strategie… Context length (context window) The maximum number of tokens a model can consider at once, including both the input prompt and the generated output. Com… Continuous batching An inference optimization used by engines like vLLM where new requests are added to the processing batch as soon as exis… CUDA NVIDIA's parallel computing platform and programming model for GPU-accelerated computing. CUDA is the foundation of most… Chunk overlap The number of tokens shared between adjacent chunks in a RAG pipeline. Overlap prevents the splitter from cutting releva… Chain-of-thought (CoT) A prompting technique where the model is asked to think step-by-step before giving a final answer. Improves performance … Code completion A specialized use of LLMs where the model predicts the next tokens in a source code file given the preceding context. Lo…
D
Dense model A model architecture where every parameter is used for every input token, as opposed to mixture-of-experts (MoE) models … DPO (Direct Preference Optimization) An alignment training technique that directly optimizes the model to prefer human-preferred outputs over rejected output… DeepSeek architecture DeepSeek's Transformer variant. DeepSeek-V2 and V3 use Multi-head Latent Attention (MLA); DeepSeek-R1 is a reasoning mod…
E
Embedding A dense numerical vector representation of text (or images, audio, etc.) in a high-dimensional space where semantically … Embedding model A model specifically designed to produce embedding vectors from input text, as opposed to generative language models tha… Epoch One complete pass through the entire training dataset during model training or fine-tuning. Fine-tuning typically uses 1… EXL2 A quantization format used by ExLlamaV2 that supports variable bitrate across model layers. EXL2 assigns more bits to im… ExLlamaV2 A high-performance CUDA-only inference engine for running quantized LLMs on NVIDIA GPUs. Known for achieving the fastest…
F
FFN (Feed-Forward Network) A component of each Transformer layer consisting of two linear transformations with a non-linear activation function in … Fine-tuning The process of continuing to train a pre-trained model on a specific dataset to adapt it for a particular task, domain, … FP16 (Float16) A 16-bit floating-point number format that can represent values with approximately 3.3 decimal digits of precision. The … FP32 (Float32) A 32-bit floating-point format with approximately 7.2 decimal digits of precision. Used internally during some computati… FlashAttention An IO-aware exact attention algorithm that reduces memory accesses by tiling. 2-4× faster than standard attention with l… Function calling The ability of an LLM to output structured JSON that invokes external functions. Used to connect LLMs to tools, APIs, an… Fill-in-the-middle (FIM) A code completion mode where the model sees code both before and after a cursor and predicts the missing middle. Used by…
G
GGML The predecessor to GGUF, created by Georgi Gerganov for the llama.cpp project. GGML was the original binary format for q… GGUF (GPT-Generated Unified Format) The standard quantization format for llama.cpp and the broader local AI ecosystem. GGUF files are self-contained (includ… GPTQ (Generalized Post-Training Quantization) A GPU-optimized quantization method that uses calibration data and second-order error correction to minimize quality los… Greedy decoding A text generation strategy that always selects the most probable next token at each step. Fast and deterministic but oft… Group size In GPTQ and AWQ quantization, the number of weights that share a single set of scaling parameters. A group size of 128 m… GGUF Q2_K A 2-bit GGUF k-quant variant. Very small model files (around 2.7 bits per weight on average) but with noticeable quality… GGUF Q3_K_M A 3-bit GGUF k-quant, medium tier. Around 3.9 bits per weight. Good for fitting large models on small GPUs; expect 3-5% … GGUF Q4_K_S A 4-bit GGUF k-quant, small tier. Around 4.1 bits per weight. Slightly smaller than Q4_K_M with marginally lower quality… GGUF Q4_K_M A 4-bit GGUF k-quant, medium tier. Around 4.8 bits per weight. The modern default; the best quality-per-byte at 4-bit. R… GGUF Q5_K_M A 5-bit GGUF k-quant, medium tier. Around 5.7 bits per weight. A quality step up from Q4_K_M with ~25% larger file. Reco… GGUF Q6_K A 6-bit GGUF k-quant. Around 6.6 bits per weight. Near-lossless quality; recommended when maximum quality matters and VR… GGUF Q8_0 An 8-bit GGUF quantization. Around 8.5 bits per weight. Essentially lossless; useful as a reference for the unquantized … Gemma architecture Google's Transformer variant with strong safety RLHF. Gemma, Gemma 2, and Gemma 3 use GQA and RoPE; Gemma 3 added vision…
H
Hallucination When a model generates information that is factually incorrect, nonsensical, or fabricated while presenting it as if it … Hugging Face The central platform for sharing open-weight AI models, datasets, and tools. Most models available for local AI are host… Hybrid search A retrieval strategy that combines vector similarity (semantic) search with traditional keyword (BM25) search. Captures … HNSW (Hierarchical Navigable Small World) A graph-based vector index that enables sub-millisecond approximate nearest neighbor search at scale. The default index …
I
Importance matrix (imatrix) A data structure that records which model weights have the greatest impact on output quality, generated by running calib… Inference The process of running a trained model to generate predictions or outputs from new inputs. In the context of LLMs, infer… Inference engine Software that loads a model into memory and executes inference. Examples include llama.cpp, Ollama, vLLM, ExLlamaV2, MLX… Instruct model A model variant that has been fine-tuned on instruction-following datasets (conversations where a user asks a question a… INT4 / INT8 Integer formats using 4 or 8 bits respectively. Used in quantization to represent model weights with lower precision tha… i-quant (IQ) Importance-aware GGUF quantizations that use an imatrix (importance matrix) to allocate bits. IQ1, IQ2, IQ3, IQ4 variant…
K
K-quants The quantization system used in GGUF format that assigns different bit-widths to different parts of the model based on t… KV-cache (Key-Value cache) A memory structure that stores the key and value tensors computed during attention for all previously processed tokens. … KV cache quantization Quantizing the key-value cache (typically to INT8 or INT4) to reduce the memory footprint of long-context inference. Can… KTO (Kahneman-Tversky Optimization) An alignment technique that uses a human-utility model (not preference pairs) to align the model. Works with single-resp…
L
Learning rate A hyperparameter that controls how much the model's weights are adjusted during each training step. Too high a learning … llama.cpp The foundational open-source inference engine for local AI, written in C/C++ by Georgi Gerganov. Supports GGUF models on… LLM (Large Language Model) A neural network with billions of parameters trained on vast text corpora to generate, understand, and manipulate human … LoRA (Low-Rank Adaptation) A parameter-efficient fine-tuning method that freezes the original model weights and trains small, low-rank adapter matr… LLaMA architecture Meta's open Transformer architecture introduced in the LLaMA paper (2023). The basis for Llama 2, Llama 3, and most curr…
M
Memory bandwidth The rate at which data can be read from or written to memory, measured in GB/s. Memory bandwidth is the primary bottlene… Metal Apple's GPU programming framework, analogous to NVIDIA's CUDA. Metal provides GPU acceleration for AI inference on Apple… MLX Apple's machine learning framework, designed specifically for Apple Silicon. MLX provides a NumPy-like API with automati… MoE (Mixture of Experts) A model architecture where the network contains multiple "expert" sub-networks, and a routing mechanism selects a subset… Model merging The practice of combining weights from multiple fine-tuned models to create a new model that blends their capabilities. … Mistral architecture Mistral AI's Transformer variant. Uses sliding window attention, RoPE, and SwiGLU. Used in Mistral 7B, Mixtral 8x7B, and… Multi-head Latent Attention (MLA) DeepSeek-V2's attention variant that compresses the KV cache into a low-dimensional latent space, reducing memory usage … MCP (Model Context Protocol) Anthropic's open protocol for connecting LLMs to data sources and tools. Standardizes how AI assistants call functions, … Multimodal model A model that can process and generate multiple modalities (text, images, audio, video). Examples: Llama 3.2 Vision, Qwen…
N
O
Ollama A user-friendly inference platform built on llama.cpp that provides a simple CLI, a model registry, and an OpenAI-compat… ONNX (Open Neural Network Exchange) An open format for representing machine learning models, enabling interoperability between different frameworks. ONNX mo… Open-weight model A model whose trained weights are publicly available for download, inspection, and use, as opposed to closed/proprietary… ORPO (Odds Ratio Preference Optimization) A single-stage alignment technique that combines SFT and preference optimization without a reference model. More memory-…
P
PagedAttention A memory management technique used by vLLM that stores the KV-cache in non-contiguous memory blocks (pages), similar to … Parameter A single learned numerical weight in a neural network. A "7B model" has approximately 7 billion parameters. Each paramet… Perplexity A metric that measures how well a language model predicts a test dataset. Lower perplexity means better prediction (the … Pipeline parallelism A model parallelism strategy where different layers of the model are assigned to different GPUs, creating a pipeline. GP… Prefill The first phase of LLM inference where the model processes the entire input prompt in parallel, computing the initial KV… Prompt The text input provided to a language model that it uses as context for generating a response. A prompt can include a sy… Phi architecture Microsoft's research architecture emphasizing data quality over parameter count. Phi-3, Phi-4, and successors use groupe… PPO (Proximal Policy Optimization) The reinforcement learning algorithm used in RLHF. Trains the language model to maximize the reward model's output while…
Q
QLoRA (Quantized LoRA) A fine-tuning method that combines LoRA with a quantized (typically NF4) base model. QLoRA allows fine-tuning models on … Quantization The process of reducing the numerical precision of model weights to decrease memory usage and increase inference speed. … Qwen architecture Alibaba's Transformer variant with strong multilingual support. Uses RoPE, SwiGLU, and tied embeddings. The Qwen, Qwen 2… QLoRA A fine-tuning method that combines LoRA adapters with a 4-bit NF4 quantized base model. Allows fine-tuning a 65B model o…
R
RAG (Retrieval-Augmented Generation) A technique that enhances LLM responses by first retrieving relevant documents from a knowledge base and then providing … Repetition penalty A sampling parameter that reduces the probability of generating tokens that have already appeared in the output, prevent… RLHF (Reinforcement Learning from Human Feedback) An alignment technique where a separate reward model (trained on human preference data) provides feedback signals that g… ROCm (Radeon Open Compute) AMD's open-source platform for GPU computing, analogous to NVIDIA's CUDA. ROCm enables AI inference on AMD GPUs. Support… RoPE (Rotary Position Embedding) A position encoding method used in modern Transformer models (including Llama, Mistral, and Qwen) that encodes token pos… Re-ranking A second-stage retrieval step that re-orders the initial retrieval results using a more expensive (but more accurate) mo… Reasoning model A language model trained or fine-tuned to perform explicit multi-step reasoning before producing an answer. Examples: Op…
S
SafeTensors A file format for storing model weights safely and efficiently, developed by Hugging Face. SafeTensors prevents arbitrar… Sampling The process of selecting the next token from the model's predicted probability distribution during text generation. Samp… Semantic search Finding documents based on meaning rather than keyword matching. Semantic search uses embedding models to convert text i… SFT (Supervised Fine-Tuning) Training a model on labeled examples (input-output pairs) to learn specific behaviors. SFT is typically the first step o… SLM (Small Language Model) A language model with a relatively small parameter count (typically 1B-7B), designed to run efficiently on resource-cons… Speculative decoding An inference optimization where a smaller, faster "draft" model generates candidate tokens that are then verified by the… Stop token A special token that signals the model to stop generating text. Different model families use different stop tokens (e.g.… System prompt An initial instruction provided to the model that sets its behavior, personality, and constraints for the conversation. …
T
Temperature A sampling parameter that controls the randomness of text generation. Temperature 0 (or near 0) produces deterministic, … Tensor parallelism A model parallelism strategy where individual layers are split across multiple GPUs, with each GPU computing a portion o… TGI (Text Generation Inference) Hugging Face's production-grade inference server for LLMs. TGI supports SafeTensors, GPTQ, and AWQ models with features … Token The fundamental unit of text processing for LLMs. Tokenizers split text into tokens, which may be whole words, subwords,… Tokens per second (tok/s) The standard measure of LLM inference speed, indicating how many tokens the model generates per second. Human reading sp… Top-k sampling A text generation strategy that restricts the candidate set to the k most probable next tokens at each step, then sample… Top-p (nucleus) sampling A text generation strategy that selects from the smallest set of tokens whose cumulative probability exceeds the thresho… Transformer The neural network architecture underlying virtually all modern LLMs, introduced in the 2017 paper "Attention Is All You… Tool use A broader term for function calling. The model can select from a list of available tools, generate arguments, and observ…
V
Vector database A specialized database designed to store, index, and search high-dimensional vectors (embeddings) efficiently. Used in R… vLLM A high-performance inference engine optimized for GPU throughput, featuring PagedAttention for efficient memory manageme… VRAM (Video RAM) Dedicated high-bandwidth memory on a GPU, used to store model weights, KV-cache, and intermediate computations during in… Vulkan A cross-platform graphics and compute API that provides GPU acceleration on a wide range of hardware, including NVIDIA, … Vector index A data structure for fast approximate nearest neighbor (ANN) search in high-dimensional spaces. Common vector index type… Vision-Language Model (VLM) A multimodal model that takes both images and text as input and produces text output. Local examples: LLaVA, Moondream, …
W
For the full long-form glossary page, see /learn/glossary/.