Local AI Glossary
What is Token?
Source: local-llm.net/glossary
The fundamental unit of text processing for LLMs. Tokenizers split text into tokens, which may be whole words, subwords, or individual characters. A typical English word is 1-2 tokens. The model operates on token IDs (integers), not raw text. Tokenizers vary between model families (Llama uses a different tokenizer than Mistral).
How Token fits in the local AI stack
Token is part of the broader local AI ecosystem. See the related guides, comparisons, and tools below to see how it applies in practice.
See also
- Full Local AI Glossary
- All learn topics
- What is Local AI?
- Quantization explained
- Hardware requirements
- How to choose a local LLM