Local AI Glossary
What is KV cache quantization?
Source: paperswithcode/kv-cache
Quantizing the key-value cache (typically to INT8 or INT4) to reduce the memory footprint of long-context inference. Can save 50%+ memory with minimal quality loss. Used in vLLM, TGI, and llama.cpp.
How KV cache quantization fits in the local AI stack
KV cache quantization is part of the broader local AI ecosystem. See the related guides, comparisons, and tools below to see how it applies in practice.
See also
- Full Local AI Glossary
- All learn topics
- What is Local AI?
- Quantization explained
- Hardware requirements
- How to choose a local LLM