Local AI Glossary
What is Continuous batching?
Source: local-llm.net/glossary
An inference optimization used by engines like vLLM where new requests are added to the processing batch as soon as existing requests complete, rather than waiting for the entire batch to finish. This maximizes GPU utilization and throughput.
How Continuous batching fits in the local AI stack
Continuous batching is part of the broader local AI ecosystem. See the related guides, comparisons, and tools below to see how it applies in practice.
See also
- Full Local AI Glossary
- All learn topics
- What is Local AI?
- Quantization explained
- Hardware requirements
- How to choose a local LLM