Local AI Glossary
What is Batch size?
Source: local-llm.net/glossary
The number of input sequences processed simultaneously during inference or training. Larger batch sizes increase throughput (total tokens per second across all requests) but require more memory. In local AI serving, batch size determines how many concurrent users the system can handle efficiently.
How Batch size fits in the local AI stack
Batch size is part of the broader local AI ecosystem. See the related guides, comparisons, and tools below to see how it applies in practice.
See also
- Full Local AI Glossary
- All learn topics
- What is Local AI?
- Quantization explained
- Hardware requirements
- How to choose a local LLM