Evaluate · Hardware
Size hardware from the workload, not the headcount
Why staff count alone isn’t enough: fifty employees who each ask a few questions a day need far less than five analysts running long-document summaries all afternoon. What drives hardware is peak concurrent requests, the context length each one holds, the model size and quantization, and the response speed you need.
Rough concurrency: peak active users × requests per user per hour × seconds per response ÷ 3,600. For example, 40 users × 6 requests/hour × 20 s ÷ 3,600 ≈ 1.3, so plan for 2–4 concurrent sequences.
Estimated First-order estimates; benchmark before you buy.
Which hardware fits
Usable memory assumes ~90% of a discrete GPU’s VRAM and ~75% of unified memory (Apple Silicon, Jetson, APUs). Speed is per stream at batch size 1; serving several streams at once raises total throughput but each stream gets no faster.
| Hardware | Memory | Fits? | Est. tok/s per stream | Approx. hardware cost |
|---|
Assumptions
Architecture values come from each model’s published config.json. GPU bandwidth and prices come from our hardware data, and prices are approximate.
Takes your inputs and results to a short request form. Nothing is sent until you submit it.
Frequently Asked Questions
How many users can one GPU serve?
It depends on how many requests run at the same time, how long their prompts and answers are, the model size and quantization, and the response speed you need. Staff count alone does not determine it. Estimate peak concurrent requests first, then size memory for weights plus a KV cache for each concurrent sequence.
Why does context length change the hardware I need?
Every token of context held for an active request occupies KV-cache memory. For Llama 3.1 8B at FP16 that is about 1.07 GB per 8,192-token sequence; eight such sequences need roughly 8.6 GB on top of the model weights.
Are the tokens-per-second figures benchmarks?
No. They are estimates from memory bandwidth divided by the bytes read per generated token, scaled by an efficiency band. Real speed depends on the runtime, kernels and batching. Treat them as a screening tool, then benchmark your workload on the shortlisted hardware.
Can I split a model across several GPUs?
Yes. Layer splitting lets a model that does not fit one card run across several, but it does not make a single stream faster. Tensor parallelism can, at the cost of interconnect bandwidth and setup complexity. This tool shows the number of cards needed for memory, not a parallelism plan.