Serving LLMs on-premise: GPU, quantization, and throughput
A technical guide to sizing GPU and memory, applying quantization, continuous batching, and KV cache when serving language models inside the company.
"Cloud, private cloud, or on-premise: where your AI runs" framed the decision at the executive level — cost, data control, latency, and vendor lock-in. This article is the technical deep dive for teams that have decided (or are seriously evaluating) running language models inside their own environment. The focus is GPU and memory sizing, quantization, continuous batching, KV cache, the p95 latency vs. throughput trade-off, high availability, and the hybrid on-premise + cloud model.
Sizing GPU and memory
The most common mistake is sizing by parameter count alone, ignoring actual usage context. Memory needed to serve a model has three components:
| Component | What it consumes | Scales with |
|---|---|---|
| Model weights | Parameters in FP16/BF16 or quantized | Model size |
| KV cache | Attention keys/values per active request | Context length × concurrent requests |
| Activations and overhead | Execution buffers, batching | Batch size |
Rule of thumb: a 7B-parameter model in FP16 uses ~14 GB in weights alone. Without headroom for KV cache and batching, a 24 GB GPU serves few concurrent requests with long context. KV cache is routinely underestimated — at 32k-token contexts with multiple simultaneous sessions, it can exceed the size of the weights themselves.
Before buying GPUs, measure: average and p95 input context length, average output length, and expected peak concurrency. Those three numbers, not "the most advanced model on the market," define the right GPU.
Quantization: where it wins and where it costs
Quantization reduces the precision of weights (and sometimes activations) to save memory and increase throughput. Options commonly used in production:
- INT8: generally low quality loss, ~2x memory savings over FP16, broadly supported.
- INT4 (GPTQ, AWQ): ~4x memory savings, but requires careful validation per use case — long-form reasoning or precise data extraction is more sensitive to degradation.
- FP8: on recent GPUs (Hopper and above), offers a good precision/throughput balance without complex calibration.
This should not be a generic decision. Run the same evaluation suite (see "Instrumenting cost and quality") on the original-precision model and the quantized version, comparing use-case-specific quality metrics, not just generic perplexity. In many support and internal search scenarios, INT4 is indistinguishable in practice; in code generation or legal analysis, the gap shows.
Continuous batching and KV cache
Modern inference servers (vLLM, TGI, TensorRT-LLM) implement continuous batching: instead of waiting for a closed batch, new requests enter as soon as there is room, and completed requests exit without blocking others. This dramatically increases throughput versus static batching, especially with variable output lengths — typical in chat and agents.
Paged KV cache (PagedAttention and equivalent techniques) avoids memory fragmentation by allocating cache in blocks, enabling prefix sharing across requests (useful when the same system prompt is reused). This matters especially in agent architectures with long, repeated system prompts.
Operational notes:
- Set context length limits per usage route; unrestricted contexts destroy capacity predictability.
- Monitor prefix cache-hit rate where available — it's a direct indicator of cost efficiency.
- Test behavior under spikes: continuous batching improves the average, but tail latency still depends on how many long requests arrive simultaneously.
p95 latency vs. throughput
These two goals compete. Increasing average batch size improves throughput (more tokens processed per second per GPU) but can worsen latency for individual requests, especially those arriving when the batch is already full.
Define the SLO per interaction type before tuning the server:
| Interaction type | Priority | Recommended configuration |
|---|---|---|
| Interactive chat | Low p95 latency | Smaller batch, queue priority for first token |
| Batch/back-office processing | Throughput | Larger batch, no real-time requirement |
| Multi-call agents | Per-call latency + total cost | Prefix caching, smaller models for intermediate steps |
Without this segmentation, a single server configuration tries to serve opposing goals and satisfies neither well.
High availability
GPU is an expensive resource, and without redundancy it's a single point of failure. Minimum practices:
- At least two replicas of the inference service behind a load balancer with real health checks (not just "process alive," but "responds correctly to a test prompt").
- A zero-downtime update strategy (rolling update) when swapping model or inference server versions.
- A degradation plan: if on-premise capacity saturates, decide in advance where overflow goes — queue, simplified response, or cloud fallback.
The hybrid model
Few companies run 100% on-premise or 100% cloud for every use case. The most common pattern is hybrid: models handling sensitive data or high-volume cost stay on-premise; peak loads, frontier models for specific tasks, or experimental use go to cloud APIs. This requires a routing layer (gateway) that decides by use case, not by accident, and treats both routes with the same discipline for observability and cost.
What to do on Monday
- 1.Gather the three numbers that drive sizing: average/p95 context, output length, peak concurrency — for your most critical use case.
- 2.Run an A/B quality test between original and quantized precision (INT8 or INT4) using your evaluation suite, not generic benchmarks.
- 3.Check whether your current inference server uses continuous batching and paged KV cache; if not, evaluate migrating to vLLM, TGI, or equivalent.
- 4.Define separate latency SLOs by interaction type and configure queues/priorities accordingly.
- 5.Document the degradation plan for load spikes before it gets tested in production without warning.
Further reading
Executive track:
- Cloud, private cloud or on-premise: where your AI should run — the business view of this same topic.