Serving Large Language Models Affordably: Quantization, Distillation, and Inference Optimization

Serving a large language model efficiently begins with a fact that determines nearly every subsequent decision: the two phases of inference have opposite performance characteristics. Prefill is compute-bound. Decode is memory-bandwidth-bound. A serving stack that treats them identically will be inefficient at both.

Why decoding is slow, and why more FLOPS will not fix it

During prefill the whole prompt is processed at once, so weights loaded from memory are amortised across many tokens and the GPU's arithmetic units are the constraint. During decode a single token is generated per step, which requires reading every weight in the model to produce one token. Arithmetic intensity collapses and memory bandwidth becomes the hard limit.

Two consequences follow, and they are the whole strategy. First, batching is nearly free during decode: the weights are read once for the entire batch, so serving thirty-two concurrent sequences costs little more per step than serving one. Throughput scales close to linearly with batch size until something else binds. Second, reducing model bytes — through quantisation — improves decode latency proportionally, because it directly shrinks the numerator.

The KV cache is what actually limits batch size

Batching would be unboundedly good if it were free, and the thing that stops it is the attention cache. Every sequence retains key and value tensors for every token it has processed, at every layer, and that memory grows linearly with both context length and batch size.

Concretely, for an 80-layer model with 8 KV heads and a head dimension of 128 in FP16: 2 × 80 × 8 × 128 × 2 = 320 KB per token. A single 8,192-token sequence therefore holds about 2.6 GB of cache. Thirty-two such sequences need 84 GB — for cache alone, on top of the 140 GB of weights.

Grouped-query attention is the architectural response, and its impact is often underappreciated. Sharing key and value projections across groups of query heads reduces H_kv by the group factor, cutting cache memory by the same factor with minimal quality cost. A model with 64 query heads and 8 KV heads uses one eighth the cache of full multi-head attention — which translates directly into eight times the batch size at the same memory budget.

Continuous batching, because sequences finish at different times

Static batching holds the whole batch until its longest member completes. With generation lengths varying from twenty tokens to two thousand, that leaves most of the batch idle for most of its life while finished sequences occupy slots doing nothing.

Continuous batching evicts each sequence the moment it emits its stop token and admits a waiting request into the freed slot at the next step. Reported throughput gains on realistic mixed workloads are large — commonly several times static batching — and the gain grows with the variance of output length, which in production is high.

The tension between the second and third rows is the central scheduling problem. Larger batches raise aggregate throughput and worsen per-user latency, because each decode step does more work. A serving stack must be told which it is optimising — an interactive assistant and an overnight batch job want opposite settings, and running them on the same pool at the same settings serves neither well.

Prefill and decode also interfere. A long prompt arriving mid-flight monopolises the GPU for a compute-heavy prefill while every decoding sequence stalls, producing a latency spike for users who did nothing. Chunked prefill splits long prompts into pieces interleaved with decode steps, converting one large stall into several small ones — a clear improvement in tail latency at a small cost in prefill efficiency.

Speculative decoding: buy tokens with spare compute

Since decode is bandwidth-bound, arithmetic capacity is sitting idle. Speculative decoding spends it: a small draft model proposes several tokens cheaply, and the target model verifies all of them in a single forward pass — a compute-heavy operation that costs barely more than generating one token, precisely because compute was not the constraint.

Everything depends on α, which depends on how well the draft model matches the target's distribution on the actual workload. A draft from the same family and training data achieves high acceptance; a generic small model does not. Measure α on production traffic rather than assuming it, because below roughly 0.6 the drafting overhead starts eating the gain.

Quantisation, where the gain is bandwidth rather than compute

Because decode is bandwidth-bound, halving weight precision roughly halves decode latency. FP8 on hardware with native support is close to free in quality terms for most models. Weight-only INT4 methods that preserve the outlier channels responsible for most quantisation error push the memory saving further, at a quality cost that must be measured on your evaluation set — not on a public benchmark, which will not exercise your domain.

Quantise the KV cache as well. It is frequently overlooked despite being, at long context and high batch, the larger consumer. FP8 cache halves that footprint and buys back batch size, which is throughput.

The order to do this in

Paged KV cache and continuous batching first — they are pure wins with no quality cost and typically the largest gains. Then quantisation of weights and cache, measured against your own evaluation set. Then chunked prefill if tail latency is the complaint. Then speculative decoding, if you have a draft model that earns its acceptance rate. And separate pools for interactive and batch traffic before tuning any of it, because a single pool cannot be optimal for both and most of the tuning arguments we are asked to arbitrate turn out to be that problem wearing a different hat.