Training a model gets the headlines, but inference — answering requests — is where most AI compute is spent once a model is popular. Serving a large language model efficiently is a hard systems problem, and a few key ideas explain how modern serving engines such as vLLM, TGI, SGLang and TensorRT-LLM achieve high throughput.

Two phases: prefill and decode

Every request goes through two phases.

  1. Prefill: the model processes the entire prompt at once. This is compute-heavy but parallel, so GPUs handle it efficiently.
  2. Decode: the model generates the answer one token at a time. Each step depends on the previous one, so it can’t be parallelized within a request. Decode is usually limited by memory bandwidth: the model’s weights must be read from GPU memory for every token.

This is why “time to first token” (mostly prefill) and “tokens per second” (decode) are measured separately.

The KV cache

During generation, the attention mechanism needs information about all previous tokens. Recomputing it each step would be wasteful, so servers store intermediate results — keys and values — in the KV cache. It makes decoding fast but consumes a lot of GPU memory, growing with the length of the conversation and the number of simultaneous requests. Managing the KV cache is central to serving performance.

Batching: serving many users at once

Because decoding reads all the weights for each token anyway, processing many requests together is almost free: the same weight read serves the whole batch. Batching dramatically increases throughput.

Simple static batching waits for a group of requests and processes them together until all finish — so short requests wait for long ones. Continuous batching (also called in-flight or iteration-level batching), described in the Orca paper presented at OSDI 2022, adds and removes requests at every generation step. Finished requests leave immediately and new ones join, keeping the GPU busy.

PagedAttention and vLLM

The vLLM project, from researchers at UC Berkeley, introduced PagedAttention in 2023. Earlier systems reserved contiguous memory for each request’s maximum possible length, wasting much of it. PagedAttention borrows the idea of virtual-memory paging from operating systems: the KV cache is split into fixed-size blocks allocated on demand. Less waste means more requests fit in memory, and the authors reported large throughput gains over earlier serving systems.

Other important techniques

  • Prefix caching: reuse the KV cache for shared prompt prefixes, such as a common system prompt.
  • Speculative decoding: a small draft model proposes several tokens that the large model verifies at once, speeding up generation.
  • Quantization: lower-precision weights and KV cache reduce memory and increase speed.
  • Tensor and pipeline parallelism: split large models across multiple GPUs.
  • Disaggregated serving: run prefill and decode on separate hardware pools, since they stress different resources.

Metrics that matter

  • Time to first token (TTFT): responsiveness.
  • Inter-token latency / tokens per second: reading speed.
  • Throughput: total tokens per second across all users.
  • Cost per million tokens: the business metric.

There is always a trade-off: bigger batches improve throughput and cost but can increase latency for each user.

Common mistakes to avoid

  • Benchmarking with one request at a time. Single-request speed says little about behavior under real concurrent load.
  • Ignoring prompt length distribution. Long prompts change memory needs and latency; test with realistic inputs.
  • Sizing GPUs for weights only. The KV cache for many simultaneous requests can need as much memory as the model.
  • Optimizing throughput at any cost. Interactive products need acceptable time to first token, not just maximum tokens per second.

Key takeaways

  • LLM inference has a parallel prefill phase and a sequential, memory-bound decode phase.
  • The KV cache speeds up decoding but consumes large amounts of GPU memory.
  • Continuous batching keeps GPUs busy by adding and removing requests at every step.
  • vLLM’s PagedAttention manages the KV cache in blocks, reducing waste and raising throughput.
  • Prefix caching, speculative decoding, quantization and parallelism further improve serving.

Frequently asked questions

Do I need vLLM to self-host a model?

Not necessarily. For a few users, simpler tools such as Ollama are enough. Serving engines like vLLM matter when many users share the same model.

Why are output tokens slower than input tokens?

Input is processed in parallel during prefill, while output tokens are generated one at a time, each requiring a full pass through the model.

What is speculative decoding?

A technique in which a fast, small model drafts several tokens and the large model checks them in one step, accepting correct ones and speeding up generation without changing the output distribution.