Paged Attention

Full definition

Introduced in vLLM (Kwon et al., SOSP 2023), paged attention solves KV-cache fragmentation. Naive serving allocates a contiguous KV-cache buffer sized to the maximum context, wasting 60-80% of GPU memory on padding and reservation. Paged attention allocates KV-cache in 16-token blocks and uses a page table per sequence, eliminating fragmentation and enabling memory sharing across requests with shared prefixes (system prompts, few-shot examples). Reported gains: 2-4x throughput vs FasterTransformer, 5-10x vs HuggingFace Transformers.

Why it matters

Paged attention is what made cheap long-context inference economically viable. Anthropic's prompt caching, OpenAI's prefix caching, and every modern serving engine implement variants of it. Without paged attention, $0.30/1M-input-token pricing for cached prompts would not exist.

Example

A coding assistant ships a 30k-token system prompt with rules and examples; paged attention shares those KV-cache blocks across 200 concurrent users, freeing 90% of the memory a naive implementation would consume.

Related terms

• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.