Paged Attention

Full definition

Introduced in vLLM (Kwon et al., SOSP 2023), paged attention solves KV-cache fragmentation. Naive serving allocates a contiguous KV-cache buffer sized to the maximum context, wasting 60–80% of KV-cache memory on fragmentation and over-reservation. Paged attention allocates KV-cache in 16-token blocks and uses a page table per sequence, eliminating fragmentation and enabling memory sharing across requests with shared prefixes (system prompts, few-shot examples). Reported gains: 2–4x throughput vs FasterTransformer and Orca at the same latency (Kwon et al., 2023); the vLLM team's June 2023 launch post reported 14–24x vs Hugging Face Transformers.

Why it matters

Paged attention is what made cheap long-context inference economically viable. Its fixed-size blocks also make prefix caching cheap to implement: vLLM's automatic prefix caching hashes each KV-cache block by its tokens and prefix, so requests that share a system prompt reuse the same blocks.

Example

A coding assistant ships a 30k-token system prompt with rules and examples; paged attention shares those KV-cache blocks across 200 concurrent users, freeing 90% of the memory a naive implementation would consume.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.

More infrastructure terms

• Model Parallelism
  Model parallelism is a distributed-training and inference technique that splits a single neural network across multiple devices when the model does not fit on one device, in contrast to data parallelism (which replicates the model).
• Tensor Parallelism
  Tensor parallelism is a model-parallelism strategy that splits individual matrix multiplications across multiple GPUs along the hidden dimension, recombining results with all-reduce or all-gather collectives.
• FlashAttention
  FlashAttention is an exact attention algorithm that reorders computation to minimize GPU high-bandwidth-memory (HBM) reads and writes, making attention 2–4x faster than standard implementations without changing outputs.