KV Cache
The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
Full definition
During autoregressive generation, the transformer's self-attention requires the keys and values of every prior token. Caching them turns generation into linear-cost-per-token rather than quadratic. KV-cache size scales with batch size × sequence length × layers × heads × head-dim, often dominating GPU memory for long contexts. PagedAttention (vLLM) and prefix caching are the two highest-impact optimizations.
Why it matters
KV-cache management is the difference between an LLM serving 1 user per GPU and 50. PagedAttention alone can deliver 2-4× throughput; prefix caching can cut cost by another 30-70% on RAG-heavy workloads with shared system prompts.
Example
A SaaS company's RAG endpoint reuses the same 4,000-token system prompt + retrieved-documents prefix across millions of requests. Prefix caching reduces both latency and cost by ~40%.
Related terms
- InferenceInference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- Context WindowA context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
- TransformerThe transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
- Inference ServerAn inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
Source & further reading
Primary source: Kwon et al. — "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM) (2023).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/sv/glossary/kv-cache/.