KV Cache

Full definition

During autoregressive generation, the transformer's self-attention requires the keys and values of every prior token. Caching them turns generation into linear-cost-per-token rather than quadratic. KV-cache size scales with batch size × sequence length × layers × heads × head-dim, often dominating GPU memory for long contexts. PagedAttention (vLLM) and prefix caching are the two highest-impact optimizations.

Why it matters

KV-cache management is the difference between an LLM serving 1 user per GPU and 50. PagedAttention alone can deliver 2–4× throughput; prefix caching cuts cost further when requests share a long prefix (Anthropic, for example, bills cache hits at 10% or less of the base input price).

Example

A SaaS company's RAG endpoint reuses the same 4,000-token system prompt + retrieved-documents prefix across millions of requests. Prefix caching reduces both latency and cost by ~40%.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Inference
  Inference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.
• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.

More infrastructure terms

• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Speculative Decoding
  Speculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.