Prefix Caching

Full definition

When two requests share the first N tokens, the KV-cache for those tokens is identical and can be cached and reused. Anthropic's Prompt Caching, OpenAI's prefix caching, Google's context caching, and vLLM's automatic prefix caching all implement variants. On current Anthropic and OpenAI models, cached input is billed at 10% of the base input price or less, and caching can cut time to first token by up to 85% on long prompts (Anthropic). Cache lifetimes vary: Anthropic's default is 5 minutes with an optional 1 hour, while OpenAI's extended retention can keep entries for up to 24 hours. Cache effectiveness depends on prefix stability — varying tokens at the start defeats caching.

Why it matters

Prefix caching is the highest-impact cost optimization for any LLM workload with long shared context (RAG, agent system prompts, multi-turn chat). Teams not using it on Anthropic or OpenAI APIs typically overpay 3–10x for production traffic.

Example

A coding assistant with a 12k-token system prompt and 8k tokens of retrieved code: with prefix caching, per-query cost drops from $0.18 to $0.03 and TTFT from 2.4s to about 0.5s — a transformative product change.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.

More infrastructure terms

• GPU (in AI Context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks many times faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
• TPU
  A TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for tensor operations — the dense matrix multiplications at the core of neural networks — and used in production to train and serve Google Search, Translate, YouTube, and Gemini.
• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.