Prefix Caching

Full definition

When two requests share the first N tokens, the KV-cache for those tokens is identical and can be cached and reused. Anthropic's Prompt Caching, OpenAI's prefix caching, Google's context caching, and vLLM's automatic prefix caching all implement variants. Cached tokens are typically 75-90% cheaper and 5-10x faster (TTFT) than uncached. Cache TTLs range from 5 minutes (Anthropic short) to 1 hour (Anthropic long, OpenAI). Cache effectiveness depends on prefix stability — varying tokens at the start defeats caching.

Why it matters

Prefix caching is the highest-impact cost optimization for any LLM workload with long shared context (RAG, agent system prompts, multi-turn chat). Teams not using it on Anthropic or OpenAI APIs typically overpay 3-10x for production traffic.

Example

A coding assistant with a 12k-token system prompt and 8k tokens of retrieved code: with prefix caching, per-query cost drops from $0.18 to $0.03 and TTFT from 2.4s to 320ms — a transformative product change.

Related terms

• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.