Context Window

Full definition

Context windows have grown from 2,048 tokens (GPT-2) to 8K (GPT-3.5), 200K (Claude Sonnet 4.5), 1M (Gemini 1.5 Pro), and beyond. Larger windows enable RAG over more documents, longer conversations, and full-codebase reasoning — but quality degrades non-uniformly across the window ("lost in the middle" effect), and inference cost grows quadratically with attention. The right context size is task-specific.

Why it matters

Context-window selection is now a real engineering decision: a 1M-token window costs 100× more per query than a 10K window for many tasks, and may not deliver better quality if the relevant signal is overwhelmed by noise. Strong RAG often beats brute-force long-context, especially under cost pressure.

Example

A legal-research team compares two patterns for a 200-page deposition: (a) load the whole document into Claude's 200K context, (b) RAG-retrieve the 8 most relevant passages. RAG produces equivalent answer quality at 12% of the inference cost.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
• Retrieval-Augmented Generation (RAG)
  Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, reducing hallucination by 30-60% and enabling citation of sources.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.