Context Window
A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
Full definition
Context windows have grown from 2,048 tokens (GPT-2) to 8K (GPT-3.5), 200K (Claude Sonnet 4.5), 1M (Gemini 1.5 Pro), and beyond. Larger windows enable RAG over more documents, longer conversations, and full-codebase reasoning — but quality degrades non-uniformly across the window ("lost in the middle" effect), and inference cost grows quadratically with attention. The right context size is task-specific.
Why it matters
Context-window selection is now a real engineering decision: a 1M-token window costs 100× more per query than a 10K window for many tasks, and may not deliver better quality if the relevant signal is overwhelmed by noise. Strong RAG often beats brute-force long-context, especially under cost pressure.
Example
A legal-research team compares two patterns for a 200-page deposition: (a) load the whole document into Claude's 200K context, (b) RAG-retrieve the 8 most relevant passages. RAG produces equivalent answer quality at 12% of the inference cost.
Related terms
- Large Language Model (LLM)A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
- Token / TokenizationA token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
- Retrieval-Augmented Generation (RAG)Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, reducing hallucination by 30-60% and enabling citation of sources.
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- KV CacheThe KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
Source & further reading
Primary source: Liu et al. — "Lost in the Middle: How Language Models Use Long Contexts" (2023).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/it/glossary/context-window/.