Context Window

Full definition

Context windows have grown from 1,024 tokens (GPT-2) and 2,048 (GPT-3) to 4K–16K (GPT-3.5 Turbo), 200K (Claude Sonnet 4.5), 1M (Gemini 1.5 Pro), and beyond. Larger windows enable RAG over more documents, longer conversations, and full-codebase reasoning — but quality degrades non-uniformly across the window ("lost in the middle" effect), and inference cost grows quadratically with attention. The right context size is task-specific.

Why it matters

Context-window selection is now a real engineering decision: a 1M-token window costs 100× more per query than a 10K window for many tasks, and may not deliver better quality if the relevant signal is overwhelmed by noise. Strong RAG often beats brute-force long-context, especially under cost pressure.

Example

A legal-research team compares two patterns for a 200-page deposition: (a) load the whole document into Claude's 200K context, (b) RAG-retrieve the 8 most relevant passages. RAG produces equivalent answer quality at 12% of the inference cost.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
• Retrieval-Augmented Generation (RAG)
  Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, which reduces hallucination and enables citation of sources.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.

More deployment & operations terms

• Knowledge Graph
  A knowledge graph is a structured representation of entities (people, products, customers, drugs) and the relationships between them, stored as a graph and used to ground AI systems in verified, queryable facts rather than free-form text.
• Feature Store
  A feature store is a centralized data system that stores, versions, serves, and reuses curated machine-learning features — guaranteeing the same feature definitions are used at training time and at online inference time.
• Model Registry
  A model registry is a versioned catalog that tracks every trained ML model artifact, along with its metadata, training data lineage, evaluation metrics, approval status, and deployment stage (staging, production, archived).