Chunking Strategy

Full definition

Common strategies: fixed-token chunks with overlap (simple, baseline), recursive character splitting (respects markdown/code structure), semantic chunking (split at embedding-distance boundaries), hierarchical chunking (parent-child with small chunks pointing to larger context), and proposition chunking (atomic factual claims). Chunk size 256–512 tokens with 10–20% overlap is the typical starting point. Too small loses context; too large dilutes retrieval signal.

Why it matters

Chunking is the most underrated lever in RAG. Teams that obsess over embedding-model choice while ignoring chunking leave retrieval quality on the table; Chroma's 2024 evaluation found some chunking strategies outperform others by up to 9% in recall. Reviewing chunks against real failed queries is the highest-ROI debugging step.

Example

A medical-records RAG fails on patient-history questions; chunks are 200 tokens but symptoms appear 600 tokens before diagnoses. Switching to hierarchical chunks with 1500-token parent context fixes 70% of failures.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Retrieval-Augmented Generation (RAG)
  Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, which reduces hallucination and enables citation of sources.
• Embedding Model
  An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Semantic Search
  Semantic search is information retrieval that ranks documents by meaning rather than exact-keyword overlap, using vector embeddings of query and documents to measure similarity in a learned semantic space.

More deployment & operations terms

• Agent Memory
  Agent memory is the mechanism by which an AI agent stores and retrieves information across turns or sessions, beyond the fixed context window of the underlying language model.
• Planner-Executor
  Planner-executor is an agent pattern that separates a high-level planning model — which produces a multi-step plan up front — from a tool-calling executor that runs each step, decoupling strategic reasoning from tactical action.
• Tool Router
  A tool router is a layer in an agent system that selects which tool (or subset of tools) to expose to the model for a given query, instead of putting all tool definitions in every prompt.