Chunking Strategy

Full definition

Common strategies: fixed-token chunks with overlap (simple, baseline), recursive character splitting (respects markdown/code structure), semantic chunking (split at embedding-distance boundaries), hierarchical chunking (parent-child with small chunks pointing to larger context), and proposition chunking (atomic factual claims). Chunk size 256-512 tokens with 10-20% overlap is the typical starting point. Too small loses context; too large dilutes retrieval signal.

Why it matters

Chunking is the most underrated lever in RAG. Teams that obsess over embedding-model choice while ignoring chunking leave 10-30% of retrieval quality on the table. Reviewing chunks against real failed queries is the highest-ROI debugging step.

Example

A medical-records RAG fails on patient-history questions; chunks are 200 tokens but symptoms appear 600 tokens before diagnoses. Switching to hierarchical chunks with 1500-token parent context fixes 70% of failures.

Related terms

• Retrieval-Augmented Generation (RAG)
  Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, reducing hallucination by 30-60% and enabling citation of sources.
• Embedding Model
  An embedding model is a neural network that maps text, images, or other inputs into fixed-dimensional vectors where semantic similarity corresponds to geometric closeness (cosine or dot-product distance).
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Semantic Search
  Semantic search is information retrieval that ranks documents by meaning rather than exact-keyword overlap, using vector embeddings of query and documents to measure similarity in a learned semantic space.