Chain-of-Thought (CoT)

Full definition

CoT was demonstrated by Wei et al. (2022) with few-shot examples of stepwise reasoning; Kojima et al. (2022) then showed that simply adding "Let's think step by step" elicits it with no examples (zero-shot CoT). The 2024–2025 reasoning-model wave (OpenAI o1/o3, DeepSeek R1, Anthropic's extended thinking) extends CoT into post-training: models are trained to spend variable amounts of inference compute on reasoning before answering.

Why it matters

CoT and reasoning models change AI unit economics: they spend more inference compute, and so more money, per answer in exchange for higher accuracy on hard tasks. For high-stakes workflows (financial analysis, medical reasoning, code generation, legal research) the trade-off is usually worth it.

Example

On the GSM8K math benchmark, Google's PaLM 540B rose from 17.9% accuracy with standard prompting to 56.9% with chain-of-thought prompting (Wei et al., 2022), without any change to the model.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Prompt Engineering
  Prompt engineering is the practice of designing the inputs to a language model — instructions, examples, role definitions, output formats, and constraints — to reliably produce a desired output without changing the model's weights.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Few-Shot / Zero-Shot Learning
  Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.

More training & optimization terms

• Hallucination
  An AI hallucination is when a language or generative model produces content that is plausible-sounding but factually incorrect, fabricated, or unfaithful to its sources — one of the biggest failure modes of LLMs in regulated and high-stakes domains.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.