Distillation

Full definition

Distillation transfers knowledge by training the student to match the teacher's soft probabilities (not just hard labels) and, in modern LLM distillation, to match generated reasoning traces and tool-use patterns. DeepSeek-R1's distilled smaller siblings, Gemini Flash from Gemini Pro, and the entire small-LLM frontier of 2024-2025 are products of distillation. Combined with quantization and fine-tuning, distillation is the standard path to cost-efficient production AI.

Why it matters

Distillation is the bridge between frontier capability and production economics. It lets organizations consume frontier-model quality where they need it (training data, complex tasks) and run distilled, cheap models where they need scale (production endpoints, edge devices).

Example

A SaaS company uses GPT-4o to generate 50,000 high-quality task examples, then fine-tunes a 7B Llama-3 on them. The result: 95% of GPT-4o's task quality at 3% of the inference cost.

Related terms

• Fine-tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Small Language Model (SLM)
  A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.