Transfer Learning

Full definition

Transfer learning works because lower layers in deep networks learn general features (edges, words, syntax) reusable across tasks. Modern foundation models are the apex of transfer learning — pretrain once on broad data, then transfer to thousands of downstream tasks via fine-tuning, prompting, or RAG. Without transfer learning, the data and compute required to train every task-specific model from scratch would be prohibitive.

Why it matters

Transfer learning is what makes "AI for everyone" economically possible. A small team can fine-tune a 70B foundation model on 5,000 examples and beat a from-scratch model trained on 5 million examples — a 1000× data efficiency gain.

Example

A retailer fine-tunes BERT (pretrained on 3.3B words) on 8,000 labeled product descriptions to build a category classifier. Accuracy reaches 94%; training takes 2 hours on a single GPU.

Related terms

• Fine-tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Few-shot / Zero-shot Learning
  Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.