Transfer Learning

Full definition

Transfer learning works because lower layers in deep networks learn general features (edges, words, syntax) reusable across tasks. Modern foundation models are the apex of transfer learning — pretrain once on broad data, then transfer to thousands of downstream tasks via fine-tuning, prompting, or RAG. Without transfer learning, the data and compute required to train every task-specific model from scratch would be prohibitive.

Why it matters

Transfer learning is what makes "AI for everyone" economically possible. A small team can fine-tune a pretrained foundation model on a few thousand labeled examples and match or beat a model trained from scratch on far more data; in the ULMFiT study (Howard and Ruder, 2018), 100 labeled examples matched the performance of training from scratch on 100× more data.

Example

A retailer fine-tunes BERT (pretrained on 3.3B words) on 8,000 labeled product descriptions to build a category classifier. Accuracy reaches 94%; training takes 2 hours on a single GPU.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Fine-Tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Few-Shot / Zero-Shot Learning
  Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.

More training & optimization terms

• Prompt Engineering
  Prompt engineering is the practice of designing the inputs to a language model — instructions, examples, role definitions, output formats, and constraints — to reliably produce a desired output without changing the model's weights.
• Chain-of-Thought (CoT)
  Chain-of-Thought (CoT) is a prompting and training technique in which a language model is encouraged to reason step-by-step before producing its final answer — improving accuracy on math, logic, and multi-hop reasoning in large models (in Wei et al., 2022, GSM8K math accuracy more than doubled for the largest GPT and PaLM models).