Pretraining

Full definition

Pretraining a frontier LLM today consumes 10^24-10^26 FLOPs and tens of millions of dollars in compute. It produces a "base model" — knowledgeable but not yet helpful — which is then post-trained (instruction tuning, RLHF, RLAIF) into the chat assistants users interact with. Most enterprises do not pretrain; they consume pretrained foundation models and adapt them.

Why it matters

Pretraining capital costs have created a small set of frontier model providers (OpenAI, Anthropic, Google, Meta, xAI, Mistral). For most enterprises this is good news: the heavy lift is amortized across the industry. The strategic decision is which foundation models to standardize on, not whether to pretrain.

Example

Meta's Llama 3.1 405B was pretrained on 15.6 trillion tokens using over 16,000 H100 GPUs — a multi-month, multi-hundred-million-dollar effort that any downstream user gets to inherit by downloading the open weights.

Related terms

• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Fine-tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Transfer Learning
  Transfer learning is the practice of using a model trained on one task as the starting point for a related task, dramatically reducing the data and compute required to reach high accuracy on the new task.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.