Pretraining

Full definition

Pretraining a frontier LLM today takes months of compute on thousands of accelerators, at a cost of tens to hundreds of millions of dollars (Epoch AI's median estimate for Grok 4, released in 2025, is about $490 million). It produces a "base model" — knowledgeable but not yet helpful — which is then post-trained (instruction tuning, RLHF, RLAIF) into the chat assistants users interact with. Most enterprises do not pretrain; they consume pretrained foundation models and adapt them.

Why it matters

Pretraining capital costs have created a small set of frontier model providers (OpenAI, Anthropic, Google, Meta, xAI, Mistral). For most enterprises this is good news: the heavy lift is amortized across the industry. The strategic decision is which foundation models to standardize on, not whether to pretrain.

Example

Meta's Llama 3.1 405B was pretrained on 15.6 trillion tokens using over 16,000 H100 GPUs — a multi-month effort whose compute the Stanford AI Index estimates at about $170 million, which any downstream user gets to inherit by downloading the open weights.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Fine-Tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Transfer Learning
  Transfer learning is the practice of using a model trained on one task as the starting point for a related task, dramatically reducing the data and compute required to reach high accuracy on the new task.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.

More training & optimization terms

• Few-Shot / Zero-Shot Learning
  Few-shot learning is the ability of a model to perform a new task given only a handful of examples at inference time; zero-shot learning is the same idea with no examples — the model relies entirely on its pretraining and the natural-language instruction.
• Prompt Engineering
  Prompt engineering is the practice of designing the inputs to a language model — instructions, examples, role definitions, output formats, and constraints — to reliably produce a desired output without changing the model's weights.