Model Parallelism

Full definition

Three flavors: (1) Pipeline parallelism — different layers on different GPUs, micro-batches stream through; (2) Tensor parallelism — single matrix multiplications split across GPUs (Megatron-LM); (3) Expert parallelism — MoE experts on different GPUs. Modern training stacks (FSDP, DeepSpeed-Zero, Megatron) combine model parallelism with sharded data parallelism. Frontier-scale training (>100B params) requires at least tensor + pipeline + data parallelism, often called 3D parallelism.

Why it matters

Model parallelism dictates training cluster topology and inter-GPU bandwidth requirements. Self-hosting Llama 3 405B requires tensor parallelism across 8+ H100s with NVLink — a hardware constraint procurement teams must plan for before signing GPU contracts.

Example

Meta trained Llama 3.1 405B on 16,000 H100s using FSDP + tensor parallelism + pipeline parallelism, sustaining ~400 TFLOPS/GPU — about 40% of peak.

Related terms

• Tensor Parallelism
  Tensor parallelism is a model-parallelism strategy that splits individual matrix multiplications across multiple GPUs along the hidden dimension, recombining results with all-reduce or all-gather collectives.
• GPU (in AI context)
  A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
• Training Cost
  Training cost is the total cost — GPU/TPU compute, energy, data acquisition, and labor — required to train a machine-learning model from scratch or to fine-tune a pretrained one to a target capability or domain.
• Pretraining
  Pretraining is the first, most compute-intensive stage of training a foundation model — typically self-supervised next-token prediction over trillions of tokens of text, code, and other modalities — that produces the base model later fine-tuned for specific tasks.