Mixture of Experts (MoE)

Full definition

In an MoE layer, a learned router sends each token to the top-k of N experts (often k=2, N=8 to 128). Only those experts run, so a "1.6T-parameter" MoE may execute only ~30B parameters per token. GPT-4, Mixtral 8x22B, and DeepSeek V3 are MoE-based. MoE is the main architectural reason frontier model serving costs have not grown linearly with parameter count.

Why it matters

MoE is one of the biggest unit-economics wins in modern LLMs. It lets providers ship models with frontier-level quality at 3-5× lower inference cost than equivalent dense models — which directly lowers per-token API prices and the cost of any RAG or agent system built on top.

Example

Mistral's Mixtral 8x22B has 141B total parameters but activates only 39B per token, delivering quality comparable to dense 70B models at a fraction of the inference cost.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.