Mixture of Experts (MoE)

Full definition

In an MoE layer, a learned router sends each token to the top-k of N experts (often k=2, N=8 to 128). Only those experts run, so a "1.6T-parameter" MoE may execute only ~30B parameters per token. Mixtral 8x22B and DeepSeek V3 are MoE models, and GPT-4 is widely reported (though not confirmed by OpenAI) to be one. MoE is the main architectural reason frontier model serving costs have not grown linearly with parameter count.

Why it matters

MoE is one of the biggest unit-economics wins in modern LLMs. It lets providers ship models with frontier-level quality at a fraction of the inference cost of quality-equivalent dense models (Microsoft's DeepSpeed-MoE research measured up to 4.5× faster and 9× cheaper inference) — which directly lowers per-token API prices and the cost of any RAG or agent system built on top.

Example

Mistral's Mixtral 8x22B has 141B total parameters but activates only 39B per token, delivering quality comparable to dense 70B models at a fraction of the inference cost.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Foundation Model
  A foundation model is a large model trained on broad data at scale — typically self-supervised — that can be adapted to many downstream tasks via prompting, fine-tuning, or retrieval, instead of being trained task-by-task.
• Transformer
  The transformer is a neural-network architecture built around the self-attention mechanism that has become the dominant model design for language, vision, audio, and multimodal AI since 2017.

More models terms

• Small Language Model (SLM)
  A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.
• Attention Mechanism
  An attention mechanism is a neural-network operation that lets each output position weigh every input position by learned relevance scores, replacing fixed-window context with content-addressed lookup.
• Transformer Architecture (Components)
  Transformer architecture is the internal design of a transformer model: a stack of identical blocks, each combining multi-head self-attention, a position-wise feed-forward network, residual connections and layer normalization, arranged as an encoder, a decoder or both. For what the transformer is and why it dominates AI, see Transformer.