Inference

Full definition

Inference is the operational counterpart to training. Training happens once (or periodically); inference happens millions or billions of times in production. As model sizes grew, inference — not training — became the dominant ongoing cost: NVIDIA reported in 2024 that inference accounts for roughly 40% of its data-center revenue, and analysts expect that share to keep rising as deployments scale.

Why it matters

For any AI product at scale, inference cost is the unit economic that decides whether the business model works. A 30% inference-cost reduction often turns a money-losing AI feature into a profitable one. Optimization techniques — quantization, distillation, KV-cache reuse, speculative decoding — are now as strategically important as model accuracy.

Example

When you send a prompt to ChatGPT, the round-trip — tokenization, forward pass through the model, response streaming — is one inference call. OpenAI charges per million input and output tokens because each is a measurable inference cost.

Related terms

• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.