Inference

Full definition

Inference is the operational counterpart to training. Training happens once (or periodically); inference happens millions or billions of times in production. As model sizes grew, inference — not training — became the dominant ongoing cost: NVIDIA reported in 2024 that inference accounts for roughly 40% of its data-center revenue, and analysts expect that share to keep rising as deployments scale.

Why it matters

For any AI product at scale, inference cost is the unit economic that decides whether the business model works. A large enough inference-cost reduction can turn a money-losing AI feature into a profitable one. Optimization techniques — quantization, distillation, KV-cache reuse, speculative decoding — are now as strategically important as model accuracy.

Example

When you send a prompt to ChatGPT, the round-trip — tokenization, forward pass through the model, response streaming — is one inference call. OpenAI charges per million input and output tokens because each is a measurable inference cost.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.

More fundamentals terms

• Model
  An AI model is the trained artifact — a specific set of numerical weights plus an architecture — produced when a learning algorithm runs against training data, and the thing that actually gets deployed and audited.
• Supervised Learning
  Supervised learning is the machine-learning paradigm in which a model learns from training examples paired with correct labels, then predicts labels for new, unseen inputs — the dominant approach in production ML today.
• Unsupervised Learning
  Unsupervised learning is a machine-learning paradigm where a model is given only inputs — no labels — and must discover structure in the data: clusters, density, low-dimensional representations, or anomalies.