Inference Cost

Full definition

Inference cost is driven by tokens (input + output), model size, hardware (GPU/TPU/CPU), batching efficiency, KV-cache reuse, and quantization. For external APIs, cost is published per million tokens; for self-hosted, it is GPU-hour amortized over throughput. As deployments scale, inference — not training — becomes the dominant lifetime cost: NVIDIA reported in 2024 that inference accounts for ~40% of its data-center revenue and is growing.

Why it matters

Inference cost is now a board-level metric for any company shipping AI at scale. A 50% inference-cost reduction often turns a money-losing AI feature into the most profitable product line. Organizations that do not measure dollars-per-task cannot optimize it.

Example

A SaaS company reduces its support-ticket-classification cost from $0.012 to $0.0009 per ticket — 92% — by routing easy tickets to a fine-tuned 8B SLM and only escalating hard cases to GPT-4o. At 6 million tickets/month, that is $66,600/month saved.

Related terms

• Inference
  Inference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
• Token / Tokenization
  A token is the basic unit a language model reads or writes — usually a sub-word fragment (about 4 characters of English text) — and the unit by which API pricing, context-window limits, and inference cost are all measured.
• Small Language Model (SLM)
  A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.