Online Inference

Full definition

Online inference requires always-on capacity, autoscaling, fast feature retrieval (online feature store), and tight observability (latency p50/p99, error rate, request volume). Cost-per-request is materially higher than batch because hardware sits warm awaiting traffic. The trade-off is direct user impact: every additional 100ms of latency measurably reduces conversion in consumer applications.

Why it matters

Online inference is the surface where model quality meets product UX. It is also where almost all GenAI cost lives in a chat or copilot product. Latency budgets, batching policies, and cache hit rates matter as much as model architecture.

Example

A payment processor scores every transaction in <40ms via an online endpoint backed by a feature store; latency above 80ms triggers a fallback rules engine to preserve checkout flow.

Related terms

• Batch Inference
  Batch inference runs a model over a large set of inputs offline on a schedule — typically nightly or hourly — optimized for throughput and cost-per-prediction rather than latency.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Feature Store
  A feature store is a centralized data system that stores, versions, serves, and reuses curated machine-learning features — guaranteeing the same feature definitions are used at training time and at online inference time.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.