Batch Inference

Full definition

Batch jobs maximize hardware utilization by saturating GPUs with large input groups, often using cheap spot/preemptible compute. Outputs are written to a warehouse or feature store and later read by downstream services. Batch is the right choice when downstream consumers tolerate hours-old predictions — churn scoring, demand forecasting, lead scoring, marketing audience generation.

Why it matters

Teams default to online inference when batch would be cheaper by an order of magnitude. A clean rule: if the consumer of a prediction does not need it within seconds, batch it. The savings often subsidize the higher-margin online use cases.

Example

A retailer scores all 40M customers nightly for churn risk in a 25-minute Spark job; the marketing platform reads results in the morning. Doing it online would cost ~30x more and serve no business purpose.

Related terms

• Online Inference
  Online inference returns model predictions synchronously in response to a live user or system request, typically under 100ms end-to-end for tabular models and under 2s for streaming LLM completions.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Feature Store
  A feature store is a centralized data system that stores, versions, serves, and reuses curated machine-learning features — guaranteeing the same feature definitions are used at training time and at online inference time.
• AI Pipeline
  An AI pipeline is the end-to-end sequence of stages that turns raw data into deployed AI predictions or content — ingestion, cleaning, feature engineering, training, evaluation, deployment, monitoring, and retraining — usually orchestrated as code.