Continuous Batching

Full definition

Static batching wastes GPU cycles because short requests sit idle waiting for long ones. Continuous batching (also called iteration-level scheduling, introduced in Orca, OSDI 2022) evicts finished requests every step and admits new ones into the freed slots. Combined with paged attention (vLLM, 2023), this raises GPU utilization from 30-40% to 70-90% on real workloads — typically a 4-23x throughput win over static batching at the same latency.

Why it matters

Continuous batching is now table stakes for any serious LLM inference stack (vLLM, TensorRT-LLM, SGLang, TGI). Buyers comparing self-hosted GPU economics should ask which serving engine the vendor uses; teams still on static batching are paying 4-10x more per token.

Example

A SaaS company switches from a custom Triton server to vLLM for Llama 3 70B serving; throughput rises from 600 to 4,800 tokens/sec on the same 8xH100 node, dropping cost-per-million-tokens from $1.20 to $0.18.

Related terms

• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.