Continuous Batching

Full definition

Static batching wastes GPU cycles because short requests sit idle waiting for long ones. Continuous batching (also called iteration-level scheduling, introduced in Orca, OSDI 2022) evicts finished requests every step and admits new ones into the freed slots. Orca reported 36.9x the throughput of NVIDIA FasterTransformer at the same latency on GPT-3 175B; in Anyscale's 2023 benchmark, continuous batching alone delivered up to 8x the throughput of naive static batching, and up to 23x combined with vLLM's paged attention, while also lowering median latency.

Why it matters

Continuous batching is now table stakes for any serious LLM inference stack (vLLM, TensorRT-LLM, SGLang, TGI). Buyers comparing self-hosted GPU economics should ask which serving engine the vendor uses; teams still on static batching can pay several times more per token on the same hardware (Anyscale measured 8x the throughput from continuous batching alone over naive static batching, and up to 23x with vLLM).

Example

A SaaS company switches from a custom Triton server to vLLM for Llama 3 70B serving; throughput rises from 600 to 4,800 tokens/sec on the same 8xH100 node, cutting cost per million tokens eightfold, from about $11.57 to $1.45 at $25/hr for the node.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.

More infrastructure terms

• Model Parallelism
  Model parallelism is a distributed-training and inference technique that splits a single neural network across multiple devices when the model does not fit on one device, in contrast to data parallelism (which replicates the model).
• Tensor Parallelism
  Tensor parallelism is a model-parallelism strategy that splits individual matrix multiplications across multiple GPUs along the hidden dimension, recombining results with all-reduce or all-gather collectives.