Model Serving

Full definition

Common serving stacks include NVIDIA Triton, KServe, vLLM (for LLMs), and managed services like SageMaker Endpoints and Vertex AI Online Prediction. Serving is where dynamic batching, paged attention (KV cache), tensor parallelism, and quantized weights translate directly into dollars: a 2x serving efficiency gain on a billion-call workload is a six-to-seven-figure annualized line item.

Why it matters

For most enterprises, lifetime inference cost dwarfs training cost. Serving choice — framework, batching policy, hardware, quantization — is one of the largest levers on GenAI unit economics: the vLLM paper reports 2–4× the throughput of FasterTransformer and Orca at the same latency. Treat serving as a platform discipline, not a config flag.

Example

A SaaS swaps a vanilla HuggingFace endpoint for vLLM with continuous batching and 8-bit quantization, cutting per-token GPU cost 4.6x at unchanged quality.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.

More infrastructure terms

• Speculative Decoding
  Speculative decoding is an inference technique where a small "draft" model proposes multiple candidate tokens that the large "target" model verifies in a single forward pass, accepting prefixes that match the target distribution.
• Continuous Batching
  Continuous batching is an inference scheduling technique that adds and removes requests from a GPU batch every decoding step, instead of waiting for all requests in a static batch to finish.
• Paged Attention
  Paged attention is a KV-cache memory-management technique that stores attention keys and values in fixed-size blocks, addressed via a page table — analogous to virtual memory in operating systems.