Model Serving

Full definition

Common serving stacks include NVIDIA Triton, KServe, TorchServe, vLLM (for LLMs), and managed services like SageMaker Endpoints and Vertex AI Online Prediction. Serving is where dynamic batching, paged attention (KV cache), tensor parallelism, and quantized weights translate directly into dollars: a 2x serving efficiency gain on a billion-call workload is a six-to-seven-figure annualized line item.

Why it matters

For most enterprises, lifetime inference cost dwarfs training cost. Serving choice — framework, batching policy, hardware, quantization — typically determines 50-80% of GenAI unit economics. Treat serving as a platform discipline, not a config flag.

Example

A SaaS swaps a vanilla HuggingFace endpoint for vLLM with continuous batching and 8-bit quantization, cutting per-token GPU cost 4.6x at unchanged quality.

Related terms

• Inference Server
  An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• KV Cache
  The KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.