Inference Server
An inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
Full definition
Mature inference servers include NVIDIA Triton, NVIDIA TensorRT-LLM, vLLM, TGI (Text Generation Inference), and llama.cpp. They differ on supported hardware, throughput optimizations (continuous batching, paged attention, speculative decoding), and operational features (multi-model hosting, A/B routing, observability). Choice of inference server can move throughput by 3-10× on the same hardware.
Why it matters
For self-hosted AI, the inference server is one of the highest-impact cost optimizations. A team running Llama 3 70B on vLLM with continuous batching can serve 5-10× the requests per GPU compared to a naive PyTorch deployment — turning the same hardware investment into 5-10× the unit economics.
Example
A SaaS company switches its Llama 3 70B endpoint from naive Hugging Face Transformers serving to vLLM with paged attention. p95 latency drops 35%, throughput rises 6×, and per-token cost falls below the public OpenAI API's rate.
Related terms
- InferenceInference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- KV CacheThe KV (Key-Value) cache stores the attention keys and values from already-processed tokens so an LLM can generate each new token without recomputing past work — the single largest consumer of GPU memory during LLM inference, and the primary lever for serving throughput.
- GPU (in AI context)A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
- MLOpsMLOps is the discipline of operating machine-learning and AI systems reliably in production — covering data pipelines, model training, deployment, monitoring, drift detection, governance, and incident response — analogous to DevOps for traditional software.
Source & further reading
Primary source: Kwon et al. — "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM, SOSP) (2023).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/en/glossary/inference-server/.