GPU (in AI context)
A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks 10-100× faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
Full definition
NVIDIA's H100, H200, and Blackwell B200 are the de-facto frontier training GPUs; AMD MI300X and Intel Gaudi 3 are credible alternatives. Beyond raw FLOPs, the strategic differentiators are memory bandwidth (HBM3/HBM3e), interconnect (NVLink, InfiniBand), and software stack (CUDA, cuDNN, ROCm). NVIDIA's lead is as much a software-ecosystem moat as a silicon advantage.
Why it matters
GPUs are now a strategic supply-chain item, not a commodity. Multi-quarter waitlists for frontier GPUs influence which companies can train large models and at what cost. Cloud GPU pricing, on-prem GPU clusters, and reserved-capacity contracts are now CFO-level decisions.
Example
Meta's Llama 3.1 405B was trained on more than 16,000 NVIDIA H100 GPUs over multiple months — the kind of compute concentration that, until 2022, only the largest hyperscalers commanded.
Related terms
- InferenceInference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- TPUA TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for tensor operations — the dense matrix multiplications at the core of neural networks — and used in production to train and serve Google Search, Translate, YouTube, and Gemini.
- Inference ServerAn inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
- Edge AIEdge AI is the practice of running AI models on local devices — smartphones, vehicles, sensors, factory equipment, AR/VR headsets — instead of the cloud, in order to deliver lower latency, stronger privacy, and continued operation when offline.
Source & further reading
Primary source: NVIDIA — "H100 Tensor Core GPU Architecture" white paper (2022).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/it/glossary/gpu/.