GPU (in AI Context)
A GPU (Graphics Processing Unit) is a massively parallel processor that, for AI workloads, executes the matrix multiplications at the heart of neural networks many times faster than a CPU — and the dominant hardware for both training and inference of modern AI models.
AIDOLS Research Team · Editorial standards
Full definition
NVIDIA's Blackwell and Blackwell Ultra GPUs, and the Vera Rubin platform it began ramping into full production in 2026, are the frontier training hardware; Hopper-generation H100 and H200 GPUs remain widely deployed, and AMD's Instinct GPUs are the main alternative. Beyond raw FLOPs, the strategic differentiators are memory bandwidth (HBM3e/HBM4), interconnect (NVLink, InfiniBand), and software stack (CUDA, cuDNN, ROCm). NVIDIA's lead is as much a software-ecosystem moat as a silicon advantage.
Why it matters
GPUs are now a strategic supply-chain item, not a commodity. Multi-quarter waitlists for frontier GPUs influence which companies can train large models and at what cost. Cloud GPU pricing, on-prem GPU clusters, and reserved-capacity contracts are now CFO-level decisions.
Example
Meta's Llama 3.1 405B was trained on more than 16,000 NVIDIA H100 GPUs over multiple months — the kind of compute concentration that, until 2022, only the largest hyperscalers commanded.
Next steps
Related terms
- InferenceInference is the process of running a trained AI model on new, unseen inputs to produce predictions, classifications, or generated content — the part that runs every time a user interacts with the system.
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- TPUA TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for tensor operations — the dense matrix multiplications at the core of neural networks — and used in production to train and serve Google Search, Translate, YouTube, and Gemini.
- Inference ServerAn inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
- Edge AIEdge AI is the practice of running AI models on local devices — smartphones, vehicles, sensors, factory equipment, AR/VR headsets — instead of the cloud, in order to deliver lower latency, stronger privacy, and continued operation when offline.
Source & further reading
Primary source: NVIDIA — "H100 Tensor Core GPU Architecture" white paper (2022).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/en/glossary/gpu/.