Quantization
Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.
Full definition
Modern LLMs are typically trained in BF16 (16-bit) and quantized post-training to INT8 or INT4 for serving. Methods include GPTQ, AWQ, SmoothQuant, GGUF (llama.cpp), and FP8 native (Hopper, Blackwell). 4-bit quantization typically reduces GPU memory 4× and increases throughput 2-3×, putting 70B-class models on a single GPU and 8B models on a phone.
Why it matters
Quantization is one of the highest-impact cost optimizations available. A team that moves from FP16 to INT4 typically cuts hardware cost 3-4× with negligible quality loss, instantly improving the unit economics of every AI feature shipped.
Example
Llama 3 70B in FP16 needs 140GB of GPU memory (two H100s); the same model in 4-bit GGUF needs 40GB and runs on a single 48GB GPU at 70%+ of original quality on most enterprise tasks.
Related terms
- Inference CostInference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
- Small Language Model (SLM)A Small Language Model (SLM) is a compact language model — typically under 10B parameters — designed to run cheaply, on-device, or in latency-sensitive workflows where a frontier LLM would be overkill or too expensive.
- DistillationKnowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
- Edge AIEdge AI is the practice of running AI models on local devices — smartphones, vehicles, sensors, factory equipment, AR/VR headsets — instead of the cloud, in order to deliver lower latency, stronger privacy, and continued operation when offline.
- Inference ServerAn inference server is the runtime system that hosts trained AI models behind an API, handling request routing, dynamic batching, KV-cache management, scheduling across GPUs, and hardware acceleration — the layer that turns a model file into a production AI endpoint.
Source & further reading
Primary source: Frantar et al. — "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (2023).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/it/glossary/quantization/.