Model Pruning

Full definition

Pruning splits into unstructured (zero out individual weights) and structured (remove entire heads, neurons, or layers). The "lottery ticket hypothesis" (Frankle & Carbin, MIT, 2019) showed that small sub-networks within larger trained networks can match full-network accuracy when retrained alone. Pruning often combines with quantization and distillation for compounding compression. Hardware support determines real-world speedup — unstructured sparsity needs sparse-aware kernels (e.g., NVIDIA 2:4 sparsity).

Why it matters

Pruning is a top-tier inference-cost lever, often delivering 2-5x speedup at <1% quality regression on production LLMs. For high-volume workloads, the engineering investment pays back within months.

Example

A search company prunes 35% of attention heads in its query-rewriting model and quantizes to int8, cutting inference cost 4.1x with measured zero precision regression on the production eval set.

Related terms

• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.