Model Pruning

Full definition

Pruning splits into unstructured (zero out individual weights) and structured (remove entire heads, neurons, or layers). The "lottery ticket hypothesis" (Frankle & Carbin, MIT, 2019) showed that small sub-networks within larger trained networks can match full-network accuracy when retrained alone. Pruning often combines with quantization and distillation for compounding compression. Hardware support determines real-world speedup — unstructured sparsity needs sparse-aware kernels, while NVIDIA's 2:4 pattern (two of every four weights zero) is a fine-grained structured sparsity that Sparse Tensor Cores on Ampere and later GPUs accelerate.

Why it matters

Pruning is a real inference-cost lever, but sparsity alone usually brings modest LLM speedups: Neural Magic's 2:4-sparse Llama 3.1 8B kept 98.4% of baseline accuracy with up to 30% higher throughput. Combined with quantization and distillation, the gains compound. For high-volume workloads, the engineering investment pays back within months.

Example

A search company prunes 35% of attention heads in its query-rewriting model and quantizes to int8, cutting inference cost 4.1x with measured zero precision regression on the production eval set.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Model Serving
  Model serving is the runtime infrastructure that hosts a trained model and exposes it as an API for low-latency online inference at scale, handling batching, autoscaling, GPU sharing, versioning, and routing.

More training & optimization terms

• Supervised Fine-Tuning (SFT)
  Supervised Fine-Tuning (SFT) is the stage of LLM post-training where a pretrained model is fine-tuned on curated input-output pairs, typically instruction-response demonstrations written by humans.
• Data Labeling
  Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.
• Active Learning
  Active learning is a labeling strategy in which the model selects which unlabeled examples a human should label next — typically the cases the current model is least certain about — focusing labeling effort on the most informative examples.