LoRA (Low-Rank Adaptation)

Full definition

Introduced by Hu et al. (Microsoft, 2021), LoRA decomposes weight updates ΔW into the product of two low-rank matrices A·B, training only A and B. A 7B-parameter model can be specialized with as few as 4-16M trainable parameters. Variants include QLoRA (quantize the base model to 4-bit, train LoRA in fp16) which made fine-tuning 65B models on a single consumer GPU possible. LoRA adapters are also hot-swappable at serving time, enabling per-tenant customization.

Why it matters

LoRA collapsed the cost of model specialization from "data center project" to "afternoon experiment." Any team that thinks fine-tuning is too expensive to evaluate is using outdated mental models. AIDOLS recommends LoRA evaluation in nearly every domain-specific GenAI engagement.

Example

A legal-tech firm trains 14 jurisdiction-specific LoRA adapters on a 13B base model overnight on 2 H100s; total training cost <$300, served via adapter swapping at <2ms overhead per request.

Related terms

• Fine-tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Training Cost
  Training cost is the total cost — GPU/TPU compute, energy, data acquisition, and labor — required to train a machine-learning model from scratch or to fine-tune a pretrained one to a target capability or domain.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0-2% quality loss.