LoRA (Low-Rank Adaptation)

Full definition

Introduced by Hu et al. (Microsoft, 2021), LoRA decomposes weight updates ΔW into the product of two low-rank matrices A·B, training only A and B. A 7B-parameter model can be specialized with as few as 4–16M trainable parameters. Variants include QLoRA (a 4-bit quantized base model with 16-bit LoRA adapters), which made fine-tuning 65B models possible on a single 48GB professional GPU and 33B models on a single 24GB consumer GPU. LoRA adapters are also hot-swappable at serving time, enabling per-tenant customization.

Why it matters

LoRA collapsed the cost of model specialization from "data center project" to "afternoon experiment." Any team that thinks fine-tuning is too expensive to evaluate is using outdated mental models. AIDOLS recommends LoRA evaluation in nearly every domain-specific GenAI engagement.

Example

A legal-tech firm trains 14 jurisdiction-specific LoRA adapters on a 13B base model overnight on 2 H100s; total training cost <$300, served via adapter swapping at <2ms overhead per request.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Fine-Tuning
  Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so it specializes in a particular style, domain, or behavior — without retraining from scratch.
• Training Cost
  Training cost is the total cost — GPU/TPU compute, energy, data acquisition, and labor — required to train a machine-learning model from scratch or to fine-tune a pretrained one to a target capability or domain.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.

More training & optimization terms

• Model Pruning
  Model pruning is the technique of removing unnecessary weights, neurons, or structures from a trained neural network to reduce model size and inference cost — often with minimal quality loss when done carefully.
• Supervised Fine-Tuning (SFT)
  Supervised Fine-Tuning (SFT) is the stage of LLM post-training where a pretrained model is fine-tuned on curated input-output pairs, typically instruction-response demonstrations written by humans.
• Data Labeling
  Data labeling is the process of attaching ground-truth annotations to raw data — text, images, audio — so a supervised model can learn from it, ranging from yes/no classification to structured extraction to multi-turn preference comparisons.