Eval Harness

Full definition

A harness handles dataset loading, prompt templating, batched API calls, retries, scoring (string match, regex, LLM-as-judge, programmatic), and result storage. EleutherAI's lm-evaluation-harness is the open-source reference; commercial alternatives include Braintrust, Langfuse, and Weights & Biases. A good harness turns evaluation from a one-off notebook into a CI step that every model change must pass.

Why it matters

Without a harness, evaluation is irreproducible and slow — and therefore skipped under deadline pressure. A harness wired into CI/CD makes "did this regress?" a button click and lets engineering velocity compound rather than degrade.

Example

A team adds their golden dataset to lm-evaluation-harness and gates every prompt change behind a CI run; within a quarter, accidental regressions drop from ~1 per release to zero.

Related terms

• Model Evaluation
  Model evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.
• Golden Dataset
  A golden dataset is a curated, expert-validated set of inputs and reference outputs that serves as the canonical benchmark for evaluating an AI model on a specific task — the team's "ground truth" for is-this-shippable decisions.
• MLOps
  MLOps is the discipline of operating machine-learning and AI systems reliably in production — covering data pipelines, model training, deployment, monitoring, drift detection, governance, and incident response — analogous to DevOps for traditional software.
• A/B Testing for ML
  A/B testing for ML is the practice of randomly splitting live traffic between a control model and a candidate model to measure the candidate's causal impact on real business metrics — revenue, conversion, retention, defect rate.