AI Bias

Full definition

Bias enters at every stage: which problem is chosen, what data is collected, who labels it, what target is optimized, and how the system is deployed. NIST's AI RMF (2023) and the EU AI Act treat bias as a managed risk requiring documentation, testing, and mitigation throughout the lifecycle. Detection uses disparity metrics (e.g., demographic parity, equalized odds), but no single metric captures all dimensions of fairness.

Why it matters

AI bias is now a regulatory, reputational, and litigation risk. The EU AI Act requires providers of high-risk systems to examine their training, validation and testing data for possible biases and to detect, prevent and mitigate them (Article 10), and NYC Local Law 144 requires an independent bias audit of automated employment decision tools; Canada's proposed AIDA died when Parliament was prorogued in January 2025. Organizations without a documented bias evaluation process for their deployed models are exposed.

Example

In 2018 Reuters reported that Amazon had scrapped an experimental hiring AI (the team was disbanded by early 2017) that had learned to penalize résumés containing the word "women's" — a bias picked up from ten years of résumés submitted to the company, most of them from men.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• AI Audit
  An AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.
• AI Governance
  AI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.
• Explainability (XAI)
  Explainability (XAI) is the property of an AI system whose decisions can be understood by humans — through model-level documentation, prediction-level attributions, and counterfactual explanations — and a regulatory expectation in finance, healthcare, hiring, and other high-stakes domains.
• Training Data
  Training data is the curated dataset used to teach a machine-learning model — every pattern the model can recognize, every bias it inherits, and every limit to its accuracy ultimately traces back to this data.
• EU AI Act
  The EU AI Act (Regulation (EU) 2024/1689) is the European Union's comprehensive, risk-tiered regulation of AI systems, the world's first horizontal AI law, with obligations phasing in from February 2025: general-purpose AI model rules from 2 August 2025 (Commission fines from 2 August 2026; models already on the market have until 2 August 2027) and, after the Digital Omnibus (Regulation (EU) 2026/1744), high-risk rules from 2 December 2027 (Annex III) and 2 August 2028 (Annex I).