Explainability (XAI)
Explainability (XAI) is the property of an AI system whose decisions can be understood by humans — through model-level documentation, prediction-level attributions, and counterfactual explanations — and a regulatory expectation in finance, healthcare, hiring, and other high-stakes domains.
Full definition
Methods include intrinsically interpretable models (linear, tree-based, GAMs), post-hoc attributions (SHAP, LIME, integrated gradients), and counterfactuals ("what would change to flip this decision?"). For LLMs, explainability is mostly behavioral (showing citations, exposing chain-of-thought, providing model cards) rather than mechanistic. Mechanistic interpretability is an active research area but not yet a deployable governance tool.
Why it matters
Explainability is required by GDPR Article 22, the EU AI Act for high-risk systems, NYC Local Law 144 for hiring, and most financial regulators (e.g., the US OCC SR 11-7 model risk guidance). It is also the foundation of trust: customers and employees adopt AI systems whose decisions they can understand.
Example
A bank's credit-decision model uses SHAP values to show, for every adverse-action notice, the top 4 factors that drove the decline — meeting both regulatory disclosure requirements and improving customer recourse.
Related terms
- AI GovernanceAI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.
- AI BiasAI bias is systematic, unfair difference in an AI system's outputs across demographic, geographic, or other groups — usually caused by biased training data, biased labels, or biased problem framing rather than the algorithm itself.
- AI AuditAn AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.
- Algorithmic AccountabilityAlgorithmic accountability is the principle that a specific person, role, or organization is identifiable and answerable for the design, deployment, outcomes, and harms of an automated decision system — and that the mechanisms to enforce that answerability exist.
- EU AI ActThe EU AI Act (Regulation (EU) 2024/1689) is the European Union's comprehensive, risk-tiered regulation of AI systems, the world's first horizontal AI law, with obligations phasing in from February 2025 and full general-purpose AI rules applying from August 2025.
Source & further reading
Primary source: Lundberg, Lee — "A Unified Approach to Interpreting Model Predictions" (SHAP, NeurIPS) (2017).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/it/glossary/explainability/.