Explainability (XAI)
Explainability (XAI) is the property of an AI system whose decisions can be understood by humans — through model-level documentation, prediction-level attributions, and counterfactual explanations — and a regulatory expectation in finance, healthcare, hiring, and other high-stakes domains.
AIDOLS Research Team · Editorial standards
Full definition
Methods include intrinsically interpretable models (linear, tree-based, GAMs), post-hoc attributions (SHAP, LIME, integrated gradients), and counterfactuals ("what would change to flip this decision?"). For LLMs, explainability is mostly behavioral (showing citations, exposing chain-of-thought, providing model cards) rather than mechanistic. Mechanistic interpretability is an active research area but not yet a deployable governance tool.
Why it matters
Explanation duties appear in the GDPR (Articles 13–15 require meaningful information about the logic of the solely automated decisions covered by Article 22, which adds human review and the right to contest), in the EU AI Act for high-risk systems (Articles 13 and 86) and in US lending rules (Regulation B requires the specific principal reasons for a credit denial); NYC Local Law 144 requires bias audits and candidate notices for hiring tools rather than explanations. It is also the foundation of trust: customers and employees adopt AI systems whose decisions they can understand.
Example
A bank's credit-decision model uses SHAP values to show, for every adverse-action notice, the top 4 factors that drove the decline — meeting both regulatory disclosure requirements and improving customer recourse.
Next steps
Related terms
- AI GovernanceAI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.
- AI BiasAI bias is systematic, unfair difference in an AI system's outputs across demographic, geographic, or other groups — usually caused by biased training data, biased labels, or biased problem framing rather than the algorithm itself.
- AI AuditAn AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.
- Algorithmic AccountabilityAlgorithmic accountability is the principle that a specific person, role, or organization is identifiable and answerable for the design, deployment, outcomes, and harms of an automated decision system — and that the mechanisms to enforce that answerability exist.
- EU AI ActThe EU AI Act (Regulation (EU) 2024/1689) is the European Union's comprehensive, risk-tiered regulation of AI systems, the world's first horizontal AI law, with obligations phasing in from February 2025: general-purpose AI model rules from 2 August 2025 (Commission fines from 2 August 2026; models already on the market have until 2 August 2027) and, after the Digital Omnibus (Regulation (EU) 2026/1744), high-risk rules from 2 December 2027 (Annex III) and 2 August 2028 (Annex I).
Source & further reading
Primary source: Lundberg, Lee — "A Unified Approach to Interpreting Model Predictions" (SHAP, NeurIPS) (2017).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/en/glossary/explainability/.