AI Red-Teaming
AI red-teaming is the practice of systematically probing an AI system — using adversarial prompts, jailbreak techniques, and edge-case inputs — to surface harmful, unsafe, biased, or policy-violating behaviors before and after deployment.
AIDOLS Research Team · Editorial standards
Full definition
Red-teaming combines manual expert testing, automated adversarial generation, and structured rubrics covering categories such as CBRN risk, cyber misuse, child-safety violations, bias, privacy leakage, and self-harm. NIST's Generative AI Profile (NIST AI 600-1, July 2024), a voluntary companion to the AI RMF, recommends red-teaming generative AI systems, and the 2023 White House AI Executive Order (EO 14110), which required developers of dual-use foundation models to report red-team test results, was revoked on 20 January 2025. AIDOLS includes red-team review in every governance engagement.
Why it matters
Internal QA finds the failures developers expect. Red-teaming finds the failures developers did not expect — which are the only ones that matter for regulatory and reputational risk. Companies without a red-team practice are effectively running an open beta on their users.
Example
A foundation model lab runs a 60-person red-team exercise across 8 risk categories pre-launch and discovers a previously unknown method for extracting training data verbatim from the model.
Next steps
Related terms
- JailbreakA jailbreak is an adversarial prompt or technique that bypasses an AI model's safety guardrails to elicit content or behaviors the model was trained to refuse — for example, hazardous instructions, restricted personal data, or off-policy assertions.
- Prompt InjectionPrompt injection is an attack in which adversarial text — placed directly in user input or hidden inside content the model retrieves — overrides developer system instructions and hijacks the model's behavior, exfiltrating data or causing unauthorized actions.
- AI AuditAn AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.
- Model EvaluationModel evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.
More governance & risk terms
- AI GuardrailsAI guardrails are the layered controls — input filters, output classifiers, policy engines, schema validation, and structured generation — that constrain an AI system to safe, on-policy, and on-task behavior.
- AI Bill of Materials (AIBOM)An AI Bill of Materials (AIBOM) is a structured, machine-readable inventory of every component used in an AI system — base models, fine-tuning datasets, third-party APIs, prompts, vector indexes, libraries, and licenses — extending the SBOM concept to AI supply chains.
- Model CardA model card is a short, structured document that describes an AI model's intended use, performance, limitations, training data, evaluation results across demographic and operational subgroups, and known failure modes — the AI equivalent of a nutrition label.
Source & further reading
Primary source: NIST — "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" (NIST AI 600-1) (2024).
Citation policy: this entry is part of the AIDOLS AI Implementation Glossary and may be quoted for research, journalism, and education with attribution to aidolsgroup.com/en/glossary/red-teaming/.