AI Red-Teaming

Full definition

Red-teaming combines manual expert testing, automated adversarial generation, and structured rubrics covering categories such as CBRN risk, cyber misuse, child-safety violations, bias, privacy leakage, and self-harm. NIST's AI RMF and the 2023 White House AI Executive Order both elevate red-teaming as a required practice for frontier systems. AIDOLS includes red-team review in every governance engagement on /governance/.

Why it matters

Internal QA finds the failures developers expect. Red-teaming finds the failures developers did not expect — which are the only ones that matter for regulatory and reputational risk. Companies without a red-team practice are effectively running an open beta on their users.

Example

A foundation model lab runs a 60-person red-team exercise across 8 risk categories pre-launch and discovers a previously unknown method for extracting training data verbatim from the model.

Related terms

• Jailbreak
  A jailbreak is an adversarial prompt or technique that bypasses an AI model's safety guardrails to elicit content or behaviors the model was trained to refuse — for example, hazardous instructions, restricted personal data, or off-policy assertions.
• Prompt Injection
  Prompt injection is an attack in which adversarial text — placed directly in user input or hidden inside content the model retrieves — overrides developer system instructions and hijacks the model's behavior, exfiltrating data or causing unauthorized actions.
• AI Audit
  An AI audit is a structured, evidence-based examination of an AI system or AI program against defined criteria — covering training data, model, deployment context, monitoring, and governance — performed by an internal team, an external firm, or a regulator.
• Model Evaluation
  Model evaluation is the systematic measurement of an AI model's performance, safety, and behavior across representative tasks — using fixed benchmarks, golden datasets, human ratings, and LLM-as-judge methods — both before and after deployment.