Jailbreak

Full definition

Jailbreak families include role-play wrappers (DAN, "grandma exploit"), encoding/translation attacks, many-shot jailbreaks, gradient-based suffix attacks (GCG), and multi-turn social engineering. Anthropic's 2024 paper on many-shot jailbreaking showed that long context windows themselves create new attack surface. Defense layers include input/output classifiers, constitutional AI training, and human review of high-risk categories.

Why it matters

Any LLM exposed to untrusted users will face jailbreak attempts within hours of launch. Treat jailbreak as an inevitable failure mode and design controls accordingly: layered guardrails, content classifiers, and an incident-response runbook for novel attacks.

Example

A customer-support bot ships without an output classifier; researchers publish a jailbreak that extracts the system prompt within 48 hours, embarrassing the vendor and triggering a multi-week patch cycle.

Related terms

• Prompt Injection
  Prompt injection is an attack in which adversarial text — placed directly in user input or hidden inside content the model retrieves — overrides developer system instructions and hijacks the model's behavior, exfiltrating data or causing unauthorized actions.
• AI Red-Teaming
  AI red-teaming is the practice of systematically probing an AI system — using adversarial prompts, jailbreak techniques, and edge-case inputs — to surface harmful, unsafe, biased, or policy-violating behaviors before and after deployment.
• AI Guardrails
  AI guardrails are the layered controls — input filters, output classifiers, policy engines, schema validation, and structured generation — that constrain an AI system to safe, on-policy, and on-task behavior.
• AI Governance
  AI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.