Hallucination

Full definition

Hallucinations arise because LLMs are trained to produce probable text, not true text. They have no built-in fact-check, no internal "I don't know" signal, and they will often produce confident, well-formed answers about subjects they have no reliable information on. Mitigations include retrieval-augmented generation (RAG), grounding with verified sources, fine-tuning on domain data, output validation, and using reasoning models that can flag uncertainty.

Why it matters

Hallucination is one of the main risks that keep AI out of production in regulated industries. RAG plus citation-grounded answers reduces hallucination, by an amount that depends on retrieval quality and the task. Any production AI system serving customers, patients, or regulated decisions needs an explicit hallucination-mitigation strategy and continuous evaluation.

Example

In 2023 a US lawyer submitted a brief citing 6 court cases that ChatGPT had hallucinated. The cases did not exist; the lawyer was sanctioned. The case (Mata v. Avianca) is now a standard cautionary example in legal AI policy.

Next steps

• Get a free workflow map →We map one of your workflows and give you a written verdict: software, a process change or AI. No obligation.
• AI consulting cost guide →Hourly rates and project budgets by firm tier, from independents to the Big Four.

Related terms

• Retrieval-Augmented Generation (RAG)
  Retrieval-Augmented Generation (RAG) is a technique where a language model retrieves relevant documents from an external knowledge base before generating a response, which reduces hallucination and enables citation of sources.
• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Generative AI
  Generative AI is a class of AI systems that produce new content — text, images, code, audio, or video — by learning the distribution of their training data and sampling from it, rather than classifying or predicting from existing inputs.
• Explainability (XAI)
  Explainability (XAI) is the property of an AI system whose decisions can be understood by humans — through model-level documentation, prediction-level attributions, and counterfactual explanations — and a regulatory expectation in finance, healthcare, hiring, and other high-stakes domains.
• AI Governance
  AI governance is the framework of policies, roles, controls, and processes an organization uses to ensure its AI systems are lawful, safe, fair, accountable, and aligned with business intent — across the full lifecycle from problem framing to retirement.

More training & optimization terms

• Quantization
  Quantization is the technique of representing a neural network's weights and activations with fewer bits — 8-bit, 4-bit, or even lower — to dramatically reduce memory footprint, inference cost, and latency, usually with only 0–2% quality loss.
• Distillation
  Knowledge distillation is the practice of training a smaller "student" model to mimic the outputs of a larger, higher-quality "teacher" model — capturing most of the teacher's quality at a fraction of the inference cost.
• Synthetic Data
  Synthetic data is artificially generated training data — produced by AI models or simulators rather than collected from the real world — used to augment, replace, or supplement real datasets while preserving privacy or covering rare cases.