The basics · Layer 4
Hallucination
A false but plausible answer, stated with confidence: a source, figure or fact that the model makes up.
A way to picture itThink of the candidate in an oral exam who would rather invent a polished answer than admit they don’t know: the tone is right, the content is not.
A lawyer asks a general-purpose assistant for recent case law on a non-compete clause. The answer cites three rulings, with dates and case numbers; one of them does not exist. She only finds out by looking the ruling up in an official database. Since then, the team’s rule has been simple: no reference produced by AI is reused until it has been opened at its source.
If the answer concerns a person, made-up information is inaccurate personal data: the accuracy principle applies (EU · GDPR, art. 5). A decision with legal or similarly significant effects on a person cannot rest solely on automated processing, save for regulated exceptions (EU · GDPR, art. 22); for high-risk uses, the AI Act requires effective human oversight (EU · AI Act, art. 14). The CNIL points out that generative models are not knowledge bases (France · CNIL, 18 July 2024): checking is your organisation’s job, not the model’s.
“How do you measure the rate of wrong answers on our own questions, and what does the tool do when it finds no source: answer anyway, or say so?”
Because it is built to produce a plausible continuation, and its training has long rewarded a guessed answer more than abstaining. Providers are working to reduce this bias, but it has not gone away.
Yes. In June 2023, a federal judge in New York sanctioned two lawyers and their firm over a brief citing six rulings invented by ChatGPT (United States · US District Court for the Southern District of New York). Similar cases have since been reported in many jurisdictions.
Supply the reference documents, for instance through RAG, require verifiable citations and explicitly allow the answer “I don’t know”. Then measure the error rate on a set of questions whose answers you already know.
Educational explainer, not legal advice. Up to date as of 29 October 2026.