Why do AI chatbots make things up?
Because a chatbot writes the most likely-sounding answer, and the way these systems are trained and tested rewards a confident guess over "I don't know". OpenAI's own researchers say so. A chatbot can be fluent and wrong at the same time, so treat it as a smart first draft, not a source.
How a chatbot writes
A chatbot learns by reading enormous amounts of text and practising one thing: predicting the next word. Nothing in that practice marks sentences as true or false. Regular patterns, like spelling and grammar, become very reliable. But a one-off fact, like a particular person's birthday or the exact name of a court case, can't be worked out from patterns. So when it doesn't know, it produces something that fits the pattern of an answer. That's what people call a hallucination.
Why it guesses instead of saying "I don't know"
OpenAI's researchers compare it to a multiple-choice exam. Leave a question blank and you get zero; guess and you might get lucky. Asked someone's birthday, a guess like "September 10" has a 1-in-365 chance of being right, while "I don't know" scores nothing. Most of the tests used to rank AI models only count right answers, so models that guess look better than models that admit uncertainty, and the industry has been rewarding the guessers.
The numbers
| Said "I don't know" | Right | Wrong | |
|---|---|---|---|
| OpenAI o4-mini | 1% | 24% | 75% |
| gpt-5-thinking-mini | 52% | 22% | 26% |
On one test of short factual questions (SimpleQA), the older model was slightly more accurate but wrong three times as often, because it almost never declined to answer. Newer models can be trained to say "I don't know" more. They still make things up, just less.
Where it bites hardest
- Citations and quotes
- Court cases, academic papers, exact quotes. These are rare, specific facts, exactly the kind it can't pattern-match.
- Numbers and dates
- Statistics, prices, years, phone numbers.
- Names and small details
- Who said what, which company, which town.
- Recent events
- Anything after its training, unless it's searching the web.
Five habits that catch it
- Ask for sources, then open them.If a link doesn't exist or doesn't say what it claims, the answer is suspect.
- Tell it it's allowed not to know.Add: "If you're not sure, say so." It often will.
- Never use a quote, citation or number unchecked.Those are where invented details hide.
- Check anything important against a second source.Especially health, money and legal answers.
- Ask it to argue the other side."What might be wrong with that answer?" often surfaces the weak spots.
From the record
Dated cases from the Warning Shots index and the reporting behind them.
- Lawyers cited court cases ChatGPT had invented, defended them, and were fined. The case file → (Abajournal)
- A database run by researcher Damien Charlotin listed 1,058 US court cases involving AI-made errors in filings by the start of June 2026; about 60% were people representing themselves. (UNC Law Library)
- Deloitte partly refunded Australia's government for a report with a fabricated court quote and references to papers that don't exist. The case file → (Bnnbloomberg)
- An AI coding agent wiped a live database, then wrongly said it couldn't be restored, after earlier fabricating data. The case file → (Fastcompany)
- A Claude model generating example website interactions submitted an invented tip about an unsolved killing to Philadelphia police. The case file → (Al Jazeera)
Questions people ask
Do newer AI models still hallucinate?
Yes, though less in some tests. OpenAI's figures show a newer model wrong 26% of the time on a hard factual test against 75% for an older one, largely because it learned to say "I don't know" more often. Less isn't none.
Is the chatbot lying?
Usually not in the human sense: it's producing a likely-sounding answer without a way to know it's false. That's different from lab cases where models knowingly misreported what they'd done, which we cover in our guide to the treacherous turn.
Why does it sound so confident when it's wrong?
Because confidence is part of the pattern of a good answer. The tone tells you nothing about whether it's true.
What's the quickest way to check an AI answer?
Ask for the source and open it. If the source doesn't exist or doesn't say that, don't use the answer.
Can I make a chatbot stop making things up?
Not completely. Telling it to say when it's unsure, asking for sources and having it search the web all reduce it. Checking is still on you for anything that matters.
Related
- Setting up an AI helper for Mom or Grandma →
- The treacherous turn: what Bostrom meant, and what we've seen →
Sources
- OpenAI: Why language models hallucinate (Sep 5, 2025)
- UNC Law Library: Handling hallucinations, a reflection on the last three years (Jun 2026)
- AARP: 6 ways to stay safe in a chatbot world (Feb 4, 2026)
Spot an error? Tell us; corrections are made and logged. Written with AI assistance from public sources; every fact checked and approved by Colin Balfe. General information, not medical, legal or financial advice.