Why does AI make things up so confidently?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, a politely tuned chatbot explained why some experts say you should eat socks after meditating. So why does AI make things up so confidently? In May 2025, researchers asked three chatbots, ChatGPT, DeepSeek and Llama, for the title of one computer scientist's PhD thesis ¹. Each gave a different title and year, and none was right ¹. The computer scientist was Adam Kalai, the paper's own first author ¹. In a letter-counting test, his team found DeepSeek, Meta AI and Anthropic's Claude all miscounted too ¹.

A knee and a serving cart 0:44

In 2022, a man named Roberto Mata sued the airline Avianca, saying a metal serving cart had struck his knee on a flight to New York ². On the first of March 2023, his lawyers filed a response to the airline's bid to have the case dismissed ². It cited earlier court decisions, among them Varghese versus China Southern Airlines ². The airline's lawyers couldn't find most of them ². Varghese didn't exist, and nor did five others ².

Is it a real case? 1:20

The lawyer who did the research, Steven Schwartz, said he had assumed ChatGPT was like a super search engine ². When doubts arose, he asked it whether Varghese was a real case, and it said the case did indeed exist ². In June 2023, Judge Kevin Castel found the lawyers had kept standing by the fake cases after the court questioned them ². He fined Mr Schwartz, his colleague Peter LoDuca and their firm five thousand US dollars, and had them write to every real judge named as the author of a fake decision ². He also wrote that there's nothing inherently improper about using a reliable AI tool; the lawyer's job is to check ².

Real parts, invented case 2:05

The fake named three real federal judges as its authors ². Its docket number belonged to a different, real case ². Its reference into the law reports pointed at another real case again ². The judge called its legal analysis gibberish ². Every piece had the right shape, which made it believable.

What hallucination means 2:28

Researchers call this hallucination ³. New South Wales' Supreme Court defines it plainly: answers that seem plausible, authoritative and coherent, but are in fact inaccurate or fictitious ⁴. Researchers split it into two kinds ³. An intrinsic hallucination contradicts what the model was handed, like a summary that turns 2019 into 2021 ³. An extrinsic one adds claims that can't be checked against it, and may be false about the world ^3,1.

A word that began as praise 3:01

The word has an odd history. One of its earliest uses in computing, in the year 2000, meant something useful ³. Simon Baker and Takeo Kanade, at Carnegie Mellon University, sharpened tiny, blurry faces, like those in security-camera footage ⁵. They wrote that the additional pixels were, in effect, hallucinated ⁵. For blurry faces, plausible invention was the goal ³. For facts, it's the problem, and a model doesn't perceive anything, so the word is only borrowed ^3,1.

Likely isn't true 3:37

Remember episode six: a language model scores which token is likely to come next. Nothing in that score checks whether it's true ¹. Here's the same kind of toy as then, counting which word follows which, this time in four true sentences. Canberra is the capital of Australia. Sydney is its biggest city. Wellington is New Zealand's capital, and Auckland its biggest city. Start with Sydney, and it can finish four ways, each with exactly the same chance, twenty-five per cent, but only one of them is true.

Rare facts 4:15

A real model knows Canberra is the capital, because that's written everywhere ¹. The trouble is facts that turn up rarely, like most people's birthdays ¹. In 2025, Kalai and colleagues, mostly at OpenAI, argued why, with a proof for a simplified case ¹. Count the training examples whose fact appears only once ¹. A freshly pretrained model should get roughly at least that share wrong ¹. Here, four of twenty mentions are one-offs, so expect it to get at least about one in five wrong ¹. Unable to tell a rare true fact from a plausible false one, the model guesses ¹.

Built to sit exams 4:57

Then the guessing gets rewarded ¹. In 2025, Kalai's team checked ten popular tests of AI: nine mark answers only right or wrong, and give nothing for, I don't know ¹. Picture a model that's thirty per cent sure. If it guesses, it averages nought point three of a mark. If it admits it doesn't know, it gets zero, so guessing always wins ¹. Kalai's team suggests telling the model up front that wrong answers cost marks ¹. If a wrong answer costs one mark, the same guess averages minus nought point four, and, I don't know, is the better answer ¹. Now guessing only pays above fifty per cent sure ¹.

Two models, one quiz 5:41

OpenAI's own numbers show the trade-off ⁶. In August 2025, it reported two of its models on a quiz of short factual questions picked because earlier models got them wrong, with no web search ^6,7. The older o4-mini got twenty-four per cent right and seventy-five per cent wrong, so it almost never declined ⁶. The newer gpt-5-thinking-mini got twenty-two per cent right and twenty-six per cent wrong, because it declined about half the time ⁶. Count only right answers, and the older model wins; charge a mark for each wrong one, and it loses badly ^1,6. That's one company testing its own models, on one quiz.

Why it sounds so sure 6:25

So why does a wrong answer sound so certain? Kalai's team notes that bluffs tend to be confident and specific: September the thirtieth, not sometime in autumn ¹. In March 2023, OpenAI reported that its pretrained GPT-4 was well calibrated: its confidence in an answer generally matched how often it was right ⁸. After the kind of tuning from last episode, its calibration error was about ten times bigger ⁸. And in 2024, researchers at Stanford and Yale found chatbots struggle to predict their own hallucinations about the law ⁹.

Part of the website 7:04

In November 2022, after their grandmother died, Jake Moffatt asked a chatbot on Air Canada's website about bereavement fares ¹⁰. It said they could claim the discount within ninety days, even after travelling ¹⁰. The airline's own policy page said the opposite ¹⁰. Air Canada argued it wasn't liable for its chatbot ¹⁰. In February 2024, a Canadian tribunal called that a remarkable submission: the chatbot is part of the website, and the airline had to pay ¹⁰. The airline never said what kind of chatbot it was ¹⁰.

Look it up first 7:42

A common fix is to have the model look things up first ¹¹. In 2020, researchers at Facebook AI Research and University College London called this retrieval-augmented generation ¹¹. A search step fetches relevant documents, and the model answers from them, so you can see what it drew on and check ¹¹. In their tests, answers were more factual than from memory alone ¹¹.

A real case that doesn't say that 8:09

But retrieval is a partial fix ¹². Some legal research companies had advertised that this approach avoided hallucinations, or that its citations were hallucination-free ¹². In May 2024, researchers at Stanford and Yale tested tools from LexisNexis and Thomson Reuters that work this way ¹². They hallucinated between seventeen and thirty-three per cent of the time ¹². That beat a general chatbot, but it isn't a cure ¹². A sneaky kind was the misgrounded answer: a real case, cited for something it doesn't say ¹².

So how often? 8:47

So how often does AI make things up? It depends heavily on the test ^1,13. Vectara, a search company, has models summarise documents using only what's in them, and Vectara's own model checks for additions ¹³. As of September 2026, its hundred and eight models, including ones from OpenAI, Google, Anthropic, Meta, Mistral and DeepSeek, ranged from about two to twenty-four per cent ¹³. But in a 2024 study, chatbots asked direct questions about real US court cases got fifty-eight to eighty-eight per cent wrong ⁹. There's no single hallucination rate, only rates for particular tests, on particular dates.

Courts are counting 9:34

In August 2025, in a murder case in Victoria's Supreme Court, a senior defence barrister apologised after his team's submissions included fictitious quotes and cases that didn't exist ¹⁴. The team had checked the first citations, and wrongly assumed the rest were right ¹⁴. As of the tenth of October 2026, a database kept by the legal researcher Damien Charlotin listed two thousand, one hundred and eighty-four court decisions involving hallucinated material, a hundred and twelve of them in Australia ¹⁵.

How to check 10:10

So how do you check what an AI tells you? Be most suspicious of specifics: names, dates, numbers, quotes and references ^1,12. Since February 2025, when lawyers in New South Wales' Supreme Court use AI to help write submissions, they must confirm that every citation exists, is accurate, and is relevant, and not check that solely with an AI tool ⁴. ChatGPT vouched for Varghese, remember ². Open the source and find the sentence. One more warning sign is to ask again: when a model is guessing, its details tend to change from one try to the next ¹⁶.

Can it be fixed? 10:53

Will hallucination ever go away? Researchers disagree ^1,17. In 2024, Ziwei Xu and colleagues argued it can't be eliminated, because, in a formal sense, no model can learn everything it might be asked ¹⁷. Kalai's team argues it's inevitable only for freshly pretrained models: a model can be built to say I don't know, and tests could reward it for that ¹. Most of that team works for a company that makes these models, and the debate is still running ¹.

Made up on purpose 11:26

So a chatbot makes things up because it writes likely text, rare facts leave it guessing, and most tests reward the guess ¹. Looking things up and checking help, but they don't cure it ^11,12. Pictures are another story: an image model invents every pixel on purpose. How does AI make a picture from a sentence?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang, "Why Language Models Hallucinate",… — arxiv.org
2. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions, US… — storage.courtlistener.com
3. Z. Ji et al., "Survey of Hallucination in Natural Language Generation", ACM Computing… — arxiv.org
4. Supreme Court of New South Wales, Practice Note SC Gen 23, "Use of Generative… — supremecourt.nsw.gov.au
5. S. Baker, T. Kanade, "Hallucinating Faces", Proc. 4th IEEE International Conference on… — publications.ri.cmu.edu
6. OpenAI, "GPT-5 System Card" (13 August 2025), §3.7 Table 8 (SimpleQA) — cdn.openai.com
7. J. Wei et al. (OpenAI), "Measuring short-form factuality in large language models"… — arxiv.org
8. OpenAI, "GPT-4 Technical Report", arXiv:2303.08774 (March 2023), §5 and Figure 8;… — arxiv.org
9. M. Dahl, V. Magesh, M. Suzgun, D. E. Ho, "Large Legal Fictions: Profiling Legal… — arxiv.org
10. Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal (14… — decisions.civilresolutionbc.ca
11. P. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks",… — arxiv.org
12. V. Magesh, F. Surani, M. Dahl, M. Suzgun, C. D. Manning, D. E. Ho,… — arxiv.org
13. Vectara, "Hallucination Leaderboard" (GitHub README, last updated 22 September 2026;… — github.com
14. The Associated Press, "Australian lawyer sorry for AI errors in murder case, including… — nbcnews.com
15. D. Charlotin, "AI Hallucination Cases" database (last updated 10 October 2026; read 15… — damiencharlotin.com
16. P. Manakul, A. Liusie, M. J. F. Gales, "SelfCheckGPT: Zero-Resource Black-Box… — arxiv.org
17. Z. Xu, S. Jain, M. Kankanhalli, "Hallucination is Inevitable: An Innate Limitation of… — arxiv.org

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.