When AI Gets It Wrong - Understanding Why AI Models Hallucinate

We've all encountered hallucinations when we use various AI tools and LLM's. Some may be relatively harmless, but others - especially when we rely on AI for important facts - can create major problems with big implications.

For example, my wife Lori is an accomplished appellate lawyer. In a recent set of court filings, attorneys for the other side in a case she was handling filed documents that were apparently created by – or at least augmented by - AI. The opposition filing included several references to court cases and rulings that Lori couldn't find in any legal records. In fact, it turns out these cases never existed! The general consensus was that the AI the opposing lawyers were using had created fictitious legal authority that supported the case they were trying to make. The opponents lost. That's one win for the good guys (my wife!), and a cautionary tale for over-reliance on AI. In fact, Lori says using general AI tools such as ChatGPT for legal research frequently results in incorrect citations and nonexistent precedents; but that AI models specifically trained for the legal sector and with access to extensive legal records - such as CoCounsel which is integrated with WestLaw - are much more accurate.

This example came to mind as I reviewed the introduction to a new research paper on AI and hallucinations, released this week in September 2025. It was written by a team who mostly work at OpenAI:*

"Language models are known to produce overconfident, plausible falsehoods, which diminish their utility. This error mode is known as “hallucination,” though it differs fundamentally from the human perceptual experience. Despite significant progress, hallucinations continue to plague the field, and are still present in the latest models." (OpenAI, 2025)

Here's the abstract of the 35-page technical article:

"Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. Such “hallucinations” persist even in state-of-the-art systems and undermine trust. We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and we analyze the statistical causes of hallucinations in the modern training pipeline. Hallucinations need not be mysterious—they originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, then hallucinations in pretrained language models will arise through natural statistical pressures. We then argue that hallucinations persist due to the way most evaluations are graded—language models are optimized to be good test-takers, and guessing when uncertain improves test performance. This “epidemic” of penalizing uncertain responses can only be addressed through a socio-technical mitigation: modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations. This change may steer the field toward more trustworthy AI systems."

What I find particularly interesting is the notion that many LLMs hallucinate because they are optimized and trained to be good test takers. During their research, the authors asked a state-of-the-art open-source language model on multiple occasions what the birthday was of one of the authors. They went on to tell the model it should only reply if it knew the answer. In each instance, the LLM offered a guess (which was incorrect every time). The presumed rationale for the model offering a guess is that the odds of being correct one-in-365 times are better than no chance at all.

Until AI model developers build in affected guardrails against this type of LLM behavior, we will likely continue to see hallucinations in AI results. And the models will probably get better and better at making them seem real, even as they continue to produce hallucinations.

Borrowing a line from the old Latin legal principal: caveat emptor!

If you would like to see the full 35-page technical paper, you can find it here.

*The authors are Adam Tauman Kalai (OpenAI); Ofir Nachum (OpenAI); Santosh Vempala (Georgia Tech); Edwin Zhang (Open AI).