Hallucination

A hallucination is when a language model generates information that’s factually wrong, presented with the same confidence as a true answer. Names of people who don’t exist, ATO rulings with the wrong number, citations of books that were never written.

Hallucinations are the defining failure mode of modern AI products. They’ve become rarer in frontier models (Claude 4.x, GPT-5, Gemini 2.5) compared to the 2023-2024 era, but they have not been eliminated. They will not be eliminated. Anyone selling you a “no-hallucination” AI product is lying.

Why they happen

LLMs are trained to predict plausible next words, not true ones. For most prompts, plausible and true are correlated tightly enough that the model is right. For some prompts, especially ones involving specific names, numbers, citations, recent events, or niche knowledge, the model produces something plausible that isn’t true.

Reducing hallucinations in practice

1. Use RAG. Ground the model in your specific documents. The model is much less likely to hallucinate about content it’s literally looking at.
2. Use web search tools. Modern Claude can search the web. For factual questions, this beats relying on training knowledge.
3. Ask the model to cite. “For every claim, cite a source.” It improves reliability and makes hallucinations easier to spot.
4. Stay in-domain. Hallucinations are more common when you push the model into very specialised territory it wasn’t trained on much.
5. Use higher-tier models for high-stakes work. Opus 4.7 hallucinates less than Sonnet 4.6, which hallucinates less than Haiku 4.5.
6. Verify externally. For any number, citation, name, or claim that matters, check it.

Common hallucination patterns we see

• Plausible but wrong URLs. AI-generated articles frequently link to URLs that don’t exist. Always test.
• Made-up ATO ruling numbers. “TR 2023/12” sounds real; check the actual ATO database.
• Invented case law. Especially common in legal queries. Multiple Australian lawyers have been sanctioned for filing AI-generated briefs with fabricated case citations.
• Wrong AUD prices. Models trained on older data hallucinate current pricing. Always verify against the vendor’s website.

When hallucinations are acceptable

In some contexts, hallucinations don’t matter or are even useful:

• Creative writing (you wanted invention)
• Brainstorming (low-stakes generation)
• Drafting where a human reviews everything
• Code completion where the model’s wrong guess fails fast

In other contexts, hallucinations are unacceptable:

• Medical, legal, tax, financial advice (regulatory + safety risk)
• Citations in published work
• Customer-facing factual claims
• Anything you can’t or won’t verify

Match your verification effort to your stakes.

Related terms

• Large Language Model
• Retrieval-Augmented Generation