Guide

LLM hallucinations explained

A hallucination is when a large language model states something false, invented, or unverifiable with the same fluent confidence as a correct answer. The model is not “lying” in a human sense — it has no beliefs and no access to ground truth at inference time. It predicts the next token that best fits the statistical patterns in its training data and your prompt. That objective produces remarkably useful prose, code, and reasoning, but it also fabricates case citations, misquotes APIs, invents historical dates, and rationalizes wrong math. For builders shipping chatbots, copilots, and autonomous agents, hallucinations are the primary reliability risk — often more common than prompt injection and harder to eliminate entirely. This guide explains why hallucinations happen, how to classify and measure them, and which mitigations actually move the needle in production.

What counts as a hallucination

In research and product teams, “hallucination” usually means output that is not supported by the model’s inputs or known facts, delivered without appropriate uncertainty. The term overlaps with confabulation in psychology — filling gaps in memory with plausible fiction — which is a useful mental model.

Not every mistake is a hallucination. If you ask for the capital of France and the model says “Berlin,” that is a factual error. If you paste a contract clause and ask what it means, and the model invents a liability cap that does not appear in the text, that is a grounding failure — the answer is not faithful to the source you provided. If the model cites “Smith et al., Nature 2023” and no such paper exists, that is a citation hallucination. Distinguishing these types matters because the fixes differ: general knowledge gaps need retrieval or refusal; grounding failures need better chunking and citation enforcement; citation fabrications need structured output and verification pipelines.

Hallucinations are especially dangerous in high-stakes domains: medical dosing, legal advice, financial compliance, security configuration, and on-chain transaction construction. A wrong token address or malformed instruction in a Solana program call can drain a wallet — the model will still sound sure.

Why language models hallucinate

Transformers are trained to minimize prediction error on enormous text corpora. They learn what fluent text looks like, not a verified database of facts. At inference, they sample or greedily pick tokens that continue the pattern. There is no built-in step that checks Wikipedia before answering.

Structural causes

Parametric knowledge is incomplete and stale. Training cutoffs, long-tail facts, and niche APIs are underrepresented. The model fills gaps with statistically likely fabrications.
Single forward pass, no deliberation. Unlike a human who can pause and look something up, a vanilla completion runs once unless you add tools or retrieval loops.
RLHF and helpfulness pressure. Models are fine-tuned to be useful and avoid saying “I don’t know” too often. That trades abstention for plausible guesses — a known side effect of alignment training.
Long contexts dilute attention. Even with large context windows, relevant evidence buried mid-prompt may receive weak attention; the model defaults to parametric memory.

Prompt and task triggers

Certain requests spike hallucination rates: obscure proper nouns, compound questions requiring multi-hop lookup, requests for exact quotes or page numbers, and tasks outside the model’s training distribution (e.g., your private schema from 2024). Asking for JSON with fields the model must invent — “list every competitor with revenue” — invites fabrication unless you supply data via RAG.

Common hallucination patterns

Factual and entity errors

Wrong dates, swapped names, incorrect version numbers, misattributed quotes. Models often conflate similar entities (two companies with overlapping names, two blockchains with similar tooling).

Citation and source fabrication

Invented DOIs, journal titles, court cases, or GitHub URLs that 404. The format looks perfect; only verification catches it. This pattern is rampant in academic and legal summarization without retrieval.

Reasoning and math slips

Multi-step arithmetic and symbolic logic fail even when individual steps sound reasonable. Chain-of-thought can help or hurt: longer rationales give more chances to correct — or to confidently derail.

Tool and API hallucinations

Agents call nonexistent functions, pass wrong parameter shapes, or assume API behavior from outdated docs. Tool schemas help, but models still improvise when schemas are ambiguous.

Visual and multimodal confabulation

Vision-language models describe objects not in the image or misread charts. The same next-token objective applies to pixels encoded as tokens.

Measuring hallucinations

You cannot improve what you do not measure. Production teams combine offline evals and online monitoring.

Offline benchmarks

Public sets like TruthfulQA probe tendency to repeat popular misconceptions. Domain-specific golden sets — question, gold answer, supporting documents — are more actionable. Score with exact match, token F1, embedding similarity, or LLM-as-judge with a separate verifier model. See LLM evaluation and benchmarking for building regression suites that run on every model or prompt change.

Faithfulness metrics for RAG

When answers must cite retrieved chunks, measure attribution precision (each claim supported by a cited span) and recall (important facts from context included). Tools like RAGAS and human rubrics score answer groundedness vs context, answer relevance, and context precision.

Production signals

Track user thumbs-down, edit distance on copied answers, support tickets citing wrong info, and automated spot-checks on high-risk intents. Log retrieved chunks alongside completions for post-incident review.

Mitigations that work (and limits)

No single technique eliminates hallucinations. Effective systems layer defenses and design UX that assumes errors will occur.

Retrieval-augmented generation (RAG)

Ground answers in your documents, database, or search index. Good RAG reduces factual drift but does not fix bad retrieval — garbage chunks in, confident garbage out. Invest in chunking, hybrid search, reranking, and metadata filters. Force the model to quote spans or return structured citations users can click.

Prompting and output constraints

System prompts that require “say I don’t know if not in context,” low temperature for factual tasks, and JSON schemas with enum fields reduce improvisation. Prompt engineering helps at the margin; it is not a substitute for verification on critical paths.

Tool use and verification loops

Let the model call search, calculators, code execution, or chain explorers — then feed results back. A second pass that checks claims against tool output catches many errors. For agents, separate planning from execution with human approval on irreversible actions.

Fine-tuning and preference optimization

Supervised fine-tuning on domain Q&A with refusal examples, and RLHF/ DPO rewards for grounded answers, improve tone and abstention. They do not guarantee correctness on unseen facts unless training data covers them.

Post-generation verification

NLI models, entailment checkers, regex validators, and rule engines flag answers that contradict retrieved text. For code, run linters and tests. For finance, cross-check numbers against APIs. This layer is where mature products invest.

UX and liability

Disclaimers, visible sources, confidence cues, and easy correction flows manage user trust. Never present LLM output as authoritative legal, medical, or investment advice without professional review.

A production checklist

Classify intents by risk — trivia chat vs wire transfer vs contract interpretation.
Require retrieval for any answer that should reflect your private or changing data.
Block uncited claims in regulated flows; show source spans inline.
Run golden-set evals in CI when models, prompts, or indexes change.
Instrument tool calls and log context + completion for audits.
Add calculators and parsers instead of trusting mental math.
Human review for edge cases the automation cannot verify.
Monitor abstention rate — zero “I don’t know” often means hidden hallucination.

Hallucinations vs other failure modes

Failure	Symptom	Primary fix
Hallucination	Confident false or unsupported content	RAG, verification, abstention training
Prompt injection	Model follows attacker instructions in untrusted text	Input sanitization, tool sandboxing, policy layers
Stale knowledge	Correct-sounding but outdated facts	Retrieval, date-aware prompts, smaller refresh cycles
Format errors	Invalid JSON, broken markdown	Structured output modes, repair passes, schemas

Teams that conflate these problems apply the wrong fix — stricter system prompts do little against bad retrieval, and better chunking does not stop jailbreaks.

Key takeaways

Hallucinations are expected behavior of next-token predictors, not occasional bugs — plan for them in system design.
Grounding via RAG and tools cuts factual drift but must be paired with retrieval quality and citation verification.
Measure faithfulness on your own data; public benchmarks are a starting point, not proof your app is safe.
High-risk actions need deterministic checks, not eloquent apologies after the fact.
Teaching models to say “I don’t know” is a feature, not a failure — it beats a convincing lie.