Hallucination

Definition: Hallucination is when a language model generates output that is fluent, confident, and internally coherent, but factually wrong, unsupported by any real source, or contradicted by the very context it was given. It is not a bug in the traditional sense — the model isn’t malfunctioning, it’s doing exactly what it was trained to do: produce statistically plausible text. The failure is that “plausible” and “true” are different targets, and nothing in standard next-token prediction forces them to converge. Hallucination ranges from a single wrong date in an otherwise accurate answer to an entirely fabricated legal case, API function, or research citation presented with total confidence.

How It Works

Next-Token Prediction Has No Truth Function

An LLM is trained to minimize the difference between its predicted probability distribution over the next token and the distribution observed in training text. There is no separate mechanism that checks “is this claim true” before the token is emitted — truth is never an explicit training signal. The model learns correlations: which words tend to follow which other words in which contexts. Most of the time, those correlations track reality closely enough that outputs are accurate, but the objective function rewards fluency and plausibility, not verified correctness, so when the two diverge, the model has no internal alarm.

RLHF (Reinforcement Learning from Human Feedback) and instruction tuning refine which plausible continuations the model prefers, steering it toward helpful, safe-sounding answers — but this still optimizes for human raters’ judgments of a good-looking response, not an independent fact-check against reality. A model can become better at sounding trustworthy without becoming more accurate, which is part of why polish and correctness can drift apart as models improve.

The Objective Function, Formally

At each generation step the model maximizes $P(x_t \mid x_{<t})$ , the probability of the next token given everything generated so far. Training minimizes cross-entropy loss over the whole sequence:

$\mathcal{L} = -\sum_{t=1}^{T} \log P(x_t \mid x_{<t}; \theta)$

Nowhere in this loss is there a term for factual correctness against an external world state. A model can reach very low $\mathcal{L}$ on its training distribution while still being wrong about facts the training text itself got wrong, or facts it never saw at all. This is the formal version of the informal point above: fluency is optimized directly, and truth is optimized only indirectly, to the extent it happens to correlate with fluent, high-likelihood training text.

Where the Gaps Come From (Training-Time Causes)

Coverage gaps: obscure facts, recent events, or niche technical details are underrepresented or absent in the training corpus, so the model has nothing reliable to draw on.
Contradictory or noisy source data: the internet contains conflicting claims about the same fact; the model compresses these into an averaged, sometimes-wrong representation.
Lossy compression of knowledge: billions of facts get encoded into a fixed set of weights, and long-tail facts are the first to blur or drop out, similar to how a highly compressed image loses fine detail first.
Distributional bias toward common patterns: if a plausible-sounding but wrong answer appeared more often in training data than the rare correct one, the model may reproduce the popular wrong answer.

Where Grounding Fails (Inference-Time Causes)

No retrieval or external lookup: a base LLM answering from parametric memory alone has no way to check itself against a live source.
Context gets ignored or overridden: even when correct information is provided in the prompt (a document, a table, a system message), attention can drift, and the model falls back on its internal, possibly wrong, prior.
Ambiguous or underspecified prompts: when a question is vague, the model still has to produce something, and it will often silently pick an interpretation and answer as if it were the only one.
Long-context degradation: information buried in the middle of a very long context window is statistically more likely to be underweighted than information near the start or end.

Model Variants and Where Risk Concentrates

Hallucination risk is not uniform — it concentrates differently depending on how a model is deployed and tuned.

Base (pretrained) models: the highest raw hallucination rate, because nothing in pretraining taught the model to defer, hedge, or express uncertainty — it always completes the pattern.
Instruction-tuned models: trained partly to hedge or decline, which reduces blunt fabrication but introduces sycophantic hallucination — agreeing with a false premise embedded in the user’s own question.
RAG-augmented systems: lower factual hallucination when retrieval is accurate, but exposed to a different failure mode — a stale, incomplete, or poisoned index gets treated as confidently as real memory would be.
Agentic / tool-calling systems: the model can hallucinate which tool to call or what parameters to pass, turning a text-only error into an executed, real-world action.

The Confidence Trap

A hallucinated sentence and a correct sentence are generated by the exact same mechanism, at the exact same surface-level “confidence.” The model doesn’t have a distinct “I’m making this up” mode — it produces its most likely next token whether that token is grounded in real knowledge or statistical filler. This is why hallucinations are dangerous: they don’t read as guesses. They read as facts, often with invented specifics (dates, page numbers, function signatures) that make them look more credible than a hedged, honest answer would.

This also means simple interventions like instructing the model to “be more careful” have limited effect. The instruction changes surface hedging language — more “I think” or “it’s possible that” — without adding a verification step the model doesn’t otherwise have. A politely hedged hallucination is still a hallucination.

Hallucination Snowballing

Once a model commits to a false claim early in a response, it tends to generate further details that are consistent with that false claim rather than backtracking — a pattern researchers call snowballing. Each new token is conditioned on everything generated before it, including the earlier error, so the model treats its own prior mistake as established context.

A wrong opening premise (“this function takes three arguments”) gets reinforced by invented detail (“the third argument controls timeout behavior”) that never existed either.
The output can become internally consistent and even more convincing as it goes, since later sentences correctly follow from the earlier false one.
This is part of why asking a model to “double-check” its own prior answer in the same context often fails — the check is generated under the influence of the same context that produced the original error.

Why It Matters

It is the single largest reliability barrier standing between LLMs and high-stakes deployment — legal, medical, financial, and safety-critical applications cannot tolerate confidently-wrong output.
It directly shapes product architecture: the popularity of Retrieval-Augmented Generation (RAG) pipelines exists largely because raw parametric generation cannot be trusted alone.
It creates a specific, measurable failure mode for benchmarking — factual QA accuracy, citation-verification rates, and faithfulness scores are now standard model evaluation metrics.
It undermines automation ROI: every hallucination that reaches a user erodes trust faster than ten correct answers build it, which caps how much human review can realistically be removed from a workflow.
It has produced real legal and professional consequences — lawyers have been sanctioned by courts for submitting filings with fabricated case citations generated by an LLM.
It complicates AI Alignment work, since a model can be “aligned” with instructions and tone while still being unreliable on facts — helpfulness and truthfulness are not the same axis.
It scales with fluency: newer, more articulate models can produce more convincing hallucinations, not fewer, because better language modeling improves style independently of factual grounding.
It is domain-sensitive: hallucination rates spike sharply on long-tail, recent, or numerically precise queries such as statistics, citations, and code APIs, compared to common-knowledge questions.
It shapes UX design: interfaces increasingly surface citations, confidence flags, or “verify this” prompts specifically to manage hallucination risk rather than eliminate it outright.
It is a moving research target — every generation of frontier models reduces certain hallucination classes while sometimes introducing new ones, so it cannot be treated as a solved, static problem.

Taxonomy of Hallucination Types

Not all hallucinations fail the same way. Distinguishing the type matters because each has a different root cause and a different fix.

Type	What it looks like	Root cause	Example
Factual / intrinsic hallucination	A confidently stated fact that is simply wrong	Gaps, errors, or contradictions baked into training data; nothing to check the claim against	Stating the Eiffel Tower was completed in 1922 (it was 1889)
Faithfulness / context hallucination	Output contradicts or ignores the source material it was explicitly given	Attention fails to stay anchored to the provided context; parametric memory overrides supplied text	Summarizing a contract and inventing a termination clause that isn’t in the document
Reasoning-chain hallucination	An intermediate reasoning step is fabricated or invalid, even if the final answer format looks right	Chain-of-thought text is generated token-by-token like everything else, with no verification of each step	Citing a nonexistent theorem or misapplying a real one to justify a math answer
Extrinsic / unverifiable hallucination	A claim that isn’t directly contradicted by context or known facts, but also can’t be verified from any available source	Model fills an information vacuum with a plausible-sounding but unsupported addition	Adding a specific but invented statistic to an otherwise accurate summary
Sycophantic hallucination	The model validates or elaborates on a false premise embedded in the user’s own question	Instruction tuning rewards agreeable, user-pleasing responses over pushback	Confirming details of a made-up historical event because the user’s question assumed it happened

Factual and faithfulness hallucinations are the two most discussed in research literature. Reasoning-chain and sycophantic hallucination have become bigger concerns as chain-of-thought prompting and conversational agents spread, since both can hide inside an answer that otherwise looks well-reasoned and cooperative.

Severity Spectrum

Type is one axis; severity is another, and the two are independent — a factual hallucination can be trivial or catastrophic depending on what it’s attached to.

Low severity: a wrong date or minor detail in an aside that doesn’t affect the substance of the answer.
Medium severity: a fabricated statistic or misquoted figure that is central to the argument being made, changing how a reader would act on the answer.
High severity: a fully invented citation, legal case, or API function presented as real and load-bearing for a downstream decision.
Critical severity: a fabricated instruction or status report inside an agentic system that gets executed automatically, with no human checkpoint before the action takes effect.

Mitigation Techniques

No single technique eliminates hallucination — each reduces a different slice of the problem and comes with its own failure mode.

Technique	What it does	Limits
Retrieval-Augmented Generation (RAG)	Injects retrieved, real documents into the context window so the model can generate from actual text instead of memory alone	Only as reliable as the retrieval step; the model can still misread, ignore, or blend retrieved passages with its own priors
Grounding and citations	Forces the model to attribute claims to specific sources, letting a human or system verify each statement	Citations can themselves be fabricated (“citation hallucination”) unless the pipeline independently confirms the source exists and says what’s claimed
Fine-Tuning on verified data	Adjusts model weights toward a curated, domain-checked dataset, reducing errors common in that narrow domain	Doesn’t generalize outside the fine-tuning distribution; verified datasets are expensive to build and go stale as facts change
Output verification / self-consistency checks	Samples multiple generations, or runs a separate verifier pass, and flags disagreement or low-confidence answers	Adds latency and inference cost; if all sampled outputs share the same wrong prior, they’ll agree and the check passes anyway
Structured decoding / constrained generation	Restricts output to a fixed schema, valid enum, or whitelist, such as only real function names from an API spec	Only works where the space of valid answers is enumerable; doesn’t help with open-ended factual claims
Confidence calibration / abstention training	Trains the model to output “I don’t know” or a calibrated confidence score when it lacks grounding, instead of always answering	Requires labeled examples of what the model doesn’t know, which is hard to enumerate; models stay reluctant to abstain because refusals were historically penalized in helpfulness training

Layered Defense in Practice

No single row in the table above is deployed alone in a serious production system. A typical high-stakes pipeline stacks them in order:

Retrieve relevant, real documents before generation starts, so the model has grounded material to draw from.
Constrain structured outputs, such as tool calls or form fields, to a validated schema so malformed or invented values are rejected outright.
Attach citations to every substantive claim so a human or downstream system can trace it back to a source.
Verify with a self-consistency or separate-verifier pass on anything routed to an irreversible or high-stakes action.
Escalate to a human when confidence is low, the claim is high-severity, or the action is irreversible.

Each additional layer reduces hallucination risk further but adds latency and cost, so teams calibrate how many layers a given use case actually needs rather than applying the full stack everywhere.

Hallucination in Agentic and Tool-Use Systems

As LLMs move from single-turn chat into Intelligent Agent systems that call tools, browse, and take actions via Function Calling (Tool Use), hallucination stops being a purely text-based nuisance and becomes an execution risk. A fabricated belief can trigger a real API call, a wrong file edit, or an incorrect downstream decision instead of just an incorrect sentence on a screen.

The failure modes multiply because each step in an agent’s plan is itself a generation the model can get wrong, and later steps often condition on earlier ones without re-verifying them first.

Tool selection hallucination: the model picks a tool that doesn’t exist, or isn’t appropriate for the task, or invents a plausible-sounding tool name that was never registered.
Parameter hallucination: the model calls a real tool with fabricated arguments — a made-up file path, an invented user ID, or a wrong date range.
Result misreporting: the model summarizes a tool’s actual, correct output but adds or drops details, effectively hallucinating on top of real data.
Compounding across steps: in a Multi-Agent System, one agent’s hallucinated intermediate output becomes the next agent’s “ground truth” input, and errors compound across the chain instead of staying contained to one turn.
Premature termination or looping: a model convinced by its own hallucinated success signal can stop a task early, or loop indefinitely believing a failed step actually succeeded.

This is why production agent frameworks increasingly require tool calls to be schema-validated, actions to be logged and reversible where possible, and multi-agent pipelines to include an explicit verification or judge step rather than trusting each agent’s output at face value.

Failure	Example	Consequence
Tool selection hallucination	Calling `send_refund()` when no such tool was registered	Silent failure, or a crash the agent then has to explain away
Parameter hallucination	Calling a real `delete_file()` tool with a plausible but wrong file path	Real data loss from a fabricated argument, not a fabricated sentence
Compounding across agents	A summarizer agent invents a number; a planner agent treats it as fact	The final action is wrong even though each individual agent step “looked” reasonable

Why Scaling Alone Doesn’t Fix It

A natural assumption is that hallucination is simply a data-and-parameters problem: train a bigger model on more text and the errors disappear. The evidence is more mixed than that.

Scaling helps common-knowledge accuracy: larger models trained on more data do get measurably better at well-represented facts, since more examples of the same fact reinforce the correct pattern.
Scaling doesn’t fill genuine coverage gaps: if a fact appears rarely or never in training data, adding more parameters gives the model more capacity to generate a convincing wrong answer, not more access to a fact that was never there.
Fluency improves faster than calibration: bigger models get better at sounding right before, or even without, getting better at being right, which can make hallucinations from frontier models harder to catch than from smaller, clumsier ones.
Long-tail queries stay hard regardless of scale: the queries most likely to trigger hallucination — rare entities, recent events, precise numbers — are exactly the queries where more pretraining data doesn’t help much, because there was never much data to begin with.
New capabilities create new hallucination surfaces: scaling unlocked agentic tool use and long-context reasoning, both of which introduced hallucination failure modes, such as tool-call and reasoning-chain hallucination, that smaller, simpler models never had the capability to produce in the first place.

This is why the field has shifted focus from “train it away” toward architectural fixes like retrieval, verification, and structured decoding — approaches that don’t rely on scale alone to close the gap.

Why It Can’t Be Fully Eliminated

Even with excellent training data and solid retrieval, some amount of hallucination risk is structurally unavoidable given how these models work.

The world changes faster than any fixed corpus or index can track: a model’s training cutoff, or a RAG index’s last refresh, is always at least somewhat stale relative to the present moment.
Not every true claim is verifiable against an existing document: novel synthesis, calculation, or reasoning about a new combination of facts has no single source to check against, even in principle.
Verification systems can themselves be wrong: a retriever can return the wrong document, and a verifier model can hallucinate its own judgment about whether a claim is actually supported.
Open-ended generation always involves some interpolation: language models generalize by design, filling gaps between training examples — the same mechanism that produces useful generalization is what produces hallucination once it interpolates past the edge of what’s actually known.

This is why the realistic goal in most engineering discussions is reducing and containing hallucination risk to an acceptable level for a given use case, rather than claiming to have eliminated it entirely.

Measuring Hallucination

Quantifying hallucination is harder than it sounds, because “truth” isn’t always a single checkable fact.

Factual QA benchmarks compare model answers against a known ground-truth answer set, giving a straightforward accuracy number for closed-domain questions.
Faithfulness scoring checks whether a generated summary or answer is entailed by the provided source document, often using a separate model or classifier as the judge — distinct from factual accuracy against the real world.
Citation verification automatically checks whether a cited source exists and actually contains the claimed statement, catching a specific and common hallucination class.
Human evaluation remains the gold standard for subtle cases, such as reasoning-chain errors or nuanced factual disputes, but is slow and expensive to scale.
Self-consistency / sampling variance treats disagreement across multiple generations at nonzero temperature as a proxy signal for low confidence, without requiring ground truth at all.
Adversarial or stress-test suites purposely probe rare, ambiguous, or leading prompts designed to induce hallucination, giving a worst-case signal rather than an average-case one.
Production monitoring tracks user corrections, thumbs-down rates, or downstream error tickets as a real-world, if noisy and delayed, hallucination signal.

No single metric captures the full picture, which is part of why hallucination remains an active, unsettled research area rather than a solved benchmark.

Benchmark Numbers Aren’t Comparable

A headline like “model X hallucinates 4% of the time” is meaningless without knowing what it was measured against.

Different benchmarks test different domains — closed-book trivia, document summarization, and code generation produce very different hallucination rates on the same model.
Some benchmarks measure factual hallucination only, silently ignoring faithfulness or reasoning-chain hallucination entirely.
Prompt difficulty and topic distribution vary widely across test sets, so a low score on an easy benchmark says little about performance on rare or adversarial queries.
Vendors sometimes report the most favorable benchmark for a given model, which is why independent, standardized evaluation suites matter more than any single self-reported number.

Notable Milestones

TruthfulQA (2021) was one of the first widely used benchmarks specifically built to measure a model’s tendency to repeat common misconceptions rather than state facts, formalizing hallucination as a measurable target rather than an anecdotal complaint.
Introduction of RAG (2020) framed retrieval-augmented generation explicitly as a way to ground knowledge-intensive tasks in real documents instead of relying purely on a model’s internal, static memory.
High-profile chatbot demo errors — including a widely reported factual mistake in a major search company’s chatbot demo in early 2023 — showed the public and the press how a single confident hallucination in a live product moment can carry real reputational and financial consequences.
Courtroom sanctions for fabricated legal citations turned hallucination from an abstract AI-safety concern into a documented professional liability issue, discussed in detail in the Example section below.
Rise of hallucination leaderboards: independent evaluators began publishing standardized, cross-model hallucination and faithfulness rankings, pushing model providers to treat hallucination rate as a competitive, disclosed metric rather than an internal detail.

Terminology Note

The term borrows from computer vision, where early image-captioning and generative vision models were said to “hallucinate” objects or details that weren’t present in the source image. It carried over to language models because the underlying pattern is the same: the system generates content with no basis in its input or grounding data, but does so fluently enough to look intentional.

Some researchers push back on the term, arguing it implies a perceptual experience the model doesn’t have, and prefer more neutral alternatives like “confabulation” — borrowed from psychology, describing confident false memories produced without intent to deceive — or simply “fabrication.” The industry has broadly settled on “hallucination” regardless, largely because it stuck first and communicates the confidently-wrong quality better than more clinical alternatives.

“Hallucination” — the popular, industry-standard term; emphasizes that the content is perceived-as-real by the system despite having no basis.
“Confabulation” — preferred by some researchers; emphasizes confident fabrication without intent to deceive, borrowed directly from human memory-error research.
“Fabrication” — the most neutral, mechanism-agnostic term; avoids implying any perceptual or cognitive process at all.

Designing Around Hallucination in Products

Since hallucination can’t be fully eliminated at the model level, most production interfaces are designed to make it survivable rather than invisible.

Inline citations with source previews: letting users hover or click a citation to see the exact source passage turns a one-line trust decision into a two-second verification.
Confidence or freshness indicators: flagging answers that rely on older training data versus live retrieval helps users calibrate how much scrutiny a given answer deserves.
Diff-based review for code and structured edits: showing a proposed change as a diff, rather than silently applying it, lets a human catch a hallucinated function call or invented field before it ships.
Explicit “I don’t know” affordances: interfaces that make abstention a normal, unpenalized response reduce pressure on the model to fill every gap with a guess.
Scoped, narrow tool permissions: limiting what an agent can actually do — read-only access, sandboxed environments, approval gates on irreversible actions — bounds the damage a hallucinated tool call can cause even if it slips through.

None of these change the underlying generation mechanism. They change what happens after a hallucination is produced, which is currently the more tractable engineering problem.

Comparison

Hallucination is frequently confused with other model failure modes that have different causes and different fixes.

Concept	What it is	How it differs from hallucination
AI Bias and Fairness	Systematic skew in outputs that favors or disadvantages particular groups, usually traceable to imbalanced training data	Bias is a directional, often measurable skew tied to demographic or categorical attributes; hallucination is fabricated content that can affect any claim regardless of fairness dimensions
Overfitting vs Underfitting	A training-time generalization failure where a model memorizes training data too closely, or fits it too loosely	Overfitting is diagnosed by comparing training versus held-out performance during development; hallucination happens at generation time in a fully trained, well-generalizing model
Explainable AI (XAI)	Techniques for making a model’s decision process interpretable to humans	XAI explains how a model arrived at an output; it doesn’t verify whether that output is true, and a well-explained answer can still be a hallucination
Miscalibration	A mismatch between a model’s stated confidence and its actual accuracy	Miscalibration is about confidence scores being wrong; hallucination is about the content being wrong — a model can be well-calibrated on average while still hallucinating individual facts
Memorization / data leakage	The model reproduces near-verbatim text, including copyrighted or private text, that it saw during training	Memorization is accurate reproduction of real training data, sometimes undesirably so; hallucination is fabrication of content that was never real to begin with

Real-World Use Cases

Framed as where hallucination risk matters most in deployed systems:

Legal research tools: case-law summarization and brief-drafting assistants risk citing plausible-sounding but nonexistent court cases, which has already led to real courtroom sanctions.
Medical information assistants: clinical decision-support chatbots or patient-facing symptom checkers risk stating incorrect drug interactions or treatment guidance with unwarranted confidence.
Code generation assistants: coding copilots frequently invent library functions, package names, or API parameters that look syntactically correct but don’t exist — sometimes called “package hallucination.”
Financial research summarization: tools summarizing earnings calls, SEC filings, or analyst reports risk fabricating specific figures or misattributing statements to the wrong company or quarter.
Academic and research writing aids: citation generators and literature-review assistants risk producing fake paper titles, authors, or journal names that read as legitimate.
Customer support chatbots: automated agents risk inventing return policies, warranty terms, or account details that were never actually part of company policy.
Enterprise knowledge-base copilots: internal search assistants built on RAG over company documents can still misquote or misattribute internal policies if retrieval or grounding is weak.
News and journalism drafting tools: summarization assistants risk introducing details, quotes, or statistics not present in the original reporting.
Voice assistants and search-answer boxes: single-shot spoken or featured-snippet answers offer no visible citation trail, so users have no easy way to catch a wrong fact.
Educational tutoring systems: a tutoring bot that hallucinates a wrong formula, date, or definition risks teaching the error directly to a student with no independent way to check it.
AI-powered search overviews: a single synthesized answer replacing a list of links removes the natural cross-checking users used to do by comparing multiple independent sources.
Translation and localization tools: machine translation systems can insert content not present in the source text, especially for low-resource language pairs with sparser training data.

Where Hallucination Risk Is Lower

Not every use case carries the same exposure — it’s worth naming where the risk is genuinely manageable rather than treating every deployment as equally dangerous.

Creative writing and brainstorming: fiction, ideation, and rough drafts have no single “correct” output, so fabrication isn’t a defect — it’s the product.
Early-stage prototyping: throwaway code or mockups reviewed by a human before anything ships tolerate a higher error rate than production systems.
Low-stakes, easily-verified answers: quick explanations of well-known concepts where a wrong detail is trivial to spot and has no real-world consequence if missed.

Common Pitfalls

Trusting fabricated citations without verification: a citation that includes a real-looking author, title, and year is not evidence it exists — always check the source independently in high-stakes work.
Equating confidence with correctness: hallucinated and accurate text are produced by the same fluent, assertive style, so tone is not a reliable truth signal.
Assuming RAG fully solves the problem: retrieval reduces hallucination but doesn’t eliminate it — the model can still ignore, misread, or contradict the retrieved passages.
Over-trusting self-reported confidence: asking a model “how sure are you?” produces another generated, and potentially hallucinated, answer, not a calibrated probability.
Treating hallucination as fully solvable engineering debt: mitigation techniques reduce rates on known failure classes; they do not guarantee zero hallucination, especially on novel or adversarial inputs.
Evaluating only on easy benchmark prompts: hallucination rates spike on rare, recent, or numerically precise queries that benchmark suites often underrepresent.
Ignoring reasoning-chain hallucinations: a correct final answer can mask a fabricated intermediate step, which becomes dangerous the moment downstream logic depends on that step being genuinely valid.
Skipping human review in high-stakes domains: removing a human checkpoint from legal, medical, or financial workflows because the model “usually gets it right” ignores that the failures are precisely the ones least likely to be caught downstream.
Conflating hallucination with lying: the model has no intent, belief, or awareness that a statement is false — treating it as deception rather than a statistical byproduct leads to the wrong mitigation strategy.
Assuming bigger or newer models hallucinate less across the board: larger models often hallucinate less on common knowledge but can produce more convincing, harder-to-catch hallucinations on obscure topics due to improved fluency.

Quick Checks Before Trusting High-Stakes Output

Does every specific claim — a number, date, name, or citation — trace back to a source you can independently open and confirm yourself?
If the model cites a source, does that source actually contain the claim, and not just a topic that sounds related to it?
Would you accept a real consequence — money, a legal filing, a medical decision — riding on this specific sentence being true, not merely plausible?
Was this answer generated purely from the model’s internal memory, with no retrieval step or external check involved anywhere in the pipeline?

Example

In 2023, a lawyer preparing a federal court filing used an LLM chatbot to research supporting case law for a personal injury claim. The model returned a set of citations — case names, docket numbers, and quoted judicial reasoning — that read exactly like genuine legal research. The lawyer, trusting the tool’s fluent and confident output, submitted the brief without independently verifying the citations against a legal database.

Opposing counsel and the court could not locate several of the cited cases. On investigation, it turned out the LLM had fabricated them entirely: plausible case names, plausible court reasoning, plausible citation formatting, but no underlying case existed. This is a textbook factual/intrinsic hallucination — the model had no real case to draw on for the specific legal question asked, so it generated something statistically consistent with what a real citation looks like, without any mechanism to check whether it referred to something real. The court sanctioned the lawyer, and the incident became one of the most widely cited cautionary examples of unverified LLM output causing real-world professional harm.

The fix, in hindsight, maps directly onto the mitigation techniques above. A RAG pipeline grounded in an actual legal database would have retrieved only real cases to cite from. A citation-verification step would have flagged the nonexistent case numbers before filing. A basic human-review policy — treating LLM legal research as a first draft requiring independent verification, not a finished citation — would have caught the error regardless of which underlying technique was used. None of these would have made the model incapable of hallucinating; they would have caught the hallucination before it reached a courtroom.

Referenced by

Dig deeper

Spot an error in this note?