Cruxly › AI engineering › Evaluation

Hallucination: Taxonomy, Detection and Mitigation in Production

37 unsupported claims in 21 answers, a verbatim-quote check, abstentions from 4 to 13, and a threshold set by rupees

Prompt: "Our RAG assistant answers questions about loan and insurance policies for an Indian NBFC. Compliance says it 'sometimes makes things up'. How do you measure the hallucination rate, how do you cut it, and why can't we just add 'do not hallucinate' to the system prompt? Also: is a wrong answer caused by bad retrieval a hallucination?"

40 minutes first. Define the unit you count, work the before and after numbers on 100 audited answers, and set an abstention threshold from a rupee cost. The trap is quoting one 'hallucination rate' without saying whether it is per claim or per answer, or what the model was allowed to do instead of answering.

A taxonomy that tells you which fix to reach for

"Hallucination" is a family of failures that look the same to a user and have different causes, so name the type before choosing the fix.

• Fabricated fact (closed-book). The model states something that is not in any provided source and is false or unverifiable: an invented policy clause, a made-up citation, a nonexistent API method. Summarisation research uses the words intrinsic (the output misrepresents the source) and extrinsic (the output adds content the source does not support, which may even be true); this bullet is the extrinsic kind. Maynez et al.'s On Faithfulness and Factuality in Abstractive Summarization (ACL 2020) found substantial hallucinated content in every model they tested, and that textual entailment correlated with faithfulness better than standard overlap metrics.
• Unsupported or contradicting claim despite the right context (faithfulness failure). The right passage is in the prompt and the answer still goes beyond it or against it: "the policy covers hospitalisation" when the text says "excluding pre-existing conditions". Contradiction is the intrinsic kind; going beyond the passage is the extrinsic kind again. This is the failure the RAGAS faithfulness metric in Evaluation Gates That Actually Block a Bad Deploy counts.
• Wrong tool result trusted. A tool returns an error or an empty list and the agent narrates success; or a tool returns a stale record and the answer repeats it. The text is faithful to the tool and false about the world.
• Instruction drift. Over a long conversation the model stops honouring a constraint ("only answer from the documents"), so a faithful system turns into a closed-book one.

Hallucination versus retrieval failure. If retrieval missed the passage and the model answered anyway from memory, it is both: a retrieval failure caused the context gap, and a generation behaviour (answering instead of abstaining) filled it. If retrieval missed and the model said "not found", the system behaved correctly and the miss is a recall problem for Choosing, Evaluating and Fine-Tuning an Embedding Model and its neighbours. If retrieval found the passage and the answer misread it, retrieval is innocent. Triage each failure with two questions: was the supporting passage in the context, and did the answer state something beyond it. The four combinations give four different tickets.

Terms. The surveys (Ji et al. 2022; Huang et al. 2023) use differing vocabularies (intrinsic and extrinsic, factuality and faithfulness), so say which one you use. In production the useful split is operational: was the claim supported by the context I gave the model, and is it true about the world. Different checks, different ground truth.

Why models hallucinate, and why 'do not hallucinate' is not a control

A language model is trained to produce a plausible continuation, and nothing in that objective, or in most of the scoring that follows, makes "I do not know" the best move. Describe the mechanism, not a motive.

Pretraining learns a distribution, not a fact table. For a fact seen often, such as a capital city, the correct token dominates. For a fact seen once or never (the premium on a niche insurance variant, the page number of a clause), several fluent continuations have similar probability, and sampling picks one that looks right. Kalai, Nachum, Vempala and Zhang's Why Language Models Hallucinate (arXiv, September 2025) frames this as an ordinary classification problem: if the model cannot reliably tell valid statements from plausible invalid ones, errors follow from pretraining statistics.

Post-training and evaluation can reward the guess. The same paper argues that most benchmarks score answers in a binary way, 1 for right and 0 for wrong or blank, so a model that guesses always scores at least as well as one that abstains, and training against such scores rewards guessing over expressing uncertainty. Their proposed remedy is to change how the dominant leaderboards score uncertain answers (penalising confident errors, crediting appropriate abstention). This is the authors' argument, not a settled empirical consensus, but it matches what you can test on your own system: if your eval gives no credit for "not found", you are training your product to guess.

Calibration helps but is limited. Kadavath et al.'s Language Models (Mostly) Know What They Know (2022) reported that larger models are well calibrated on multiple-choice and true/false questions when the format is right, that a model can estimate the probability that its own answer is correct (P(True)), and that a trained "do I know this" predictor generalises only partly to new tasks. Treat model confidence as a signal to measure on your distribution, not a guarantee.

Why "do not hallucinate" fails as a control. An instruction changes the distribution of outputs; it adds no knowledge and does not give the model a way to check itself. If the question needs a fact the context lacks, "do not make things up" competes with every learned pull to be helpful and fluent. In practice a clear instruction ("answer only from the passages; if the answer is not there say NOT_FOUND") does shift behaviour, which is why it belongs in the prompt, but how much it shifts is an empirical number that moves with the model and the wording. It is a hint to be tested on your eval, not a control you can trust. Controls are things you can verify outside the model: a schema that makes abstention a legal output, a check that a quoted sentence exists in the source, a threshold on a measured signal.

Long context and drift. A grounding rule at the top of a long prompt competes with thousands of tokens of retrieved text and history (Multi-Turn Conversation State and Context-Window Limits). Re-state it near the end and re-test on long conversations.

Measuring the rate: claims, answers and an audited judge

Report hallucination at two levels, per claim and per answer, because they differ by a lot and answer different questions. The method has four steps.

@@ 1 | Fix the unit and the reference For a RAG assistant the reference is the retrieved context plus the question; "unsupported" means a claim that the context does not entail. Closed-book correctness against the world is a different test with a different gold set.

@@ 2 | Decompose each answer into atomic claims A model or a rule splits "the policy covers X up to INR 5 lakh with a 30-day waiting period" into claims. Atomic claims make errors countable and localised.

@@ 3 | Judge each claim against the context An entailment model or an LLM judge labels each claim supported, contradicted or unsupported. Calibrate the judge against a human on 100 or more claims as in LLM-as-a-Judge: Evaluation That Scales Without Lying to You, because a judge that flags 3 percent too many or too few moves your headline number.

@@ 4 | Audit and aggregate Humans re-check every flagged claim (or a sample of 40) and a sample of the unflagged ones, then report both rates with their denominators.

The worked example (100 audited questions; illustrative numbers I made up to show the arithmetic). The assistant answered 96 of the 100 questions and abstained on 4. Its 96 answers contained 412 claims. The judge marked 37 claims unsupported, and a human audit confirmed them.

• Per claim: 37 / 412 = 9.0% unsupported.
• Per answer: the 37 bad claims sit in 21 answers (an average of 1.76 per bad answer), so 21 / 96 = 21.9% of answered questions contain at least one unsupported claim, and 75 of 96 answers are fully clean.

The two numbers look unrelated, which is the point. A reader who sees "9 percent" thinks 91 percent of answers are fine, but a user reads whole answers: about one in five has a problem. With 4.3 claims per answer, independent errors at 9 percent would put at least one bad claim in 1 - 0.91^4.29 = 33% of answers; the observed 21.9 percent is lower, which tells you errors cluster (some answers are bad throughout, usually because retrieval missed). That is useful for diagnosis: the clusters are where to look.

Confidence intervals still apply. 21 of 96 has a 95 percent interval of about 15 to 31 percent (Wilson). A drop from 21 to 8 bad answers (the before and after below) is encouraging, but announce it only after the paired comparison from Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance on the same 100 questions, with the interval.

Do not skip the unanswerable questions. Put 10 to 20 percent of questions in the set whose answer is not in the corpus. A system that never abstains looks fine on answerable questions and fabricates on these. The set also lets you measure abstention directly, the subject of the last section.

Detection tools: quotes, entailment, consistency and logprobs

Four detectors cover most production needs; each has a cost and a blind spot, so combine them.

1. Citation and quote checking (cheap, strong for RAG). Ask the model to return, for each claim, the exact sentence from the source that supports it, then check mechanically that the string exists in the retrieved text after normalising whitespace, case and quotation marks. This catches invented quotes with no model call. Its blind spot is important: a quote can exist and still not support the claim, as when the model quotes "covered" from a sentence about what is not covered. So the quote check is a floor; follow it with an entailment or judge check on the pair (claim, quote) for high-stakes answers.

:::key Verified October 2026 Anthropic's citations feature (documentation retrieved October 2026) returns citations parsed by the API with the cited text extracted from your document; the docs say this guarantees valid pointers into the provided documents, that the cited_text field does not count towards output tokens, that all active models support it, and that citations cannot be combined with structured outputs. A valid pointer is not proof that the pointed-to text supports the claim, so keep an entailment check for high-stakes use. Other vendors' features differ; verify each against its own documentation. :::

2. Entailment or judge check on claims. Decompose, then ask "does the context entail this claim" using an NLI model or a judge. Maynez et al. found entailment measures track faithfulness better than overlap metrics. Costs one extra call per claim (batch them), and it inherits the judge's biases and error rate.

3. Consistency across samples. SelfCheckGPT (Manakul, Liusie and Gales; EMNLP 2023) samples several answers to the same prompt and checks whether a statement is repeated across them: facts the model knows tend to be stable, invented ones vary. It is black-box, needing no logprobs and no external database. Costs k extra generations, and a blind spot: a model can be consistently wrong, repeating the same popular misconception every time. The related Chain-of-Verification method (Dhuliawala et al., 2023) has the model draft an answer, plan verification questions, answer them independently of the draft, and revise; the authors report reduced hallucination on list-style questions and long-form generation, which does not guarantee the same gain on your task.

4. Logprob and confidence signals. Token probabilities of the answer, or a model's own P(True), correlate with correctness when measured. Kadavath et al. found large models reasonably calibrated in suitable formats, but calibration shifts after instruction tuning and with prompt wording, and varies by domain. Use them only after you plot reliability on your data: bucket answers by confidence, and check that the 90 percent bucket is about 90 percent correct.

Selecting. A reasonable stack for the policy assistant: quote check on every answer (milliseconds), claim-level entailment on answers about money and eligibility, consistency on the sampled monitoring traffic, and logprobs only as a routing hint to a human. Keep each detector's output in the trace (Observability and Cost Governance for LLM Systems) so the unsupported-claim rate can be charted.

Mitigations ranked by cost and evidence, with the before and after

Reach for the cheap structural fixes first, because they change what the system is allowed to say, then add checks, then pay for a stronger model. The ladder, in the order I would try it, with honest notes on the evidence:

1. Ground with retrieval and make the answer cite. Lewis et al.'s retrieval-augmented generation paper (2020) found retrieval-augmented models produced more specific, diverse and factual language than a parametric-only baseline on generation tasks. Grounding turns "know the fact" into "read the fact", and citations make the reading checkable. It does not remove hallucination: the model can still misread, which the faithfulness metric measures.
2. Narrow the task. "Extract the waiting period from this clause" hallucinates less than "tell me about the policy". Smaller outputs have fewer claims to get wrong; structure the response as fields.
3. Make abstention legal and cheap. Put NOT_FOUND in the schema as a first-class value with a reason, tell the model when to use it, and score it as a correct answer on unanswerable questions. A model forced to fill a required field will fill it.
4. Verify. Quote check, then claim entailment, and regenerate or escalate on failure. Chain-of-Verification style self-checks help on some tasks.
5. Self-consistency or best-of-n on high-stakes answers, accepting the extra cost.
6. A stronger model. Often reduces errors, but is the most expensive lever and the least targeted; confirm on your eval that it moves the numbers before you pay.

The before and after (same 100 questions, illustrative numbers continuing the example). The change is rung 3 plus rung 4: "quote the supporting sentence or return NOT_FOUND", with a verbatim quote check.

Unsupported claims fell by 70 percent and answers with a problem from 21 to 8. But coverage fell from 96 to 87 percent, and fully clean answers rose only from 75 to 79 per 100 questions, because 9 answers moved to NOT_FOUND. Some were fabricated answers to unanswerable questions (a good trade) and some were correct answers lost to over-caution (a cost). Count both using your unanswerable subset; "hallucinations fell 70 percent" alone would hide 9 points of coverage.

Prices make it a decision. Assume (these are made-up, for the arithmetic) a wrong answer reaching a customer costs ₹500 in rework and goodwill and an abstention routed to a human costs ₹40. Per 100 questions: before 21 x 500 + 4 x 40 = ₹10,660; after 8 x 500 + 13 x 40 = ₹4,520. The change saves about ₹6,140 per 100 questions on those assumptions. If a wrong answer cost only ₹50, break-even would fall to 20 percent and abstaining would rarely pay, which is why the threshold is a business input.

Abstention thresholds, coverage versus accuracy, and the user interface

The threshold at which the system should answer instead of abstaining is set by the cost of being wrong, not by a model score you happen to like. With a cost C_wrong for a wrong answer and C_abstain for handing off, answering is better when the chance of being right p satisfies (1 - p) x C_wrong < C_abstain, so the break-even is p* = 1 - C_abstain / C_wrong. With ₹500 and ₹40, p* = 1 - 0.08 = 0.92: answer only when you are at least 92 percent likely to be right.

Plot coverage against accuracy. Rank answers by a confidence signal (a judge score, an entailment probability, a calibrated logprob), sweep the threshold, and tabulate (illustrative figures):

Cost per 100 questions is abstentions x 40 + wrong answers x 500; for the strict row that is 70 abstentions and 100 x 0.30 x 0.03 = 0.9 wrong answers, so 70 x 40 + 0.9 x 500 = 2,800 + 450 = 3,250. The rule is that the marginal accuracy of the answers you add matters, not the average. Going from strict to medium adds 25 points of coverage; those added answers are (0.55 x 0.93 - 0.30 x 0.97) / 0.25 = 88% accurate, below the 92 percent break-even, so adding them costs more than it saves. If a wrong answer cost only ₹200, the break-even drops to 80 percent, the loose row's marginal accuracy of 82 percent clears it, and the best operating point moves to the loose threshold (cost ₹2,500 against ₹2,980 for strict). One curve, two businesses, two thresholds.

Calibrate on held-out data. Choose the threshold on one split and report on another; thresholds drift when the corpus, the model or the question mix changes, so recheck monthly. This is the same cost-threshold logic as in Evaluation Beyond Accuracy.

The interface is part of the control. Streaming Interfaces That Show Their Work covers rendering; for hallucination the rules are:

• Show sources next to claims so the user can check; a citation the user can click converts a silent error into a visible one.
• Label uncertainty only when a real signal exists. A decorative "AI can make mistakes" footer does not change behaviour; a "low confidence: please verify with the policy PDF" label driven by a measured threshold does, and a label that fires on 60 percent of answers is ignored.
• Make the handoff a first-class path. "I could not find this in the policy; here is how to reach an agent" is a good answer, and it should be a one-tap flow, not an apology.
• Do not stream an unverified answer when the stakes are high. Streaming shows tokens before a quote check can run; verify first for sensitive intents, or stream into a "draft" state and replace it.

How to say it in the interview

"First I separate the cases, because the fix differs: an unsupported claim when the right passage was retrieved is a faithfulness failure, a wrong answer because retrieval missed is a retrieval failure, and a made-up fact with no context is the closed-book kind. I measure by splitting answers into claims and judging each against the retrieved text, then report both per-claim and per-answer rates, with a human audit of the judge. To cut it I require a supporting quote or an explicit not-found, check the quote verbatim against the source, and make abstaining a legal output. In my example that moved unsupported claims from 9.0 to 2.9 percent and abstentions from 4 to 13 per 100, which is a trade I price in rupees, not a free win. Telling the model not to hallucinate adds no knowledge, so I treat it as a hint to be tested, not a control."

Checklist

• Triage every failure with two questions: was the supporting passage in the context, and did the answer go beyond it
• Decomposed answers into atomic claims and judged each against the retrieved context
• Reported both per-claim and per-answer rates with denominators
• Audited the judge against human labels on 100 or more claims, and re-checked its flags
• Included 10-20 percent unanswerable questions in the eval set
• Made NOT_FOUND a legal, scored output in the schema
• Required a supporting quote, verified it verbatim, and checked entailment on the claim and quote pair for high-stakes intents
• Treated 'do not hallucinate' prompts as a hint to be measured, not a control
• Reported coverage and the extra abstentions next to every drop in the unsupported-claim rate
• Set the answer-or-abstain threshold from p* = 1 - C_abstain / C_wrong on held-out data and checked marginal accuracy
• Showed sources, labelled uncertainty only from a measured signal, and gave a one-tap human handoff

Related lessons

• Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks
• Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance
• LLM-as-a-Judge: Evaluation That Scales Without Lying to You
• Offline Versus Online Evaluation: Shadow Mode, A/B Tests and Feedback Loops