Trust Me, I ’m Wrong: LLM s Hallucinate with Certainty Despite Knowing the Answer (2025)

Abstract: Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety.

## introduction llms often hallucinate-generate outputs which are not grounded in real-world facts and may thus hinder their reliability @xcite @xcite . numerous studies have attempted to identify hallucinations, with a particular line of research highlighting a strong relationship between hallucinations and a model's low certainty @xcite . these studies demonstrate that certainty estimation metrics @xcite @xcite can be used to detect and * {adi.simhi,itay.itzhak}@campus.technion.ac.il figure 1 : do high-certainty hallucinations exist even when the model knows the answer? an illustrative categorization of hallucinations based on a model's knowledge and certainty. highlighted is the phenomenon of high-certainty hallucinations (purple) -where models confidently produce incorrect outputs, when they have the correct knowledge. while other types of certain hallucinations can potentially be explained by the model not knowing, or being mistaken, high-certainty hallucinations, despite knowledge, are harder to rationalize, making their existence particularly intriguing. mitigate hallucinations based on an apparent correlation between low certainty and hallucinations. while low certainty has shown promise for addressing hallucinations, its relationship with hallucinations is not always straightforward. indeed, recent work suggests that models may hallucinate even when highly certain @xcite . however, these studies may conflate certainty with lack of knowledge-that is, the model could be certain of a wrong answer simply because it is missing the correct information. in this work, we focus on a distinct failure mode: certain hallucinations that occur even when the model does have the correct knowledge (figure 1 ). such hallucinations pose serious risks in highstakes knowledge-intensive domains, where certainty is often taken as a sign of decision reliability, such as medicine @xcite , law @xcite , and military @xcite . we term these hallucinations choke: certain hallucinations overriding known evidence. to detect choke examples, we integrate two frameworks into a two-stage procedure: one for identifying hallucinations despite knowledge, and one for estimating model certainty. for the first stage, we build on the approach of @xcite , who identify cases where a model knows the correct answer but hallucinates following a prompt perturbation. next, we estimate the model's certainty with three widely used but conceptually different methods: tokens probabilities @xcite , probability difference between the top two predicted tokens @xcite , and semantic entropy @xcite . our findings show that choke examples are widespread, and appear across two datasets with both pre-trained and instruction-tuned models. furthermore, choke examples exhibit much higher consistency across prompts than other hallucinations, indicating that they form a distinct category. to evaluate how well existing hallucination mitigation strategies handle these distinct examples, we introduce the choke-score: a metric tailored to measure mitigation effectiveness specifically on choke examples. unlike standard metrics, choke-score isolates performance on examples where the model is both wrong and certain despite knowing the correct answer, revealing failures that may be hidden by overall accuracy. to improve performance on these challenging examples, we introduce a new probe-based mitigation method that focuses the training on choke examples and outperforms existing methods. this evaluation exposes blind spots in current mitigation methods, which is crucial when model certainty guides decisions @xcite . our contributions are three-fold: ## background this section overviews related work on uncertainty in llms and their tendency to hallucinate, even when the correct answers are known. we rely on these findings throughout our study. ## methodology to show the existence of choke examples, we need to identify them and provide evidence that their portion from the total set of hallucinations is not negligible. to identify choke examples, we use the following procedure: we first identify hallucinations that occur even when the model possesses the required knowledge (section 3.1). next, we use common metrics for measuring model certainty (section 3.2) and set certainty thresholds to separate certain and uncertain generations (section 3.3). the process of choke examples detection is shown in figure 2 . additional experimental details are provided in section 3.4. ## results first, we identify whether choke examples are consistently present across certainty metrics and across different models (section 4.1). next, we show that choke examples are consistently present in instruct-tuned and larger models (section 4.2). lastly, we show that choke examples cannot be explained as noise (section 4.3). ## choke-score having established the existence and distinctiveness of choke examples, we hypothesize that the performance of hallucination mitigation methods will differ when evaluated on choke examples, compared to their overall effectiveness. these cases represent a unique challenge, and assessing mitigation on them may expose limitations or tradeoffs not captured by standard metrics. to this end, we introduce choke-score, a novel evaluation metric for assessing the ability of hallucination mitigation methods to reduce choke examples. this perspective is particularly valuable in high-stakes settings, where model certainty impacts decisions. formally, given choke examples detected via a certainty measurement method d (section 3.2), we measures the proportion successfully mitigated: here, c d is the set of choke examples flagged by a detection method d, and m is the set of successfully mitigated hallucinations. the score ranges from 0 to 1, with 1 indicating that all c d examples were mitigated (found in m). to ensure a broad definition of choke-score, we combine all three detection methods from section 3.2 to create two score variations: where c ∩ is the set of choke examples flagged by all detection methods (intersection), and c ∪ is the set flagged by at least one method (union). these two variants allow us to evaluate mitigation effectiveness under strict (chk) and flexible (chk-f) detection criteria. while chk targets strict cases flagged by all methods, chk-f includes any case flagged by at least one method. together, they offer a fuller picture of mitigation performance. ## discussion and conclusion this work investigated high certainty hallucinations occurring despite the model having the knowledge to answer correctly-a phenomenon we termed choke. while hallucinations are typically linked to uncertainty or ignorance, choke, arises with both certainty and sufficient knowledge. choke examples are especially concerning in high-stakes domains, where certainty often proxies reliability. to address this, we introduce choke-score, a new metric for evaluating mitigation methods on choke. while existing mitigation methods perform well on standard benchmarks, they struggle on choke-score. to overcome this, we propose a new method that outperforms others on this metric. our findings highlight the need to understand choke and develop targeted mitigation.