Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation (2025)

Abstract: Large Language Models (LLMs) often exhibit hallucinations, generating factually incorrect or semantically irrelevant content in response to prompts. Chain-of-Thought (CoT) prompting can mitigate hallucinations by encouraging step-by-step reasoning, but its impact on hallucination detection remains underexplored. To bridge this gap, we conduct a systematic empirical evaluation. We begin with a pilot experiment, revealing that CoT reasoning significantly affects the LLM’s internal states and token probability distributions. Building on this, we evaluate the impact of various CoT prompting methods on mainstream hallucination detection methods across both instruction-tuned and reasoning-oriented LLMs. Specifically, we examine three key dimensions: changes in hallucination score distributions, variations in detection accuracy, and shifts in detection confidence. Our findings show that while CoT prompting helps reduce hallucination frequency, it also tends to obscure critical signals used for detection, impairing the effectiveness of various detection methods. Our study highlights an overlooked trade-off in the use of reasoning. Code is publicly available at: https://github.com/ECNU-Text-Computing/cot-hallu-detect .

## introduction large language models (llms) have demonstrated remarkable performance across a range of natural language processing (nlp) tasks @xcite @xcite , showing a strong ability to follow instructions and answer questions @xcite . however, llms may also exhibit hallucinations. specifically, they may produce factually incorrect or semantically irrelevant answers to a given prompt without indicating any lack of confi-* corresponding author. to reduce hallucinations in llms, extensive research has focused on hallucination detection and mitigation methods @xcite @xcite . existing detection methods are built upon various aspects of llms, including output consistency (e.g., evaluating the consistency of responses to multiple identical queries) @xcite , analysis of internal states (e.g., computing confidence metrics) @xcite @xcite , and self-evaluation of knowledge ( e.g., querying its certainty) @xcite . regarding hallucination mitigation, chain-of-thought (cot) prompting improves response reliability and interpretability by guiding llms to generate intermediate reasoning steps before arriving at a final answer @xcite . in contrast to more complex solutions such as fine-tuning llms or improving decoding strategies, cot prompting is widely adopted due to its simplicity and effectiveness @xcite . the performance improvements shown by @xcite and @xcite through enhanced reasoning have further fueled growing research interest in reasoning-centric llms. however, as illustrated in figure 1 , the use of cot prompting may diminish the effectiveness of hallucination detection methods. when an llm is prompted to perform step-by-step reasoning, it semantically amplifies the llm's internal confidence in its output. as a result, even when the final answer deviates from the ground truth, the llm tends to produce incorrect tokens with high confidence (i.e., high probability). consequently, hallucinations induced by cot prompting often appear more plausible, making them more difficult for detection methods to identify. ## pilot experiment and observation recent studies on multiple-choice question answering (mcqa) tasks @xcite @xcite have identified a strong correlation between the probability assigned by llms to answer tokens and the correctness of their final predictions. in these datasets, which are primarily composed of factual questions, we adopt the common practice of treating answer incorrectness as a proxy for hallucination labels @xcite . motivated by these insights, we perform a pilot experiment to analyze, at the token level, how reasoning with chain-of-thought (cot) prompting shapes the distribution of probabilities over possible answer tokens. our analysis pays special attention to instances where cot prompt-ing fails to eliminate incorrect (i.e., hallucinated) answers. ## results and analysis this section presents a comprehensive analysis of the experimental results, focusing on three key aspects: (1) the performance enhancement of llms through cot prompting across diverse tasks and datasets; (2) the impact of cot prompting on the internal mechanisms of llms, as revealed by the distribution shifts in hallucination detection scores; and (3) the consequential effects on the performance and reliability of hallucination detection methods. our findings demonstrate that while cot prompting significantly improves output quality, it also alters the models' internal representations and confidence calibration. as a result, this ultimately complicates the hallucination detection process. we validate our automated metrics by having two annotators manually annotate 150 samples, achieving a cohen's kappa of 0.863. the spearman's ρ between our automated scores and human annotations is 0.690 for exact match, 0.656 for rouge-l f1, and 0.683 for semantic similarity. these results confirm that our automated evaluations reliably reflect human judgment. ## discussion and conclusion in this study, we systematically investigate the impact of cot prompting on hallucination detection in llms. we evaluate several representative detection methods and find that, although cot prompting improves text-level and semantic accuracy across most datasets, it simultaneously undermines hallucination detection performance. more specifically, cot prompting results in more concentrated detection score distributions, lower classification accuracy, weaker alignment with ground truth, and poorer confidence calibration. the sensitivity of this impact varies according to the detection method: self-evaluation-based approaches are the most affected, whereas consistency-based methods show greater robustness. we further observe that cot prompting diminishes and occasionally eliminates the performance advantage of advanced detectors over a simple perplexity baseline. these findings suggest that cot prompting can obscure hallucination cues and compromise the effectiveness of existing detection systems, highlighting the need to develop more robust approaches specifically tailored to reasoning-oriented llms.