HICD : Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Models (2025)

Abstract: Large Language Models (LLMs) often generate hallucinations, producing outputs that are contextually inaccurate or factually incorrect. We introduce HICD, a novel method designed to induce hallucinations for contrastive decoding to mitigate hallucinations. Unlike existing contrastive decoding methods, HICD selects attention heads crucial to the model’s prediction as inducing heads, then induces hallucinations by dispersing attention of these inducing heads and compares the hallucinated outputs with the original outputs to obtain the final result. Our approach significantly improves performance on tasks requiring contextual faithfulness, such as context completion, reading comprehension, and question answering. It also improves factuality in tasks requiring accurate knowledge recall. We demonstrate that our inducing heads selection and attention dispersion method leads to more “contrast-effective” hallucinations for contrastive decoding, outperforming other hallucination-inducing methods. Our findings provide a promising strategy for reducing hallucinations by inducing hallucinations in a controlled manner, enhancing the performance of LLMs in a wide range of tasks.

## introduction large language models(llms) have demonstrated exceptional performance across a wide range of nlp tasks @xcite . however, they are prone to hallucinations, where they generate content that deviates from facts or relevant contexts, hindering their practical applications in real-world scenarios. to address this challenge, efforts have been devoted to mitigate knowledge hallucinations in llms @xcite . in this work, we focus on mitigating hallucinations during inference generation @xcite . to address this, some studies have focused on developing effective inference-time decoding strategies. among these, contrastive decoding based approaches have demonstrated strong performance @xcite . however, current contrastive decoding methods typically compare the model's inherent outputs, such as those from earlier layers or smaller models, with the original outputs @xcite . existing contrastive decoding approaches have rarely explored constructing hallucinated outputs to improve their efficacy in hallucination mitigation @xcite . previous work has highlighted that current contrastive decoding methods, due to their coarse contrast and simplistic subtraction operations, may disrupt the original output distribution of the llm @xcite . therefore, investigating the construction of hallucinated outputs for more effective contrast with original outputs warrants further research. building on this, @xcite proposed inducing hallucinations in llms via slight fine-tuning or zero-shot prompting, and mitigating them through contrastive decoding with the original outputs. and there's a method that prunes retrieval heads to generate hallucinated outputs for comparison with the original outputs @xcite . however, these hallucinationinducing methods require additional fine-tuning or rely on the inherent properties of the model postpretraining, limiting their adaptability in different datasets. moreover, the plausibility of the hallucinations and their effectiveness for contrastive decoding have not been validated. other works have addressed the issue of hallucinations by focusing on model interpretability. some studies examined attention heads that play a key role in output quality @xcite ). another study revealed that key points causing hallucinations in llms are the inconsistencies in the information flow integration between memory heads figure 1 : illustration of hallucination-inducing contrastive decoding method(hicd). the method include calculation of the importance scores and identification of the inducing heads (yellow), dispersing attention of inducing heads to induce hallucinations (pink) and applying contrastive decoding for hallucination mitigation (blue). and context heads, and effectively mitigated hallucinations by pruning conflicting attention heads @xcite . this suggests that targeting the attention heads critical to hallucinated outputs can effectively control hallucination generation. inspired by these studies,we propose hicd, a method that induces hallucinations through attention dispersion on inducing heads for contrastive decoding to mitigate hallucinations. to address the limitation that existing hallucination-inducing methods rely on model's internal parameters, restricting adaptability to different datasets, we construct correct and incorrect (adversarial) samples by pairing questions with corresponding right and wrong answers. we then compute task-relevant importance scores for attention heads that are critical to generating correct outputs (right heads) and incorrect outputs (wrong heads). finally, we select heads that contribute to correct outputs while suppressing those leading to incorrect outputs, resulting in a set of inducing heads. to improve the effectiveness of contrastive decoding methods, the attention maps of the inducing heads are averaged, ensuring attention values are equalized across all tokens within each head. this redistribution disperses attention, effectively inducing hallucinated outputs optimized for contrastive decoding, as demonstrated by experiments. finally, these hallucinated outputs are compared with the original model's outputs to mitigate hallucinations. our experiments are primarily conducted using models from the llama @xcite , qwen @xcite , and mistral (jiang et al., 2023) families. our findings show that compared to existing contrastive decoding methods, hicd significantly improves faithfulness in tasks requiring contextual understanding, such as hellaswag @xcite , race @xcite , openbookqa @xcite . furthermore, hicd also enhances the model's accuracy in factual recall tasks like truthfulqa @xcite and factor @xcite , as well as generation tasks on xsum @xcite and nq-swap @xcite . our contributions are as follows: ## method the overall algorithm of hicd is shown in figure 1 . first, we identify the inducing heads that are closely associated with generating hallucinations(3.1). next, we apply attention dispersion to these inducing heads to induce task-relevant hallucinations (3.2). finally, these hallucinated outputs are compared with the original model outputs through contrastive decoding to alleviate hallucinations (3.3). ## conclusion in this paper, hicd are introduced to induce hallucinations on inducing heads for contrastive decoding to mitigate hallucinations. experiments on several tasks show that hicd outperforms existing methods in contextual tasks and achieves competitive results in factual consistency tasks. we also find that selecting task-relevant inducing heads improves performance compared to out-of-domain selections. and attention averaging induces more contrast-effective hallucinations compared to other methods. our work opens new directions for hallucination induction and mitigation, providing a promising strategy to reduce hallucinations and enhance llm robustness across tasks.