D e C o R e: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations (2025)

Abstract: Large Language Models (LLMs) often hallucinate, producing unfaithful or factually incorrect outputs by misrepresenting the provided context or incorrectly recalling internal knowledge. Recent studies have identified specific attention heads within the Transformer architecture, known as retrieval heads, responsible for extracting relevant contextual information. We hypothesise that masking these retrieval heads can induce hallucinations and that contrasting the outputs of the base LLM and the masked LLM can reduce hallucinations. To this end, we propose Decoding by Contrasting Retrieval Heads (DeCoRe), a novel training-free decoding strategy that amplifies information found in the context and model parameters. DeCoRe mitigates potentially hallucinated responses by dynamically contrasting the outputs of the base LLM and the masked LLM, using conditional entropy as a guide. Our extensive experiments confirm that DeCoRe improves performance on tasks requiring high contextual faithfulness, such as summarisation (XSum by 18.6%), instruction following (MemoTrap by 10.9%), and open-book question answering (NQ-Open by 2.4% and NQ-Swap by 5.5%).

## introduction large language models (llms) emerged as powerful natural language generators, demonstrating remarkable capabilities across a range of tasks @xcite @xcite . however, llms often produce hallucinations, generating content that is factually incorrect or lacks grounding in the given context @xcite @xcite . the tendency of llms to hallucinate undermines their reliability, especially in high-stakes domains such as clinical decision-making or legal reasoning @xcite . understanding the internal mechanisms responsible for hallucinations in llms remains challenging. @xcite found that specific attention heads are responsible for retrieving relevant information from a given context, referred to as retrieval heads. while identifying these mechanisms is key to understanding llms, little research has explored how to use these insights to effectively mitigate hallucinations, which is the focus of our work. we propose a novel decoding method termed decoding by contrasting retrieval heads (decore), as illustrated in figure 1 . decore builds on the assumption that masking retrieval heads can induce hallucination by impairing the ability of the model to retrieve relevant information from the context. it leverages contrastive decoding @xcite to amplify the differences between the original and the hallucinating outputs, leading to more accurate final responses. furthermore, we propose using the conditional entropy of the model's next-token distribution to control the contrastive decoding mechanism. our findings show decore significantly improves accuracy in contextual faithfulness tasks (xsum @xcite , memotrap @xcite , open-book natural questions (nq; @xcite , nq-swap @xcite ) and factual recall tasks. furthermore, our experiments show that decore improves the accuracy of the model in factual recall tasks. on triviaqa @xcite and popqa @xcite , decore outperforms other hallucination mitigation methods. when applied to truthfulqa @xcite , llama3-8b-instruct @xcite with decore generates more truthful and informative responses than comparable methods. finally, our experiments on musique @xcite , decore significantly improves accuracy in long-form generation and reasoning tasks, particularly when combined with chain of thought (cot; @xcite . input llm with masked retrieval heads base llm contrastive decoding chelsea arsenal fulham ✅ chelsea arsenal fulham predict chelsea arsenal fulham predict uncertainty estimation (entropy) high entropy contrast more low entropy contrast less context: the 2012 victory marked not only chelsea's first ever champions league, but also the first time a london team had won the competition question: which london-based football club was the first to win the champions league? figure 1 : overview of the decore workflow. given the same input, the base llm (llm base ) and the variant with masked retrieval heads (llm masked ) predict the next token. an uncertainty estimation is applied to the base model's output using conditional entropy: higher conditional entropy increases the contrastive factor (α), penalising predictions that align with the llm masked . the final prediction is selected based on weighted contrastive decoding of the outputs from both models, leading to a more grounded response. ## decore -decoding by contrasting retrieval heads decore works by masking specific retrieval heads to trigger hallucinations, followed by contrastive decoding to penalise outputs resembling those from the hallucinating model. this amplifies the more accurate predictions of the base model. we further enhance decore with a dynamic entropycontrolled mechanism to adjust the contrastive effect based on the entropy of the next token distribution of the model, as illustrated in figure 1 . ## experiment setup hallucinations in llms can be categorised into two types: factuality and faithfulness hallucinations @xcite . factuality hallucinations refer to instances where the generated content is factually incorrect. faithfulness hallucinations refer to instances where the generated content fails to adhere to the given source of information. we use diverse benchmarks to assess contextual faithfulness, factual accuracy, and multi-hop reasoning ability. given that retrieval heads are essential in correctly retrieving contextual information and looking back over long reasoning processes @xcite , our experimental setup is designed to answer the following key research questions: 1) can decore improve contextual faithfulness? 2) can decore maintain or enhance the factual recall capabilities of llms? 3) does coupling decore with cot improve the multi-hop reasoning capability of the llm? ## results in the following, we present the evaluation results of decore across faithfulness, factuality, and multi-hop reasoning tasks. we show that decore mitigates faithfulness and factuality hallucinations and improves the accuracy of the model when combined with cot prompting. these effectively an-swer our research questions stated in section 3. additionally, we examine the impact of the number of masked retrieval heads on task performance. finally, we demonstrate that decore reduces conditional entropy over time in long-generation tasks, contributing to more accurate outputs. ## conclusions decore (decoding by contrasting retrieval heads) is a novel decoding strategy that reduces faithfulness and factuality hallucinations in llms. decore masks retrieval heads to create a version of the model that is more likely to generate hallucinations and combines it with the original model via a contrastive decoding scheme (section 2.2). furthermore, we use the conditional entropy of the next-token distribution of the model to control the strength of the contrastive decoding scheme (section 2.3). our experimental results show that decore significantly improves the accuracy of the model in tasks requiring contextual faithfulness and in some factual recall and reasoning tasks.