Entropy-Based Decoding for Retrieval-Augmented Large Language Models (2025)

Abstract: Augmenting Large Language Models (LLMs) with retrieved external knowledge has proven effective in improving the factual accuracy of generated responses. Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and internal knowledge sources. In this paper, we introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue. Our approach utilizes entropy-based document-parallel ensemble decoding to prioritize low-entropy distributions from retrieved documents, thereby enhancing the extraction of relevant information of context. Additionally, it incorporates a contrastive decoding mechanism that contrasts the obtained low-entropy ensemble distribution with the high-entropy distribution derived from the model’s internal knowledge across layers, which ensures a greater emphasis on reliable external information. Extensive experiments on open-domain question answering datasets demonstrate the superiority of our method.

## introduction in recent years, large language models (llms) have revolutionized natural language processing, showcasing remarkable performance across various downstream tasks @xcite @xcite . however, they still struggle with hallucination due to the inaccuracy of parametric memory @xcite and inherently tend to produce outdated information @xcite . in contrast, explicitly augmenting llms with retrieved external knowledge from reliable datastores @xcite can enable llms to generate content that exhibits less deviation from the truth, and benefit downstream knowledge-intensive tasks @xcite . despite the success of retrieval-augmented llms, the augmented generation is still suboptimal due to the distractibility issue, where the generated responses are easily negatively affected by noise from both external knowledge and intrinsic model knowledge. as for the input context, llms' understanding of context can be explicitly distracted by irrelevant parts within the retrieved context @xcite . a typical illustrative case is the "lost in the middle" distraction phenomenon observed in the synthetic multi-document question-answering scenario @xcite , where the oracle document containing the correct answer is encircled by numerous retrieved distracting documents. in this scenario, llms frequently fail to deliver the correct answer unless the oracle document is strategically placed at the very beginning or end of the context. with regard to intrinsic knowledge, llms are easily implicitly distracted by the parametric knowledge acquired during pre-training. this is in conflict with retrieval-augmented generation which is expected to generate responses based on reliable retrieved context. particularly in the domain of question answering, previous works @xcite show that llms stubbornly adhere to their built-in knowledge even when it conflicts with external knowledge. how to eliminate the impact of the abovementioned distractibility issue, so as to extract useful knowledge from the retrieved context for the input query, is our research focus. although existing works strive to effectively leverage the retrieved context by directly fine-tuning retrievalaugmented llms @xcite @xcite or incorporating trainable encoder modules @xcite , these approaches require additional training, rendering them potentially impractical in resource-constrained environments. in this paper, we propose a novel decoding method guided by entropy considerations to simultaneously mitigate the impact of noisy information from both the external context and parametric knowledge. the proposed method can be seamlessly integrated into llms without requiring additional tuning. specifically, to enhance llms' ability to extract useful information from multiple retrieved documents, we let llms process each retrieved document in parallel and ensemble of the output distributions from each document to determine the next-token distribution, with the ensemble weights adaptively assigned based on the uncertainty of each document-conditioned distribution. at each generation step, documents with lower uncertainty (i.e., lower entropy) in the llm output are given more attention during decoding. ultimately, we obtain a low-entropy distribution aggregated among documents. furthermore, to alleviate the potential distraction from parametric knowledge, we refine the next-token distribution by contrasting the obtained low-entropy distribution when feeding the retrieved documents, against the distribution without context. here, we propose to use the distribution from the layer exhibiting the highest entropy without context for contrast, in order to highlight the proportional changes in token probabilities after introducing external knowledge. the proposed decoding method shows an impressive performance in the synthetic challenging multi-document scenario @xcite where the negative impact of retrieved distractor documents is emphasized. we further conduct extensive experiments across four llms of varying sizes on four diverse open-domain question answering tasks including nq @xcite , trivi-aqa @xcite , webq @xcite and popqa @xcite . experimental results confirm the superiority of our methods and validate the effectiveness of each component. ## methodology in this work, we will investigate a new decoding approach for retrieval-augmented generation (rag). given an input query x, rag first retrieves top-k relevant documents d := {d 1 , d 2 , . . . , d k } from the knowledge base via a retriever as external evidence, which is then incorporated with the query as the input to large language model parametrized by θ for generating a faithful response y. despite the integration of external knowledge, the decoding manner does matter to achieve desirable performance in rag. a common approach, termed as "naive rag", involves concatenating the query x, the previously generated response @xmath0, and the retrieved documents d as the input sequence, resulting in the following decoding method ## experiments baselines. we mainly compare three trainingfree baselines, two of which are ensemble-based methods. naive: this method concatenates all retrieved documents directly along with the question to form the prompt for llms; replug @xcite : it utilizes a normalized retriever weight to ensemble during the decoding process; avgens : it follows the formulation in eq.( 2 ) and assigns the same weight to each document during each generation step. in § 4.2.2, we also incorporate two contrastive decoding methods (i.e., cad @xcite and dola @xcite ) into our setting and compare with them to validate the effectiveness of our method. experiments are conducted on 4 llms: llama-2-7b, llama-2-13b, mistral-7b-v0.1, and llama3-8b. among them, llama-3-8b supports an 8k context length, while the other three llms support a 4k context length. few evaluation samples exceed 4k in context length, and they are removed to ensure the validity of the assessment. ## conclusions in this paper, we proposed a novel decoding method that is guided by entropy considerations to mitigate the distractibility issue from both external retrieved documents and parametric knowledge. first, we conducted parallel retrieval augmentation with entropy-based ensemble weight to obtain the low-entropy distribution of context. furthermore, we contrasted this distribution against the highestentropy distribution among layers when without context to amplify the external knowledge preserved in context. extensive experiments showed the proposed method's effectiveness in retrievalaugmented open-domain question answering.