Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models (2025)

Abstract: Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD) , a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD’s effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.

## introduction large language models (llms) @xcite @xcite excel in generating fluent and contextually relevant responses. however, they often struggle with factual accuracy, especially when relying on external information. @xcite . retrieval-augmented generation (rag) @xcite mitigates this by grounding outputs in retrieved context, making it effective for tasks like question answering and reasoning @xcite . however, models often fail to faithfully utilize retrieved context, resulting in context faithfulness hallucinations, where outputs deviate from the retrieved context @xcite . these hallucinations undermine the reliability of rag systems, particularly in critical domains where factual accuracy is paramount @xcite † equal contribution. * corresponding author. this work was done during yanwen huang's internship at ping an technology (shenzhen) co., ltd., china. to better understand context faithfulness hallucinations and explore potential solutions, we take an internally-driven approach, analyzing intrinsic model signals that may explain why retrievalaugmented large language models struggle to utilize retrieved context faithfully @xcite . motivated by prior research linking token-level uncertainty to factual hallucinations @xcite , we examine whether token-level probability distribution entropy correlates with context faithfulness hallucinations. our analysis reveals a strong correlation between higher uncertainty and context faithfulness hallucinations. specifically, as shown in figure 1 (a1), wrong answers exhibit higher entropy in the tokenlevel probability distribution, indicating greater uncertainty in generation. even for correct answers (figure 1 (a2)), it often assigns low confidence to the highest-ranked token, suggesting incomplete integration of retrieved context. notably, in incorrect responses, most gold answer tokens appear in the top 10 in the token-level probability distribution, but are not assigned the highest probability (figure 1 (b) ), implying that the model identifies relevant context but fails to prioritize it effectively. these findings indicate that while models retrieve relevant context, they struggle to integrate and prioritize it during generation. since most gold answer tokens appear within the top 10, an effective strategy to mitigate context faithfulness hallucinations is to dynamically identify and prioritize these tokens during generation. this requires detecting reliable signals that indicate how retrieved context influences the model's predictions. attention mechanisms in transformer models naturally emerge as a key source of such signals, since they facilitate information flow between tokens @xcite @xcite . we hypothesize that attention distributions encode intrinsic indicators of context utilization. to validate our hypothesis, we trained a probing classifier using logistic regression on attention distributions, achieving over 0.99 auc in distinguishing contextually relevant tokens. even with just 100 training samples, the classifier demonstrated strong generalization across in-domain and cross-domain test sets, indicating that attention distributions inherently encode context utilization signals. these results support attention-based context utilization as a fundamental mechanism in llms. by leveraging these intrinsic signals, attention distributions provide a lightweight and interpretable means to assess how models integrate retrieved context into their predictions. motivated by these findings, we propose dynamic attention-guided context decoding (dagcd), a novel method to mitigate context faithfulness hallucinations. inspired by the copygenerator framework @xcite , dagcd integrates attention weights to estimate the relevance of tokens in the retrieved con-text, dynamically adjusting output probabilities. additionally, token-level uncertainty guides these adjustments by emphasizing underconfident yet contextually relevant tokens. by combining these strategies, dagcd ensures output alignment with the retrieval context and maintains efficiency. our contributions are as follows: comprehensive analysis of context faithfulness hallucinations: we identify a strong correlation between token-level uncertainty and context faithfulness hallucinations, showing that incorrect responses exhibit higher entropy and retrieved context is often recognized but not effectively prioritized. 2. attention-driven interpretability framework: we propose dynamic attention-guided context decoding (dagcd), leveraging attention distributions to amplify contextually relevant tokens and ensure faithful utilization of retrieved context. 3. lightweight and efficient decoding: dagcd operates in a single decoding pass, integrating attention signals and uncertainty measures without additional overhead, improving efficiency. 4. extensive validation across datasets and models: dagcd outperforms greedy decoding across multiple qa datasets, improving em by 17.67% on pretrained models and 2.25% on instruction-tuned models, demonstrating robustness and scalability. ## why can't generate faithful answers? token-level uncertainty is closely related to factual hallucinations, as models often exhibit higher entropy in the token-level probability distribution when generating factually incorrect outputs @xcite . while uncertainty measures help detect hallucinationprone predictions, most studies focus on factual hallucinations, where responses are incorrect without retrieved context. in contrast, context faithfulness hallucinations arise when models rely on retrieved information but generate misaligned or contradictory outputs. despite their significance in rag, their relationship with uncertainty remains unclear. inspired by prior research, we investigate whether unfaithful responses in rag exhibit higher entropy and whether contextually relevant tokens are recognized but assigned insufficient confidence. ## context utilization signal in attention our analysis in section 2 highlights that while models often identify relevant context tokens, they fail to assign them sufficient confidence, leading to context faithfulness hallucinations. to better understand this phenomenon, we seek reliable signals that indicate which retrieved context tokens are effectively utilized by the model during generation. due to their role in integrating and propagating information across tokens, attention mechanisms naturally emerge as a key candidate for capturing such signals @xcite @xcite . we hypothesize that attention distributions encode intrinsic indicators of context utilization, providing a lightweight and interpretable means to assess how models incorporate retrieved context into their outputs. ## method inspired by the copy-generation mechanism @xcite , we propose dynamic attention-guided context decoding (dagcd) to mitigate context faithfulness hallucinations. dagcd leverages utilization signals to dynamically guide the generation process, focusing on relevant contextual tokens. it integrates three steps: detecting utilized context tokens during inference ( §4.1), constructing a utilization distribution ( §4.2), and adjusting the token-level probability distribution to enhance contextual utilization ( §4.3). ## conclusion in this paper, we mitigate context faithfulness hallucinations in llms by proposing dynamic attention-guided context decoding, a lightweight framework that integrates attention distributions and entropy-based uncertainty signals to amplify contextually relevant tokens during generation. our analysis revealed a strong correlation between high uncertainty and hallucinations, and probing experiments validated that attention mechanisms encode signals indicative of contextual utilization, and further demonstrated that this signal is a fundamental mechanism in transformer-based llms. experiments across multiple open-book qa datasets demonstrated that dagcd achieves consistent improvements in context faithfulness, robustness, and scalability, providing an effective solution for context-sensitive generation tasks.