Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models (2025)

Abstract: Large Vision-Language Models (LVLMs) have exhibited impressive capabilities across various visual tasks, yet they remain hindered by the persistent challenge of hallucinations. To address this critical issue, we propose Mixture of Decoding (MoD), a novel approach for hallucination mitigation that dynamically adapts decoding strategies by evaluating the correctness of the model’s attention on image tokens. Specifically, MoD measures the consistency between outputs generated from the original image tokens and those derived from the model’s attended image tokens, to distinguish the correctness aforementioned. If the outputs are consistent, indicating correct attention, MoD employs a complementary strategy to amplify critical information. Conversely, if the outputs are inconsistent, suggesting erroneous attention, MoD utilizes a contrastive strategy to suppress misleading information. Extensive experiments demonstrate that MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs. Code is available at https://github.com/xlchen0205/MoD.

## introduction recently, large language models (llms) @xcite have achieved remarkable success, which has spurred significant research interest in extending them into large vision-language models (lvlms) @xcite . these lvlms demonstrate exceptional performance across various visual tasks. however, despite their impressive versatility, lvlms are confronted with a critical limitation known as "hallucination" @xcite @xcite , a phenomenon in which models generate text that appears semantically coherent, but contains content that either contradicts or lacks grounding in the provided visual information. this fundamental flaw significantly compromises the reliability of lvlms as trustworthy ai assistants in realworld applications @xcite , particularly in high-stakes domains such as autonomous driving systems @xcite , where inaccurate interpretation of visual data could lead to severe consequences. a prominent approach to mitigating hallucinations in lvlms is developing training-free decoding strategies, particularly contrastive decoding. this method works by contrasting the original logits for next-token prediction with those generated under hallucination scenarios, thereby producing more reliable outputs. specifically, vcd @xcite mitigates the language prior bias inherent in the llm backbone by contrasting the original logits with those generated from gaussiannoised image. similarly, m3id @xcite reduces language prior bias by contrasting the original logits with those derived from pure text inputs, strengthening the interaction between visual inputs and model outputs. avisc @xcite proposes a novel perspective, suggesting that certain image tokens with excessively high attention weights may trigger hallucinations. thus, it selects high-attention image tokens from specific layers to obtain hallucination logits for contrastive decoding. despite these advancements, existing methods still exhibit notable limitations. vcd and m3id primarily attribute hallucinations to language prior bias, neglecting the potential influence of visual inputs, such as spurious correlations @xcite . although avisc considers the model's attention distribution over the input image, it may erroneously weaken the role of high-attention image tokens when the model has already accurately focused on relevant information, leading to unreliable contrastive results. these limitations underscore a critical gap in current methods: the insufficient consideration of both (1) the influence of visual inputs on hallucination and (2) the variability of the attention distribution, which may be either right or wrong, ultimately restricting their performance in complex scenarios. in the right figure, there is a significant positive correlation between the chair i metric and js divergence, with a pearson correlation coefficient of 0.85 (@xmath0.01). the key to addressing these limitations lies in assessing the correctness of the model's attention on image tokens and dynamically adjusting the decoding strategy accordingly. if the model's attention is correct, indicating that the attended image tokens are relevant and beneficial, amplifying their information can enhance the output probability of proper tokens; conversely, if the model's attention is incorrect, suggesting that the attended image tokens are irrelevant or misleading, suppressing erroneous information is crucial to counteract hallucinations. inspired by methods that judge correctness based on consistency @xcite , we observe that measuring the consistency between the original output logits and the logits derived from the model's attended image tokens using jensen-shannon (js) divergence, can effectively distinguish hallucinated outputs from nonhallucinated ones (as illustrated in fig. 1 ). this approach enables us to determine the correctness of the model's attention on image tokens and adaptively select decoding strategies to effectively mitigate hallucinations. building on these insights, we propose mixture of decoding (mod), a method that adaptively selects decoding strategies by evaluating the consistency between the original output and the output derived from the model's attended image tokens. specifically, if the outputs are consistent, it indicates that the model's attention on image tokens is correct, and we complement the two to amplify critical information. conversely, if the outputs are inconsistent, it suggests that the model's attention on image tokens is incorrect, and we contrast the two to suppress misleading information derived from erroneous attention. extensive experiments on multiple mainstream benchmarks demonstrate that mod significantly outperforms existing methods in mitigating hallucinations. the main contributions of this paper are summarized as follows: ## conclusion in this paper, we propose mixture of decoding (mod), an adaptive decoding strategy designed to mitigate hallucinations in lvlms. by evaluating the consistency between the outputs generated from the original image tokens and those derived from the model's attended image tokens, mod initially assesses the correctness of the model's attention over image tokens. subsequently, informed by this evaluation, mod dynamically selects either the complementary or contrastive decoding strategy to optimize the decoding process. a series of comprehensive experiments have been conducted to validate the effectiveness and robustness of mod, highlighting its potential as a promising solution to enhance the accuracy and reliability of lvlms across diverse applications.