Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (2025)

Abstract: Multi-modal large language models (MLLMs) integrate the inherent text generation capabilities of large language models with an understanding of other modalities, promising wide applications in open-ended tasks. Despite their success, they often generate plausible but incorrect content. This phenomenon, known as hallucination, significantly impacts their practical deployment. In this paper, we delve into the intrinsic characteristics of hallucination from the perspective of interaction between input and output tokens. We find that the hallucination typically occurs with attention reduction of output tokens to image tokens. Based on this observation, we introduce image Token attention-guided Decoding (iTaD), a plug-and-play method which leverages MLLMs’ internal representations to mitigate their hallucinations. We first define an image token attention vector to measure the inter-layer differences in attention of output tokens to image tokens across different layers. Based on the vector, we design a novel layer selection strategy and conduct inter-layer contrastive decoding to highlight the progression in image understanding, thereby exploiting attention to image tokens to mitigate hallucinations. Extensive experiments well demonstrate iTaD’s effectiveness across different MLLMs and benchmarks.

## introduction multi-modal large language models (mllms) @xcite @xcite process inputs from language and other modalities to generate open-ended responses. recent advancements in mllms, such as llava-1.5 @xcite , instructblip @xcite , and @xcite , have demonstrated their outstanding performance in a variety of visual tasks, such as object detection @xcite , image captioning @xcite , visual question answering @xcite , etc. despite their success, mllms usually suffer from a severe hallucination issue @xcite @xcite , which refers to the phenomenon where mllms generate grammatically coherent yet factually incorrect content. such issue should be carefully managed in scenarios like healthcare @xcite , autonomous systems @xcite , and robotics @xcite , where the presence of incorrect outcomes is unacceptable and potentially disastrous. therefore, mitigating hallucinations in mllms is crucial for enhancing their practical deployment and reliability in real-world scenarios. many efforts have been made to mitigate hallucinations in mllms @xcite @xcite @xcite . early works typically employ additional training datasets or external knowledge bases @xcite @xcite . despite the exploitative depth and effectiveness, they necessitate substantial human and computational costs. notably, recent studies have leveraged the inherent knowledge or internal representations of llms to mitigate their hallucinations during inference @xcite . following this trend, some studies attempt to develop training-free mllm hallucination mitigation methods @xcite @xcite . those studies mainly focus on applying perturbations to the input text or images from the input side @xcite or on identifying internal representations of the output tokens associated with hallucinations from the output side @xcite . despite their success, they rarely explore the interaction between output and input tokens. moreover, as the most critical characteristic distinguishing mllms from llms, the image please describe this image in detail. ## method in this section, we first briefly introduce inter-layer contrastive decoding, and then detail the proposed method itad. ## conclusion and future work in this paper, we introduce itad, a plug-and-play method to mitigate hallucinations in mllms. itad is motivated by the observation that hallucinatory outputs in mllms typically occur with attention reduction to image tokens. to address this, we first define the image token attention vector (itav) to measure the distance in image understanding across different layers. then we leverage the inherent layer-level progression of mllms to extract and highlight the improvement in image understanding and derive the output token distribution p, thus exploiting the attention to image tokens to mitigate hallucinations. extensive experiments demonstrate that itad achieves state-of-the-art results and consistently exhibits superior performance across different models, datasets, and benchmarks. this paper focuses on hallucinations in visual and language modalities, where most research has concentrated. our future work will explore hallucinations in other modalities, such as video and audio. given the complexity of their origins, further research is critical for a more thorough understanding of the underlying causes.