M ask CD : Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding (2025)

Abstract: Large vision-language models (LVLMs) have shown remarkable performance in visual-language understanding for downstream multimodal tasks. While their capabilities are improving, problems emerge simultaneously. Among those problems, the hallucinations have attracted much attention, which stands for the phenomenon where LVLMs generate contradictory content to their input visual and text contents. Many approaches have been proposed to deal with this issue, such as contrastive decoding and attention manipulation. However, contrastive decoding methods struggle in constructing appropriate contrastive samples, and attention manipulation methods are highly sensitive, lacking stability. In this work, we propose image head Masked Contrastive Decoding (MaskCD). Our approach utilizes the “image heads” in LVLMs, masking them to construct contrastive samples for contrastive decoding. We evaluated MaskCD on LLaVA-1.5-7b and Qwen-VL-7b, using various benchmarks such as CHAIR, POPE, AMBER and MME. The results demonstrate that MaskCD effectively alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs. Corresponding resources could be found at: https://github.com/Deng-Jingyuan/MaskCD.

## introduction large language models (llms) @xcite have achieved remarkable success in understanding human instructions and performing diverse tasks. building on this progress, recent efforts have extended llms to develop large vision-language models (lvlms) @xcite @xcite @xcite , which integrate visual and textual modalities for multimodal reasoning. although researchers have already achieved remarkable success in applying * corresponding author lvlms into several tasks, problems have emerged as well. within these problems, the hallucination @xcite @xcite has attracted significant attention. the hallucination of lvlms is a phenomenon in which models tend to generate contradictory contents for the inputs, especially images. this may manifest as generating non-existent objects, mistakenly described attributes, or non-sense sentences. all kinds of hallucinations enormously lower users' trust in the model and even cause fatal damage when applied in real-world tasks like auto-driving and medical image processing. to mitigate the hallucination phenomenon, researchers have promoted multiple methods, which could be classified into two main categories according to whether training is needed. firstly, training-involved methods @xcite @xcite collect massive delicately-constructed data to fine-tune or posttrain the lvlms, so as to teach models to generate less hallucinated content. it features with a highcost of computation resources and massive human labor. on the contrary, training-free methods are developed to alleviate hallucination at a lower cost. prevailing methods include contrastive decoding (cd) @xcite @xcite and attention manipulation @xcite @xcite . cd methods need an injured input as the bad sample, whose output logits will be subtract from the original one. it would take twice times of inference cost, once for the original input, the other for the injured one. but with delicately constructed bad samples, cd methods have presented prominent performance in mitigating the lvlms' hallucination phenomenon. recently, with the progress in understanding models' inner working mechanisms, methods of attention manipulation have come up. specifically, abnormal attention map phenomena, such as excessive local attention in the attention sink phenomenon @xcite , will be directly reduced or redistributed. this method features in better align visual and text modal, enabling models to better and truly utilize visual information. although the need for training has been eliminated, both the cd and attention manipulation methods have their drawbacks. the performance of cd methods is heavily depended on the quality of the constructed bad sample. if the injured sample still contains a lot of useful information, then the contrast operation may even cause worse results. for the attention manipulation methods, models are highly sensitive to changes in the attention score and are not as stable as cd methods in terms of overall testing scenarios. therefore, we hope to construct high-quality bad samples through the angle of attention distribution, thereby combining the advantages of cd methods and attention manipulation methods, balancing stability and hallucination-mitigating performance. to observe the model's attention preference to images at head-level, we randomly select 500 images in the validation set of coco 2014 @xcite and 500 dall-e generated artificial images from mmrel @xcite . put them into llava-1.5-7b @xcite with the prompt text "please describe this image in detail." and record the sum of the attention scores obtained by each head in each layer of the model for each token generation. finally, under different thresholds, the number of times each head pays excessive attention to the image token during the generation of each token is calculated. the normalized result is visualized in figure 2 . we observed that whether real-world images or ai-generated images, there are certain heads in llava-1.5-7b that prefer to give image tokens comparably high attention scores. since they present an inclined focus on visual information, we name them "image heads". given that the essence of cd methods is making the subtracted samples contain only invalid information as much as pos-sible, we choose to mask these image heads to construct bad samples, so as to prevent the bad samples from accessing useful visual information more precisely. in this way, we proposed image head masked contrast decoding (maskcd), which uses image head attention masks to construct delicate bad samples and attain significant hallucination-mitigating performance. our contributions can be summarized as follows: ## result and analysis amber mme ## conclusion in this paper, we first introduce the image heads: the heads in lvlm's llm backbone that tend to pay comparably high attention proportion on image tokens. then we propose the image head masked