DAMRO : Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024)

Abstract: Despite the great success of Large Vision-Language Models (LVLMs), they inevitably suffer from hallucination. As we know, both the visual encoder and the Large Language Model (LLM) decoder in LVLMs are Transformer-based, allowing the model to extract visual information and generate text outputs via attention mechanisms. We find that the attention distribution of LLM decoder on image tokens is highly consistent with the visual encoder and both distributions tend to focus on particular background tokens rather than the referred objects in the image. We attribute to the unexpected attention distribution to an inherent flaw in the visual encoder itself, which misguides LLMs to over emphasize the redundant information and generate object hallucination. To address the issue, we propose DAMRO, a novel training-free strategy that **D**ive into **A**ttention **M**echanism of LVLM to **R**educe **O**bject Hallucination. Specifically, our approach employs classification token (CLS) of ViT to filter out high-attention tokens scattered in the background and then eliminate their influence during decoding stage. We evaluate our method on LVLMs including LLaVA-1.5, LLaVA-NeXT and InstructBLIP, using various benchmarks such as POPE, CHAIR, MME and GPT-4V Aided Evaluation. The results demonstrate that our approach significantly reduces the impact of these outlier tokens, thus effectively alleviating the hallucination of LVLMs.

## introduction large vision-language models (lvlms) research @xcite @xcite has witnessed rapid advancement in the past few years, particularly demonstrating strong capabilities in visual reasoning tasks. however, lvlms still face significant challenges related to object hallucination @xcite , where the objects described in the generated text do not align with the visual ground truth of the input. this issue is prevalent across various models, posing a critical problem for the reliability and safety of lvlms @xcite . recently, the issue of object hallucination in lvlms has gained increasing attention. early work has tried many methods, such as optimizing the training and fine-tuning methods @xcite , incorporating external information or models, e.g. detr @xcite @xcite , providing feedback on hallucinated information and reprocesses @xcite . efforts also include llm decoding methods, like contrastive decoding @xcite and other novel decoding methods @xcite . these approaches mainly focus on improving the overall model architecture or specific modules within lvlms, such as the visual encoder or llm decoder. however, they often overlook the fundamental component of lvlms, the vision transformer (vit) structure @xcite , and its impact on the hallucination generation mechanism during the llm decoding stage. based on llava-1.5 @xcite , we explore the attention map in both the visual encoder and the llm decoder. we find outlier tokens in the attention map of both components, which are highly consistent with each other. these high-norm outlier tokens often contain globally redundant visual information @xcite . additionally, our analysis reveals a correlation between attention to these tokens and the occurrence of object hallucination. to address the aforementioned issue, we propose the dive into attention mechanism of lvlm to reduce object hallucination (damro) method, as illustrated in figure 1 . damro filters out high-norm outlier tokens from the vit attention please describe the image in detail. map, identifying them as negative tokens, and then projects them into the llm along with normal tokens. contrastive decoding is then applied to reduce the llm decoder's reliance on these tokens that contain globally redundant information and to enhance its focus on object-level details, thus mitigating model hallucination. our method is training-free and does not introduce external information or models. it outperforms similar approaches such as m3id @xcite and vcd @xcite in overall effectiveness. additionally, since vit is such a popular backbone of visual encoder @xcite that our approach demonstrates strong generalizability due to its utilizing on attention mechanism. in conclusion, our main contributions are summarized as follows: ## conclusions in this paper, we investigate the relationship between the attention maps of the visual encoder and the llm decoder, and explore its impact on the mechanism of object hallucination in lvlms. based on our analysis of attention mechanism, we propose the dive into attention mechanism to mitigate object hallucination (damro) method. our method demonstrates its effectiveness and generalizability on various models and benchmarks. experiments show that our method effectively reduces hallucination issues in lvlms across multiple domains, especially in fine-grained semantic hallucinations. additionally, we hope our findings on encoder-decoder attention mechanism will inspire further research on lvlm foundation model structures.