Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024)

Abstract: Though advanced in understanding visual information with human languages, Large Vision-Language Models (LVLMs) still suffer from multimodal hallucinations. A natural concern is that during multimodal interaction, the generated hallucinations could influence the LVLMs’ subsequent generation. Thus, we raise a question: \textit{When presented with a query relevant to the previously generated hallucination, will LVLMs be misled and respond incorrectly, even though the ground visual information exists?} To answer this, we propose a framework called \\textit{MMHalSnowball} to evaluate LVLMs’ behaviors when encountering generated hallucinations, where LVLMs are required to answer specific visual questions within a curated hallucinatory conversation. Crucially, our experiment shows that the performance of open-source LVLMs drops by at least 31\\% , indicating that LVLMs are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. We term this \textit{Multimodal Hallucination Snowballing} . To mitigate this issue, we further propose a training-free method called \textit{Residual Visual Decoding}, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with direct access to the visual information. Experiments show that our method can mitigate more than 24\\% of the snowballed multimodal hallucination while maintaining capabilities.

## introduction large vision-language models (lvlms) have shown remarkable abilities in observing and understanding the real world in human languages @xcite @xcite @xcite . however, multimodal hallucinations, in which lvlms provide where am i now? please describe the scene in detail. ## evaluating the multimodal hallucination snowball phenomenon in this section, we design a question-answer task in the conversation scenario, where a model is first asked to describe a picture in detail and then answers a visual question. as shown in figure 3 , we propose the mmhalsnowball framework to carefully simulate hallucinatory conversations and evaluate whether the model generates a wrong answer due to the hallucinatory context. next, we will describe our evaluation framework in detail, including conversation creation, experimental settings, and evaluation metrics. we experimentally analyze the multimodal hallucinations snowball in §2.7. the prompts used are listed in appendix a.2. ## residual visual decoding from the phenomenon of multimodal hallucination snowballing, we find that lvlms tend to condition on text context when there are plausible clues to help make responses, thereby ignoring the visual information and could be easily misled by erroneous context. to remedy this, we manage to emphasize the visual information during the inference process without additional training or external tools under the multi-turn conversation scenario. ## conclusion in this paper, we raise the question of whether lvlms suffer from multimodal hallucination snowballing. we meticulously designed the mmhalsnowball framework to simulate hallucinatory conversations and study models' behaviors when encountering hallucinations. our investigation proved that lvlms are being severely affected by hallucinations in the context, thus generating snowballed hallucinations. further, we proposed the residual visual decoding to alleviate the multimodal hallucination snowballing while maintaining its contextual abilities. however, our methods still have limitations when deployed to a generalpurpose assistant, which we left as future works.