Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024)

Abstract: Large Multimodal Models (LMMs) often suffer from multimodal hallucinations, wherein they may create content that is not present in the visual inputs. In this paper, we explore a new angle of this issue: overly detailed training data hinders the model’s ability to timely terminate generation, leading to continued outputs beyond visual perception limits. By investigating how the model decides to terminate generation with EOS, the special end-of-sentence token, we find that the model assesses the completeness of the entire sequence by comparing the generated text with the image. This observation suggests that the model possesses an inherent potential of making proper EOS decisions based on its visual perception to avoid overly lengthy outputs. To take advantage of such potential, we explore two methods to mitigate multimodal hallucinations: a training objective that enables the model to reduce hallucinations by learning from regular instruction data, and a data filtering strategy to prevent harmful training data from exacerbating model hallucinations. Both methods significantly improve the hallucination performance of LMMs, without requiring any additional data or knowledge.

## introduction ever since large vision-language models (lvlms) @xcite were achieved through bridging vision encoders with large language models (llms) @xcite , they have been plagued by the problem of multimodal hallucinations, i.e., their text outputs may include unfaithful content to the visual inputs, such as non-existent objects @xcite , which greatly harms the reliability of their applications. extensive research has shed light on the origins of multimodal hallucinations, including the inability of vision encoders to represent finegrained visual details @xcite , model reliance on inherent parametric knowledge such as language priors and statistical biases @xcite , and pervasive hallucinations in the training data itself @xcite . in response to these insights, a variety of strategies have been proposed to mitigate hallucinations in lvlms @xcite @xcite . although significant progress has been made, in this paper, we highlight a crucial but often overlooked source of hallucinations: the excessively detailed training data. for example, in the detailed image captioning task, the caption data for an image typically integrates rich visual semantics from multiple human annotations or vision expert models, and is rewritten into lengthy paragraphs by llms @xcite , as shown in fig. 1 . these training data, while high-quality and meeting our expectations for detail, may exceed the visual perception capability of lvlms, especially for subtle image features such as small or easily confusable objects. when trained with such data, in an attempt to fit the detail level and length distribution of ground truth captions, the model may risk expressing details that it cannot discern from the image, and therefore exhibit hallucinations. ideally, models should be trained to terminate generation upon reaching their visual perception limits to avoid hallucinations. however, because gauging such limits is not trivial, it is difficult to provide explicit supervision to teach models to stop generation timely, and it is impossible to construct training data that well matches model capabilities. fortunately, we can draw inspiration from a closer examination of the model's decisions regarding the generation of the end-of-sentence (eos) token. we first employ a saliency-based method to analyze how information flows from the context to the target position where the model predicts the next word. we discover that in predicting the eos token, the model tends to rely more on all preceding sentences rather than just the current sentence. this leads to a hypothesis that the model assesses the completeness of the entire sequence when deciding whether to terminate the generation. then, by manipulating the context, we observe that the model's tendency to predict eos clearly varies depending on the semantic completeness of the generated text relative to the visual input. for instance, reducing visual context (easier to reach textual completeness) makes the model more likely to end the generation, whereas concealing textual context (further away from textual completeness) prompts continued generation. this confirms the hypothesis above and implies that such a completeness assessment is accomplished by comparing the generated text with the perceived visual information. these observations suggest that the model inherently holds the potential to make timely eos decision to terminate the generation based on its visual perception. when the model decides to end the generation, it indicates that the current generated context sufficiently captures the visual information it can perceive, and any further outputs may exceed the model's visual perception limits, possibly leading to hallucinations. to unlock such potential of models, we explore two approaches to enhance the model for better eos decisions. (1) a learning objective for model training, termed selective eos supervision. simply modified from the maximum likelihood es-timation (mle), it enables the model to mitigate hallucinations through learning from regular instruction data. it is applicable both for further training to reduce hallucinations in existing models, and for initial instruction tuning @xcite to alleviate the onset of hallucinations. specifically, by briefly further training on the original instruction data, the sentence-level and instance-level hallucinations of llava-1.5 @xcite are reduced by 26% and 27%, respectively. (2) a data filtering strategy based on scoring eos supervision, to eliminate harmful training data that can impair the model's ability to end sequences. we design two metrics to assess the positive and negative impact of data on the model's eos tendency, and combine them to rank and filter the training data. experimental results show that removing a small portion of the data can significantly reduce the hallucinations of models trained on it. these findings further validate our hypothesis and provide simple yet effective solutions for mitigating multimodal hallucinations in lvlms. ## eos decision in autoregressive language models, sequences are completed through continuous next-token prediction (ntp) . the termination of the process is achieved by introducing a special end-of-sentence (eos) token, denoted as v eos , into the vocabulary. at each ntp step, the model chooses between a regular content token and v eos , deciding whether to continue the sequence generation or end it, which we refer to as eos decision @xcite . in this section, we delve into how lvlms reach eos decisions. specifically, in section 2.1, we analyze the contextual information that the model relies on to predict v eos ; in section 2.2, we explore how the model adjusts its tendency to terminate generation with the multimodal input. corresponding findings are further discussed in section 2.3. ## mitigating multimodal hallucinations inspired by the preceding analysis, we propose two approaches to mitigate multimodal hallucinations: ## conclusion this paper investigates the multimodal hallucination issue in large multimodal models. we suggest that overly detailed training data can prevent the model from stopping generation at the appropriate time, thus leading to hallucinated outputs. by examining the model's inner behavior of eos prediction, we discover that the model inherently holds the potential to terminate generation based on its visual perception limits. to enhance such potential, we develop two approaches, a learning objective for training models and a data filtering strategy for selecting training data, both of which facilitate the model learning to timely terminate generation and significantly reduce hallucinations.