Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization (2025)

Abstract: Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications. Recent works have attempted to apply Direct Preference Optimization (DPO) to enhance the performance of MLLMs, but have shown inconsistent improvements in mitigating hallucinations. To address this issue more effectively, we introduce Hallucination-targeted Direct Preference Optimization (HDPO) to reduce hallucinations in MLLMs. Unlike previous approaches, our method tackles hallucinations from their diverse forms and causes. Specifically, we develop three types of preference pair data targeting the following causes of MLLM hallucinations: (1) insufficient visual capabilities, (2) long context generation, and (3) multimodal conflicts. Experimental results demonstrate that our method achieves superior performance across multiple hallucination evaluation datasets, surpassing most state-of-the-art (SOTA) methods and highlighting the potential of our approach. Ablation studies and in-depth analyses further confirm the effectiveness of our method and suggest the potential for further improvements through scaling up.

## introduction large language models (llms) have been verified in various fields, demonstrating their potential @xcite @xcite , while they encounter challenges such as hallucination. multimodal large language models (mllms) are also known to hallucinate @xcite . specifically, they often produce unfaithful content that does not align with the visual input, which undermines their reliability and practicality, particularly in critical applications such as * equal contribution † corresponding author autonomous driving @xcite or medical tasks @xcite . hence, addressing mllm hallucination (m-hallu) is essential. recently, some pioneer preference optimization methods like direct preference optimization (dpo) @xcite have been introduced, which encourages the model to learn from comparisons between positive and negative samples, alleviating hallucinations @xcite . however, most current methods cannot deliver consistent improvements across all types of m-hallu tasks (e.g., vqa and captioning tasks, as shown in our experiments of different from the hallucinations in llms, mhallu primarily arises from the following three aspects: (1) insufficient visual capability: this occurs when the mllm's visual encoder lacks the necessary strength, being distracted by relatively unimportant visual information, leading to hallucinations; (2) incapable long-context generation: we observe that hallucinations become more pronounced as the generated content grows longer, similar to long-range forgetting, which needs to be addressed in practical applications; (3) multimodal conflicts: multimodal conflicts frequently arise in real-world scenarios due to the inevitable noises in texts and images. mllms are more prone to hallucinations with conflicting information existing between text and image @xcite . to address the aforementioned challenges, we propose hallucination-targeted direct preference optimization (hdpo) to mitigate m-hallu. our approach constructs hallucination-targeted preference pairs, specifically designed to address various forms and causes of hallucinations. specifically, we design three types of dpo data reflecting the corresponding hallucination causes as follows: (1) for insufficient visual capability, during the model's autoregressive decoding, we preserve only some visual tokens with the lowest attention scores to produce targeted negative responses that reflect incorrect visual information distraction, urging mllms to pay attention to more effective visual information. (2) for incapable long context generation, we specifically select positive examples from high-quality long-form captions, while creating negative examples where the latter part of the response deviates from the image content, simulating long-form hallucinations. (3) for multimodal conflicts, we add conflicting information with images into prompts to generate negative examples. we provide positive and negative pairs with questions featuring conflicting prefixes to train the model to correctly respond to the question even containing conflicting information. we conduct extensive experiments to evaluate our approach across various types of m-hallu tasks. the results demonstrate that our hdpo framework achieves the overall best performance in effectively mitigating mllm hallucinations on various tasks. our contributions are summarized as follows: preference optimization on mllms. recently, preference optimization like dpo has been used to enhance models. ha-dpo @xcite views hallucinations as models' preferences. by leveraging chatgpt @xcite alongside ground truth annotations from existing image datasets, it generates positive examples aligned with image content, while the model's original outputs serve as negative examples for direct preferences optimization. although effective, the construction of negative examples is suboptimal, as it may not fully capture the diverse forms of mhallu. seva (zhu et al., 2024) generates negative examples by introducing noise into images and treats the model's original outputs as positive examples, constructing pairs for dpo. in addition to adding noise, bpo (pi et al., 2024) injects errors into positive examples via the llm backbone of mllms to construct negative examples. however, our experiments indicate that while these methods demonstrate strong capabilities, their performance in hallucination-related evaluations is not particularly impressive. nonetheless, these works demonstrate the superiority of dpo in enhancing models' capabilities. inspired by these findings, we aim to develop methods to further mitigate m-hallu from its diverse forms with hallucination-targeted direct preference optimization. additionally, there exist other dpo-based methods that differ in application scenario: mia-dpo @xcite for multi-image, videodpo @xcite for video, and symdpo @xcite for in-context learning. all these works bring valuable insights to our community. hdpo differs from existing methods. unlike other existing preference optimization approaches, we primarily focus on hallucination-targeted preference optimization. we analyze and address hallucinations in mllms from diverse causes and forms. during the preference optimization process, the model learns to distinguish between positive and negative examples. ha-dpo enables the model to be aware of hallucinated content in its original outputs, though its effectiveness is limited to capturing the diverse range of hallucinations as the data is insufficient. in contrast, other works use general preference data, which improves overall model capability but shows inconsistency across different hallucination benchmarks. therefore, we aim to enhance the effectiveness of dpo by constructing examples that reflect a wider range of hallucination forms and characteristics, allowing the model to align better to make fewer hallucinations. ## experiments in this section, we empirically investigate the evaluation of hdpo. we begin by describing the experimental settings, including the evaluation datasets and training details. next, we present the results on various hallucination evaluation datasets, demonstrating the promising performance of hdpo. additionally, we validate the expected functions of lch and mch. finally, we provide ablation studies and conduct in-depth analyses in more detail. ## conclusions in this paper, we present hdpo, a novel approach designed to effectively mitigate hallucinations in mllms. we analyze three types of hallucinations observed in mllms and create hallucination preference data based on the identified causes. extensive experiments across different benchmarks demonstrate the ability of hdpo to effectively reduce hallucinations in mllms.