m DPO : Conditional Preference Optimization for Multimodal Large Language Models (2024)

Abstract: Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. Through a comparative experiment, we identify the unconditional preference problem in multimodal preference optimization, where the model overlooks the image condition. To address this problem, we propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. Moreover, we introduce a reward anchor that forces the reward to be positive for chosen responses, thereby avoiding the decrease in their likelihood—an intrinsic problem of relative preference optimization. Experiments on two multimodal LLMs of different sizes and three widely used benchmarks demonstrate that mDPO effectively addresses the unconditional preference problem in multimodal preference optimization and significantly improves model performance, particularly in reducing hallucination.

## introduction direct preference optimization (dpo) has emerged as the predominating method for aligning large language models (llms) with human preferences @xcite . building on its success in the language modality, recent studies have extended dpo to multimodal scenarios @xcite @xcite . however, transferring this approach across modalities presents significant challenges. merely substituting textual preference data with multimodal preference data does not consistently yield positive outcomes and can exacerbate issues such as hallucinations @xcite . while recent efforts in multimodal preference learning focus on improving performance through figure 1 : we train bunny-v1.0-3b @xcite on 10k multimodal preference data from silkie @xcite with different variants of dpo. we perform dpo (no image) where all images are removed from the preference data. counterintuitively, the overall score on the mmhalbench @xcite for dpo (no image) is similar to that of dpo with images. this finding suggests that dpo may suffer from unconditional preferences, neglecting the visual modality during optimization. our proposed method, mdpo, effectively addresses this issue and improves model performance. in this paper, we propose mdpo, a multimodal dpo objective that utilizes conditional preference optimization on images to prevent the overly prioritization of language-only preferences. as depicted in fig. 2 , in addition to the original preference pairs contrasting responses, mdpo introduces new preference pairs contrasting images. the rejected image is derived from the original (i.e., chosen) image by reducing effective visual information. this approach, combined with the standard dpo objective, encourages the multimodal llm to simultaneously emphasize both visual and language features. furthermore, we observe that dpo often experiences a decrease in the likelihood of chosen responses in multimodal scenarios despite an increase in the implicit reward. to address this, mdpo incorporates a reward anchor that maintains the likelihood of chosen responses by regularizing the reward to be positive. to validate the effectiveness of mdpo, we conduct experiments using bunny-v1.0-3b @xcite and llava-v1.5-7b @xcite . both automatic and human evaluations on mmhalbench @xcite , object halbench @xcite , and amber @xcite consistently demonstrate that mdpo outperforms standard dpo in multimodal scenarios, effectively reducing hallucinations across varying model and data scales. detailed analyses reveal that conditional preference plays a crucial role in enhancing the effectiveness of dpo for multimodal llms. fine-grained and qualitative studies further illustrate that mdpo significantly improves the model's ability to comprehend images and mitigates language biases in model responses. our contributions are three-fold. first, we identify unconditional preference towards the visual modality as a primary reason for the pitfalls of dpo in multimodal llms. second, we propose mdpo, a multimodal dpo objective that incorporates conditional preference optimization and anchored preference optimization to mitigate these pitfalls. third, we verify the effectiveness of mdpo across different model and data scales. using mdpo, we achieve the best-performing 3b multimodal llm in terms of reducing hallucinations. figure 3 : qualitative results from mmhalbench. top: when trained with standard dpo, bunny often assumes the image description in the question is correct, responding accordingly, even if the question contains an adversarial premise regarding the image. in contrast, mdpo identifies the false premise in the question by referencing the image. bottom: bunny trained with standard dpo may disregard the image and provide an educated guess for the answer. conversely, mdpo delivers a correct answer that is conditioned on the image. ## mdpo in this section, we introduce mdpo, an improved dpo approached dedicated to multimodal preference alignment. as depicted in fig. 2 , mdpo introduces two additional preference optimization objectives to dpo: conditional preference optimization to address the issue of ignoring visual information (see §3.1), and anchored preference optimization to prevent a decrease in the likelihood of the chosen response (see §3.2). ## experiment in this section, we begin with the experimental setup ( §4.1). then we present the main results on three benchmarks ( §4.2) and the human evaluation ( §4.3) of mdpo. we further provide in-depth analysis ( §4.4) and fine-grained results ( §4.5). finally, we conduct a qualitative study ( §4.6). ## related work reinforcement learning from human feedback (rlhf; @xcite ) has proven to be an effective approach for aligning llms with human values. direct preference optimization (dpo; @xcite , which involves directly optimizing llms based on human preferences, has been widely adopted in rlhf due to its strong performance and the elimination of the need for a separate reward model. significant efforts have been made to further enhance the efficacy and efficiency of dpo, which can be categorized into algorithmic and data-related advancements. on the algorithmic side, various approaches aim to improve the efficiency of dpo. for example, orpo @xcite ) models preferences using an odds ratio and combines instruction fine-tuning and preference optimization into a unified training process. methods such as cpo @xcite , tpo @xcite , and simpo @xcite simplify dpo by eliminating the use of a reference model, thereby reducing computational and memory overhead. additionally, @xcite addresses the issue of reward overfitting in dpo. on the data side, approaches such as kto @xcite and nca @xcite seek to overcome dpo's requirement for paired preference data by designing optimization goals that can also utilize unpaired data. iterative dpo @xcite @xcite and sppo @xcite propose sampling preference data in an on-policy manner, achieving better results than off-policy dpo. wpo @xcite adapts off-policy data to resemble on-policy data by reweighting preference pairs. more closely related to our work, studies by @xcite and @xcite address the reward hacking problem in textual preference optimization, where human preference may be biased towards longer outputs. they propose to calibrate the rewards in dpo with respect to output length. in this work, we discover a novel challenge in multimodal dpo, where preference optimization often neglects images. we then propose a solution to this problem through conditional preference optimization. in multimodal scenarios, recent works mainly focus on creating multimodal preference data @xcite @xcite @xcite @xcite . these efforts include collecting human preference @xcite , preference from advanced multimodal llms @xcite , and preference from the model to align itself @xcite . in terms of learning objectives, recent works mainly follows dpo for llms @xcite @xcite . some also apply reinforcement learning @xcite and contrastive learning @xcite . our work studies an overlooked but crucial problem in the multimodal dpo objective. ## conclusion we propose mdpo, a preference optimization method dedicated to multimodal scenarios. mdpo leverages conditional preference optimization to encourage multimodal llms to capture preference labels based on both visual and language cues. it further introduces anchored preference optimization to prevent the likelihood of preferred responses from decreasing. experiments show that mdpo consistently enhances multimodal llm performance and reduces hallucination across different model sizes on three widely used benchmarks. our method could be extended to reduce hallucinations and enhance trustworthiness across broader multimodal data and scenarios, including multiple images @xcite , videos @xcite , in-context learning @xcite , and risk-sensitive domains @xcite .