V - DPO : Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization (2024)

Abstract: Large vision-language models (LVLMs) suffer from hallucination, resulting in misalignment between the output textual response and the input visual content. Recent research indicates that the over-reliance on the Large Language Model (LLM) backbone, as one cause of the LVLM hallucination, inherently introduces bias from language priors, leading to insufficient context attention to the visual inputs.We tackle this issue of hallucination by mitigating such over-reliance through preference learning. We propose Vision-guided Direct Preference Optimization (V-DPO) to enhance visual context learning at training time. To interpret the effectiveness and generalizability of V-DPO on different types of training data, we construct a synthetic dataset containing both response- and image-contrast preference pairs, compared against existing human-annotated hallucination samples. Our approach achieves significant improvements compared with baseline methods across various hallucination benchmarks. Our analysis indicates that V-DPO excels in learning from image-contrast preference data, demonstrating its superior ability to elicit and understand nuances of visual context. Our code is publicly available at https://github.com/YuxiXie/V-DPOhttps://github.com/YuxiXie/V-DPO.

## introduction recent advancements in large language models (llms) @xcite @xcite @xcite have catalyzed the evolution of large vision-language models (lvlms) @xcite @xcite in understanding and reasoning across visual and textual modalities. despite their impressive performance on various vision-language tasks, existing lvlms still struggle with the issue of hallucination, where the model outputs are not factually grounded in the input visual contents @xcite what is the little boy eating? a plate of worms. a slice of pizza. ## related work hallucination has emerged as a significant challenge to model reliability and generalizability in lvlm development. to alleviate hallucinated content, existing works can be divided as following two directions. the first focuses on post-processing approaches, including post-hoc corrections @xcite @xcite and specialized decoding @xcite . however, these methods often require increased inference time, obstructing their generalizability and scalability across diverse data domains and sizes @xcite . the second line of work attempts to collect hallucination-aware data to mitigate hallucination in lvlms through preference optimization leaning toward hallucination-free outputs. for example, sun et al. (2023) and yu et al. (2023 ) adapt the reinforcement learning from human feedback (rlhf) and direct preference optimization (dpo) paradigms in llms, respectively, to align lvlms with hallucination-aware human preferences. zhao et al. (2023) and @xcite propose data augmentation pipelines to construct (accurate, hallucinatory) preference pairs for contrastive tuning. our work mitigates hallucination in the context of preference optimization with not only augmented data including both response-and image-contrast preference pairs, but also a vision-specific optimization target to enhance visual understanding. ## background and motivations we explore strategies to enhance visual understanding in lvlm preference optimization. our framework starts from a supervised fine-tuned (sft) model, obtained by jointly training a visual encoder and a pre-trained llm via visual instruction tuning @xcite . specifically, we incorporate visual guidance by integrating classifier-free guidance (cfg) into vanilla dpo. ## vision-guided preference learning in this work, we focus on mitigating hallucinations in lvlms caused by insufficient context at-tention to visual information. we propose visionguided direct preference optimization (v-dpo) to enhance visual understanding on both responseand image-contrast preference data. ## experiments we now assess v-dpo across various multimodal hallucination benchmarks. to interpret how v-dpo improves visual understanding, we compare performance using various preference data. specifically, unlike previous studies focusing on performance improvement using specific data, this work aims to demonstrate the effectiveness and generalizability of v-dpo across different training datasets and benchmarks for fair comparison. ## conclusion we propose v-dpo, utilizing classifier-free guidance (cfg) to integrate visual guidance in lvlm preference learning. integrating visual guidance into the training process enhances visual context understanding via preference optimization, improving the accuracy and specificity of model generations. extensive experiments on various preference data demonstrate the generalizability of v-dpo. we hope our work sheds light on visual guidance for more general tasks in lvlm alignment.