Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024)

Abstract: Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. However, the current MLLMs trained with visual-question-answering (VQA) datasets could suffer from degradation, as VQA datasets lack the diversity and complexity of the original text instruction datasets with which the underlying language model was trained. To address this degradation, we first collect a lightweight, 5k-sample VQA preference dataset where answers were annotated by Gemini for five quality metrics in a granular fashion and investigate standard Supervised Fine-tuning, rejection sampling, Direct Preference Optimization (DPO) and SteerLM algorithms. Our findings indicate that with DPO, we can surpass the instruction-following capabilities of the language model, achieving a 6.73 score on MT-Bench, compared to Vicuna’s 6.57 and LLaVA’s 5.99. This enhancement in textual instruction-following capability correlates with boosted visual instruction performance (+4.9% on MM-Vet, +6% on LLaVA-Bench), with minimal alignment tax on visual knowledge benchmarks compared to the previous RLHF approach. In conclusion, we propose a distillation-based multi-modal alignment model with fine-grained annotations on a small dataset that restores and boosts MLLM’s language capability after visual instruction tuning.

## introduction recent advancements in artificial intelligence have led to the rise of multi-modal large language models (mllms), which combine textual and visual interpretation capabilities in a single model @xcite . however, effectively blending multimodality in one system has proven non-trivial. integrating diverse data forms often creates internal * corresponding author. driven by the limitations observed in distillationbased instruction tuning, particularly its constrained generalizability and the narrow performance improvements on tasks outside the training distribution, this study investigates the efficacy of distillation-based preference alignment in addressing modality conflict in mllms. the decision to explore this avenue is predicated on the hypothesis that integrating ai-generated preference data can provide a more granular and nuanced alignment with human expectations, potentially mitigating the adverse effects of modality conflict. this study rigorously evaluates three baseline methodologies-direct preference optimization (dpo), steerlm, and rejection sampling-as potential solutions to utilize the distilled preference data to enhance the instruction-following capabilities and address the modality conflict inherent in mllms. each of these methods offers a unique approach to model alignment, from the direct opti-mization of preferences in dpo to the conditional supervision in steerlm and the selective acceptance in rejection sampling. our empirical analysis reveals that dpo, in particular, demonstrates a pronounced efficacy in reconciling the performance disparities observed between textual and visual modalities. by leveraging a refined preference dataset, fine-tuned with the dpo objective and supplemented with comprehensive annotations from advanced ai models, dpo not only addresses the modality conflict but also significantly enhances the model's performance across a spectrum of benchmarks. the results indicate that, through the application of dpo, mllms can achieve a more robust alignment with human-like preferences, thereby mitigating the adverse effects of catastrophic forgetting and modality conflict and elevating the models' capabilities to a level that surpasses traditional instruction tuning methods. our main contributions are: exploration of modality degradation: this work is the first to identify and address modality degradation in mllms, a phenomenon where visual instruction tuning detrimentally impacts language instruction capabilities. our systematic investigation into this issue contributes novel insights to the field, laying the groundwork for further research in mitigating such degradation. efficient and scalable preference alignment pipeline as remedy our data collection strategy employs a granular quality metric annotation format, leveraging cost-effective commercial apis. this scalable approach enables the efficient production of high-quality datasets. we are able to surpass llava and vicuna's language instructionfollowing capability with dpo on a 6k dataset. ## method the focus of this study is to investigate improving the text instruction following of mllms while retaining or potentially enhancing their multi-modal capabilities. for this purpose, we propose to harness alignment methods that utilize self-sampled responses and preference annotations in addition to supervised fine-tuning (sft) as a baseline. ## limitations this research, while pioneering advancements in direct preference optimization (dpo) and multimodal large language models (mllms), confronts several limitations. the preference dataset, derived from artificial annotations, may not fully capture the nuanced spectrum of human values, potentially affecting the model's alignment with true human preferences and exacerbating existing biases in automated decision-making. additionally, the generalizability of these findings across different multimodal contexts and languages is yet to be thoroughly tested, and the deployment of such models in critical domains raises significant safety risks due to the potential unreliability of responses under varied real-world conditions. these limitations highlight the need for cautious application, further empirical validation, and ongoing research to mitigate ethical concerns and enhance model robustness and fairness. ## conclusion in this paper, we investigated and addressed the performance decline of the widely-used multi-modal llm, llava, on language instruction-following benchmarks. we investigated various alignment strategies to enhance the model's language instruction capabilities, including standard sft, steerlm, and dpo. our comprehensive benchmark evaluations revealed that dpo stands out as the most effective method, not only restoring but also surpassing the language capabilities of llava's language backbone (vicuna) on both mt-bench and alpacaeval. specifically, llava's performance on mt-bench had dipped from vicuna's 6.57 to 5.99, whereas our dpo model advanced to 6.73. beyond language proficiency, dpo also bolstered performance on multi-modal benchmarks, enhancing accuracy by 4.9% on mm-vet and 6% on llava-bench. remarkably, dpo achieved this with just 5k preference examples, proving more dataefficient than other strategies like rlhf, which typically demand over 80k training examples. our findings highlight dpo's minimal alignment tax effectively boosting one performance area without substantially detracting from another. this aspect is particularly valuable for practitioners aiming to maintain the pre-training knowledge and avoid catastrophic forgetting. in this image, the room has some artistic or creative significance due to the presence of several sculptures placed on a long wooden table . the sculptures appear to be made from various materials, such as stone and wood. additionally, the room features a chair and a light fixture that resembles a boot. these elements, combined with the assortment of sculptures, give the room an eclectic and artistic atmosphere, suggesting that the space might be used for displaying art, inspiration, or as a workspace for an artist or designer. the presence of the boot-shaped light also adds a unique and imaginative touch to the room's overall aesthetic. the image features a person running along a beach near the water's edge. behind the person, multiple birds are flying in the sky, creating a dynamic and lively scene. the man is wearing a swimsuit and appears to be enjoying his time at the beach. the birds' positions vary across the sky, with some flying closer to the person and others further away, spreading out across the background. response 2 in the room, a collection of wooden shoes on a long table next to a chair can be considered as having artistic or creative significance. the display of shoes may represent a unique form of artistic expression or a design concept. additionally, the white-walled setting with a dim light allows for a minimalist and visually striking atmosphere that enhances the shoes' display and creates a sense of depth and artistic interest in the space. the image captures a beach scene where a man is running along the shore near the water, with waves crashing behind him. the man is in motion, as he is probably enjoying an invigorating run on the beach. in the sky above, several birds fly in a line, adding a sense of movement and liveliness to the scene. the birds' positions vary, with some being closer to the water and others higher up in the sky. the overall atmosphere of the image suggests a pleasant, active day at the beach.