VLF eedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment (2024)

Abstract: As large vision-language models (LVLMs) evolve rapidly, the demand for high-quality and diverse data to align these models becomes increasingly crucial. However, the creation of such data with human supervision proves costly and time-intensive. In this paper, we investigate the efficacy of AI feedback to scale supervision for aligning LVLMs. We introduce VLFeedback, the first large-scale vision-language feedback dataset, comprising over 82K multi-modal instructions and comprehensive rationales generated by off-the-shelf models without human annotations. To evaluate the effectiveness of AI feedback for vision-language alignment, we train Silkie, an LVLM fine-tuned via direct preference optimization on VLFeedback. Silkie showcases exceptional performance regarding helpfulness, visual faithfulness, and safety metrics. It outperforms its base model by 6.9% and 9.5% in perception and cognition tasks, reduces hallucination issues on MMHal-Bench, and exhibits enhanced resilience against red-teaming attacks. Furthermore, our analysis underscores the advantage of AI feedback, particularly in fostering preference diversity to deliver more comprehensive improvements. Our dataset, training code and models are available at https://vlf-silkie.github.io .

## introduction large vision-language models (lvlms), exemplified by the groundbreaking achievements of gpt-4v (openai, 2023b) and gemini @xcite , have evolved rapidly. while they have demonstrated the capability to perform reasoning tasks over images and deliver responses tailored to user inquiries @xcite , lvlms still face significant challenges in achieving better alignment with humans. these challenges can manifest in the generation of misleading content lacking visual grounding @xcite biased responses against minority groups @xcite , and susceptibility to multimodal jailbreaking @xcite . addressing these issues is paramount to the responsible usage of lvlms. to tackle this, exploring preference alignment for lvlms through human or ai feedback becomes imperative, evidenced by previous successful exploration with llms @xcite . however, the applicability of such approaches to lvlms remains largely unexplored due to the lack of large-scale feedback datasets in the first place. given the additional visual modality involved, soliciting high-quality and scalable human feedback becomes inherently more challenging and resource-intensive. previous studies @xcite therefore target a narrow aspect such as, visual faithfulness, while still yielding high cost as demonstrated in in this work, we introduce the first largescale gpt-4v annotated vision-language feedback (vlfeedback) dataset for aligning lvlms comprehensively. we begin by constructing a diverse multi-modal instruction set sourced from various datasets, encompassing general conversations, academic tasks and specialized domains, and incorporating red teaming instructions for safety alignment. there are 82.4k instructions in total, covering 67k unique images and 399.4k preference pairs. furthermore, we establish a pool of 12 lvlms, including blip-family @xcite , llava-series @xcite , fuyu-8b @xcite , qwen-vl-chat @xcite , and gpt-4v @xcite , to generate corresponding responses conditioned on our collected instructions. to comprehensively evaluate preferences, we define annotation templates focusing on three critical aspects of vision-text interaction: (i) helpfulness, assessing the relevance of responses to user queries and their contribution to user understanding of visual content; (ii) visual faithfulness, examining the consistency between visual clues and responses to detect potential ungrounded hallucinations; and (iii) ethical considerations, scrutinizing responses for offensive, biased or harmful content. given the images and corresponding instructions, gpt-4v is then queried with these annotation templates to assess the response of different models, as illustrated in figure 1 . the consistency of preferences between gpt-4v and human annotators is evaluated on a subset of vlfeedback, demonstrating an impressive average agreement rate of 83.1%, validating the suitability of gpt-4v for accurate preference annotation tasks. with the constructed vlfeedback dataset, we delve into lvlm alignment using direct preference optimization (dpo) @xcite to enhance the performance of an open-sourced lvlm, i.e., qwen-vl-chat. our experimental findings showcase significant enhancements in the resulting model, named silkie, across all evaluated benchmarks. specifically, silkie achieves a remarkable performance improvement of 6.9% and 9.5% in perception and cognition tasks on the mme benchmark @xcite , as well as surpassing its base model on challenging mathematical reasoning benchmarks mathvista @xcite and mmmu @xcite . silkie also generates responses better aligned with the visual context, as evidenced by its improved score of 3.02 on the hallucination evaluation benchmark mmhal-bench @xcite . besides, after performing dpo on the red-teaming subset of our vlfeedback, the model demonstrates improved resilience to red-teaming attacks without compromising its perception abilities. furthermore, we observe that ai-annotated preferences boost lvlms more effectively than humanannotated preference datasets @xcite , validating the quality and comprehensive coverage of our preference dataset. ## vision-language feedback dataset in this section, we elaborate on the construction of our vision-language feedback (vlfeedback) dataset for comprehensively aligning lvlms, as illustrated in the figure 1 . we first introduce the multi-modal instructions sources ( §2.1), followed by the details of selected lvlms for decoding ( §2.2) and the annotation with gpt-4v ( §2.3). the statistics of our vlfeedback are presented in §2.4. ## experiments in this section, we explore alignment training using dpo @xcite to explore the effect of our vlfeedback. we first introduce the experimental setups ( §3.1), including training details, evaluated benchmarks and baseline methods. we further present the main results and discuss the findings ( §3.2), followed by analysis explorations and a case study ( §3.3). ## related works preference alignment the requirements of building helpful and safe models necessitate aligning their behaviors with human values @xcite . common techniques for achieving this include instruction tuning @xcite and reinforcement learning from human feedback (rlhf) @xcite . as preference feedback often contains subtle differences, rlhf has emerged as a preferred approach to alignment, with ppo @xcite and dpo @xcite being representative implementations. however, gathering highquality human feedback is costly. therefore, leveraging ai feedback offers an alternative to scale up the preference alignment process @xcite , where preferences are generated by off-the-shelf models. large vision-language models the development of lvlms has accelerated recently @xcite @xcite . to better fuse visual and textual modalities, research has focused on architectural improvements @xcite , instruction tuning @xcite @xcite , and scaling @xcite . however, lvlms still face systematic issues, such as hallucination, where responses are not grounded in the visual context @xcite . these deficiencies highlight the need for more fine-grained alignment in lvlms. preference alignment for lvlms preliminary explorations into preference alignment for lvlms have shown promising results. llava-rlhf @xcite creates a human-annotated, factually oriented preference dataset. building on this, rlhf-v @xcite enhances llava-rlhf by collecting a more fine-grained preference annotation dataset. however, the amount of preference feedback (10k and 1.4k instances) remains limited due to the high cost of labeling. povid @xcite instead injects hallucinated content into text responses and then adopts them as dis-preferred responses during dpo. ha-dpo @xcite uses gpt-4 to detect and correct the hallucinated content in image descriptions and then gather these pairs for dpo training. similarly, dress @xcite leverages gpt-4 to generate natural language feedback for improving the alignment and interaction capabilities of lvlms. in this work, we explore a scalable alignment paradigm for lvlms. we construct vlfeedback, the first large-scale ai feedback dataset, and demonstrate its effectiveness in improving overall capabilities and safety while reducing hallucinations. concurrent works @xcite explore similar ideas, query: which wooden stool has a vase with red flower on it? qwen-vl-chat: there is no existence of a vase with a red flower on any of the wooden stools. silkie: the wooden stool with a vase with a red flower on it is on the left. query: compare the average kinetic energies of the particles in each sample. the diagrams below show two pure samples of gas in identical closed, rigid containers. each colored ball represents one gas particle. both samples have the same number of particles. which sample has the higher temperature? sample a or sample b? qwen-vl-chat: sample b has the higher temperature. ## conclusions this paper explores lvlm alignment via ai preference by constructing vlfeedback, the first large-scale ai-annotated vision-language feedback dataset. our exploration with direct preference optimization on vlfeedback highlights the substantial performance enhancement achieved by the silkie model across various multi-modal benchmarks. notably, ai-annotated preferences demonstrate superior efficacy in driving comprehensive improvements compared to human annotations. we anticipate that vlfeedback will be an invaluable asset for future alignment studies.