Aligning Large Multimodal Models with Factually Augmented RLHF (2024)

Abstract: Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in “hallucination”, generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-language model is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement of 60% on MMHAL-BENCH over other baselines.

## introduction large language models (llms; @xcite ; @xcite ; openai (2023)) * equal contribution. ordering is determined by dice rolling. †equal advising. to mitigate the challenges posed by the scarcity of high-quality visual instruction tuning data for lmm training, we introduce llava-rlhf, a vision-language model trained for improved multimodal alignment. one of our key contributions is the adaptation of the reinforcement learning from human feedback (rlhf) @xcite @xcite , a general and scalable alignment paradigm that shows great success for text-based ai agents, to the multimodal alignment for lmms. by collecting human preferences with an emphasis on detecting hallucinations foot_0 , we utilize those preferences in reinforcement learning for lmm fine-tuning @xcite . our approach can improve the multimodal alignment with a rel-question: where is this photo taken? ground truth: from the slogan "welcome to houston, texas", we can infer that this photo is taken at an airport in houston, texas. llava: this photo is taken at an airport. llava-sft + : this photo is taken at the houston airport. llava-rlhf: this photo is taken in the baggage claim area of an airport, specifically in the lobby of the george bush intercontinental airport in houston, texas. the large sign in the background indicates the airport's name and location. the presence of multiple people, handbags, and suitcases in the scene further supports the fact that this is indeed an airport setting. question: what is the gender of the person riding the motorcycle in the image? ground truth: no one is riding the motorcycle in the image. ## method in this study, we employ a multimodal reinforcement learning from human feedback (rlhf) approach to align large multimodal models (lmms) with human values (sec. 2.1). the process begins with multimodal supervised fine-tuning to establish a foundational understanding of multimodal inputs (sec. 2.2). this is enhanced by multimodal preference modeling, where a reward model is trained with human-annotated comparisons to discern better responses (sec. 2.3). the approach culminates with reinforcement learning and factually augmented rlhf, which refine the model's responses for accuracy and factual alignment, leveraging high-quality instruction-tuning data and additional ground-truth information to combat reward hacking and hallucinations (sec. 2.4). ## discussions & concluding remarks hallucination phenomena are observed in both llms and lmms. the potential reasons are twofold. firstly, a salient factor contributing to this issue is the low quality of instruction tuning data for current lmms, as they are typically synthesized by more powerful llms such as gpt-4. we expect our proposed high-quality vision instructiontuning data and future efforts on manually curating high-quality visual instruction tuning data can alleviate this problem. secondly, the adoption of behavior cloning training in instruction-tuned lmms emerges as another fundamental cause @xcite . since the instruction data labelers lack insight into the lmm's visual perception of an image, such training inadvertently conditions lmms to speculate on uncertain content. to circumvent this pitfall, the implementation of reinforcement learning-based training provides a promising avenue, guiding the model to articulate uncertainties more effectively @xcite . our work demonstrates a pioneering effort in this direction. figure 2 illustrates the two sources of hallucination in current behavior cloning training of llms. in conclusion, we proposed several strategies to tackle the multimodal misalignment problems, particularly for lmm, which often produce text inconsistent with the associated images. especially, we introduce the factually augmented rlhf, leveraging additional factual information such as image captions to enhance the reward model, countering reward hacking in rlhf, and boosting model performance. for tangible real-world impact assessment, we have devised mmhal-bench, an evaluation benchmark targeting the penalization of hallucination. remarkably, llava-rlhf, being the first lmm trained with rlhf, shows a notable surge in performance across benchmarks. we opensource our code, and data and hope our findings could help the future development of more reliable and human-aligned llms and lmms.