Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback (2024)

Abstract: Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). Previous approaches for VLMMs involve Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and additional learnable parameters. Here, aligning video with text, and vice versa, remains a challenge, primarily due to the insufficient quality and quantity of multimodal instruction-tune data compared to that of text-only. This discrepancy often results in alignments that poorly ground the video content. To address this, we present a novel alignment strategy that employs a multimodal AI system equipped with Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignment of video and text modalities. Our approach uniquely integrates detailed video descriptions as context into a multimodal AI system during the preference feedback generation to enrich the understanding of video content, a process we call context-aware reward modeling. Empirical evaluations on various video benchmarks demonstrate that our VLM-RLAIF outperforms existing approaches, including the SFT model. We commit to open-sourcing our code, models, and datasets to foster further research in this area.

## introduction large language models (llms) are advancing many language and multimodal ai tasks, including those involved with video large multimodal models (vlmms) @xcite @xcite . extending the logical reasoning and advanced cognitive capabilities of llms to the visual domain, vlmms are now remarkably proficient in tasks such as video understanding @xcite , video question answering @xcite and instruction-following msvd-vqa anet-vqa msrvtt-vqa ucf101-ar hmdb51-ar correctness temporal consistency 3.4 76.5% 3.3 57.4% 63.1% ## vlm-rlaif framework to overcome the limited scalability of human feedback in rlhf, we use ai's feedback to align multimodality between video and text, reducing the reliance on exhaustive human-annotated preferences @xcite . in particular, we improve the feedback process by using detailed video descriptions, thereby achieving better contextual clarity in video content. the training procedure of vlm-rlaif can be summarized into three stages as follows: supervised fine-tuning (sft). we first fine-tune an llm, e.g., vicuna, using supervised learning on synthetically generated video-text instruction-tune data @xcite . this involves the integration of a vision encoder with two linear layers and additional learnable parameters using lora @xcite , into the training process. this fine-tuning allows the model to better follow the instructions (muhammad maaz and @xcite . additionally, we improve the sft process by expanding the instruction-tune data and introducing simple curriculum learning (sec. 3.2). we refer to this fine-tuned model as the video large multimodal model with sft or vlm-sft for short. ## conclusion we propose a novel alignment strategy for vlmms, termed vlm-rlaif, that uses reinforcement learning from ai feedback. to improve multimodal alignment, we propose a context-aware reward modeling, enabling ai to generate feedback for self-improvement. in addition, we expand the instruction-tune dataset for sft and adopt a curriculum-based training approach, that is particularly effective in the gradual learning of complex video-text relationships. in our empirical validations, the vlm-rlaif significantly outperforms previous models in multiple multimodal video-text understanding benchmarks, which implies good generalization performance across tasks.