Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward (2025)

Abstract: Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for open-ended conversations, remains a significant challenge. While previous studies have explored using large multimodal models (LMMs) as reward models for guiding preference modeling, their ability to accurately assess the quality of generated responses and their alignment with video content has not been conclusively demonstrated. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model’s reward mechanism, which directly takes video frames as input. Furthermore, we show that applying our reward mechanism to DPO algorithm significantly improves model performance on open-ended video QA tasks.

## introduction this paper addresses the challenge of aligning lmms, particularly in tasks that involve video instruction following. despite recent advancements in reinforcement learning (rl) @xcite @xcite and dpo @xcite @xcite , which have been effective in guiding llms towards generating more honest, helpful, and harmless content, their effectiveness in video domain remains limited. the critical obstacle lies in developing a robust reward system capable of distinguishing preferred responses from less preferred ones based on video inputs. ## method as shown in fig. 1 , our methodology enhances video lmm alignment through dpo method using rewards from a language model. we elaborate on constructing a video caption dataset in section 3.1. subsequently, in section 3.2, we discuss the generation of video instruction data and the fine-tuning process of our model. lastly, section 3.3 details the incorporation of generated captions as a feedback mechanism for dpo method to refine our model's factual alignment in video instruction-following tasks. ## assessment of evaluator with gpt-4v caption as video content to assess the effectiveness of our proposed reward assignment method, we conducted a comparative analysis the gpt-4v used as a video qa evaluator. our method utilizes detailed captions as a proxy of actual video frames, while gpt-4v directly takes in video frames as inputs. both reward systems follow the same set of guidelines for scoring reward (prompt in fig. 23 ). to compare the two methods, we sample 200 videos from each of the webvid, vidal, and ac-tivitynet datasets, each associated with one question and two model predictions from our sft model, with one preferred and one dispreferred by chatgpt. this results in 1, 200 examples, for which we used gpt-4v to assign scores. filtering through the azure api backend resulted in 196, 151, and 143 videos from each dataset, respectively, having both answers evaluated. the average scores of all examples from chatgpt and gpt-4v evaluations were 2.9 and 3.5 respectively, indicating a tendency of gpt-4v to yield slightly positive evaluations. the pearson correlation coefficient (pcc) of 0.47 (@xmath0.01) suggests a moderate positive correlation. in fig. 4 (left), the distribution of the difference between chatgpt and gpt-4v scores reveals that majority (> 75%) of chatgpt scores fall within one standard deviation (σ = 1.31) of gpt-4v scores. additionally, in fig. 4 (right), the agreement on preference between chatgpt and gpt-4v, excluding ties, exceeded 70%. these findings cautiously support our benchmark's applicability in video qa evaluation. further refinements for better alignment-such as incorporating likert scales @xcite or gpt-4 evaluation-are areas for future research. ## experiments we adopt video-llava @xcite as the backbone of our video lmm, but our method can be applied to any other architectures as well. ## conclusion we study the techniques for effective video lmm alignment. specifically, we propose an costeffective reward system that utilizes detailed captions as proxies for video content. we have shown the reward scores is well-aligned with the evaluation metrics of gpt-4v, and dpo training greatly enhances model performance. in addition, we have released 900k detailed video caption, 900k video instruction-following data, and 17k preference data pairs, with a complete code pipeline including pretraining for video captioning, fine-tuning for video instruction following and reinforcement learning with dpo for better lmm alignment.