I ntern LM - XC omposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model (2025)

Abstract: Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data.

## introduction despite their crucial role in both training and inference, multi-modal rms for large vision language models (lvlms) remain notably underexplored compared to language-only rms for llms. because current preference data is predominantly text-based and skewed toward specific domains (e.g., safety), data scarcity poses a significant challenge to training multi-modal rms for diverse modalities such as images, videos, and text. consequently, existing multi-modal rms @xcite are largely constrained to narrow domains (e.g., mitigating hallucinations) or rely on prompting lvlms with evaluation prompts, effectively functioning as generative rms @xcite . the limitation of multi-modal rms subsequently constrains the capabilities of opensource lvlms such as instruction following and safety-should-refuse, thereby hampering user interaction experience in multi-modal chat scenarios. reward vision encoder chosen: rejected: (b) reward model (a) multi-modal preference data here are some frames of a video. what's the man's reaction towards the dog playing with the rake? ixc-2.5-chat multi-modal rms, which motivates us to present internlm-xcomposer2.5-reward (ixc-2.5-reward). instead of directly transferring unimodal (text) reward models (rms) to the vision modality, we augment the existing lvlm (internlm-xcomposer2.5) with an additional scoring head to predict reward scores. an effective multi-modal rm should ideally possess two key properties: (1) the ability to predict reward scores for both image, video, and textual inputs and (2) the capacity to generalize across diverse domains, such as instruction following, knowledge, text-rich images (e.g., documents), reasoning tasks, etc. to this end, we develop a pipeline (fig. 1(a )) to construct multi-modal preference data, and also incorporate existing high-quality datasets. this pipeline selects prompts across diverse domains for text, image, and video inputs, generates corresponding re-sponses, and then uses gpt-4o @xcite or verifier @xcite to perform preference judgments. trained on our preference data, ixc-2.5-reward effectively evaluates both visual (image and video) and textual inputs (fig. 1 (b) ). ixc-2.5-reward achieves best performance on multi-modal vl-rewardbench @xcite (70.0%) that beat all previous generative rms including gemini-1.5-pro (62.5%) and gpt-4o (62.4%). even on uni-modal (text) rm benchmarks, ixc-2.5-reward also demonstrates good results, with an average score of 88.6% on reward-bench @xcite and 68.8% on rm-bench @xcite . we further demonstrate the effectiveness of ixc-2.5-reward in the following three aspects: ## related work reward model in large language models. reward models (rms) are crucial for both reinforcement learning from human feedback (rlhf) @xcite and test-time scaling laws @xcite . rms have different implementation forms, such as (1) discriminative rm @xcite @xcite @xcite @xcite , usually a sequence classifier that classifies input sequences into different categories, such as binary classification ("good" or "bad,") or on a more granular scale @xcite . (2) generative rm @xcite @xcite that are prompted to generate the feedback in the form of text, often a critique or explanation of why a certain output is good or bad. (3) implicit rms @xcite that are models optimized using dpo @xcite that the predicted log probabilities are interpreted as implicit reward signal. besides, rms can also be divided into outcome rms (orms) @xcite and process rms (prms) @xcite @xcite . our ixc-2.5-reward belongs to the discriminative rm and orm. reward model in large vision-language models. previous rms for lvlms @xcite @xcite are limited to specific domains (e.g., reducing hallucination) or developed using relatively weak base models, which makes the implemented models significantly inferior to llm rms. the lack of effective multi-modal rms has created a bottleneck in vision rlhf, forcing researchers to merely use the variants of the off-poly dpo algorithm @xcite . previous work using open-source lvlms as generative rms @xcite @xcite , injection of hallucinations with data augmentation techniques @xcite @xcite @xcite @xcite and rule-based selection @xcite for dpo data selection, which potentially compromise performance compared to the on-policy rl solutions like ppo @xcite . moreover, lacking multimodal rms has also led to the reliance on human annotation @xcite or the use of proprietary models @xcite like gpt4 as generative rms for dpo pair selection, which is expensive and unsustainable for large-scale applications. although open-source rms for lvlms have lagged behind their llm counterparts, the growing community interest highlights the need for multi-modal rms, which motivates our work. in this work, we demonstrate that ixc-2.5-reward is capable of combining with the ppo training and for dpo data selection at a low cost. reward model evaluations. the development of evaluation benchmarks is essential for improving rms. several comprehensive benchmarks have been proposed for evaluating rms of llms, such as general abilities @xcite @xcite , multilingual @xcite , rag @xcite , and mathematical process reward @xcite . the limited availability of multimodal rms has hampered the development of evaluation benchmarks, with existing benchmark @xcite focusing solely on generative rms and lacking the evaluation of process supervision. however, given the critical importance of rms, we expect significant progress in multi-modal rm benchmarking in the future. ## the applications of ixc-2.5-reward in this section, we further validate three applications of ixc-2.5-reward for (1) rl training (sec. 4.1), ( 2 ) test-time scaling (sec. 4.2), and (3) data cleaning (sec. 4.3). ## conclusion and future work we present ixc-2.5-reward, a multi-modal reward model that is capable of multi-modal rl training, test-time scaling, and data cleaning. using ixc-2.5-reward, we further trained ixc-2.5-chat via rlhf techniques to optimize the multi-modal user chat experience, focusing on providing detailed explanations and in-depth answers. we believe that exploring multi-modal reward models with on-policy reinforcement learning algorithms holds significant promise for future research, such as exploring reward benchmarks and rl algorithms for video alignment.