D oge RM : Equipping Reward Models with Domain Knowledge through Model Merging (2024)

Abstract: Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the **Do**main knowled**ge** merged **R**eward **M**odel (**DogeRM**), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.

## introduction modern large language models (llms), such as gpt-4 @xcite and gemini @xcite , showcase impressive capabilities across various tasks @xcite , with aligning their behavior with human preferences. reinforcement learning from human feedback (rlhf) is a prominent technique for enhancing the alignment of desired behaviors in llms @xcite @xcite . a key component of rlhf is its reward models (rms), which assess entire sentences generated by policy models. the reward signals produced by these rms are instrumental in adjusting the parameters of the policy models, thus directly impacting the policy models' effectiveness. rms are developed by training llms on paired preference data to simulate human judgment @xcite . this preference data consists of two responses to a given user input, merge solve 2x+3=8 step1: 2x=5 step2: @xmath0: @xmath1: 1 domain rm domain sft sft rm pretrained 4.95 scalar reward 0.05 scalar reward instruction+response step1: 2x=5 step2: @xmath2: @xmath3.5 answer: 2.5 text response solve 2x+3=8 step1: 2x=5 step2: @xmath4: @xmath5: 1 instruction+response user instruction solve 2x+3=8 accompanied by a human-assigned label indicating which response is more preferred. however, gathering such preference data can be costly and time-consuming due to the requirement of human annotation @xcite . this challenge becomes more pronounced when handling domainspecific preference data, as it necessitates expertise from domain specialists. recent developments have demonstrated the effectiveness of model merging techniques in strategically integrating multiple domain-specific models into a multi-domain model without requiring additional training @xcite . furthermore, domain-specific sft data is relatively more accessible compared to preference data. moreover, many high-quality domainspecific models are available on open-source platforms @xcite , which can be directly employed in the merging process. this brings us to consider a novel approach: is it possible to equip reward models with domain knowledge through merging with domain-specific language models? in this work, we propose domain knowledge merged reward model (dogerm), exploring the potential of merging a reward model trained on a general open-sourced preference dataset with a language model fine-tuned on domain-specific datasets, such as math and coding. an illustration of dogerm is presented in figure 1 . we evaluate dogerm using rewardbench @xcite , auto-j eval @xcite and best-of-n sampling on gsm8k @xcite and mbpp @xcite . our results demonstrate that dogerm improves performance and can be generalized to different model architectures. we also conduct a comprehensive analysis to demonstrate the impact of model merging. ## related work reward modeling rms are crucial for aligning language models with human preferences, providing proxy rewards as training signals for policy models. previous work has employed rl algorithms with rms to guide language models towards human preferences in various nlp tasks @xcite @xcite @xcite and instruction-following @xcite . in rlhf literature, rms evaluate the quality of instructions and responses based on criteria like helpfulness and harmlessness @xcite or more fine-grained objectives @xcite . several open-source paired preference datasets are available for training rms for rlhf, such as openai summarization @xcite , hh-rlhf @xcite , shp @xcite , ultrafeedback @xcite , pku-saferlhf @xcite , helpsteer @xcite , nectar @xcite , and ultrainteract @xcite . however, most datasets are not domain-specific. to address this, our work focuses on merging rms with domain-specific language models, aiming to equip rms with domain knowledge. model merging model merging integrates multiple task-specific models into a single unified model without additional training. a straightforward approach involves averaging parameters from models fine-tuned from the same initial model @xcite . another method employs weighted averaging of model parameters @xcite . another innovative approach involves creating task vectors by subtracting the weights of a pretrained model from those of the same model after fine-tuning for a specific task. this method showcases the flexibility and composability of these vectors through arithmetic operations @xcite @xcite . some recent work focused on model merging to align with user preferences. they interpolated model parameters fine-tuned on diverse rewards @xcite @xcite , or merging rms for combining different aspects of rewards @xcite . however, these methods still rely heavily on substantial domain-specific preference data to integrate domain knowledge. in contrast, our approach significantly reduces the need for such data by focusing on incorporating domain-specific knowledge into rms through model merging. ## methodology 3.1 reward modeling to train a reward model, we replace the decoding layer of a transformer-based pre-trained language model with a linear regression layer. this new layer projects the logits from the final transformer layer to a scalar, representing the reward of a given input. given an input prompt x, the chosen response y c , and the rejected response y r , we use the following loss function to optimize our reward model: ## conclusion in this work, we introduce a novel approach, dogerm, which integrates domain knowledge into rm by merging it with the domain-specific sft models. we demonstrate that dogerm enhances performance on math and coding benchmarks and can be generalized to different model architectures. a series of analyses show that dogerm effectively affects the reward signal corresponding to chosen and rejected prompts. the results highlight dogerm's potential to enhance model alignment and generation verification through model merging, offering promising results across various benchmarks.