On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization (2024)

Abstract: Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an EXplicit Reward Model (EXRM) as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Direct Preference Optimization (DPO). Prior work has shown that the implicit reward model of DPO (denoted as DPORM) can approximate an EXRM on the limit infinite samples. However, it is unclear how effective is DPORM in practice. DPORM’s effectiveness directly implies the optimality of learned policy of DPO and also has practical implication for more advanced alignment methods, such as iterative DPO. We compare the accuracy at distinguishing preferred and rejected answers using both DPORM and EXRM. Our findings indicate that even though DPORM can fit the training dataset, it generalizes less effective than EXRM, especially when the validation datasets contain distributional shifts. Across five out-of-distribution settings, DPORM has a mean drop in accuracy of 3% and a maximum drop of 7%. These findings highlight that DPORM has limited generalization ability and substantiates the integration of an explicit reward model in iterative DPO approaches.

## introduction large language models (llms) have demonstrated exceptional performance on many tasks in diverse applications, including mathematical reasoning, coding capabilities, and knowledge-based question answering @xcite @xcite . whilst llms have broad knowledge and reasoning skills, the pre-training objective is often misaligned from the objective of instruction following @xcite according to human preferences @xcite , and llms can exhibit undesirable behaviors including hallucinating, and providing harmful or biased instructions @xcite . as llms become more commonplace, it is important for them to be aligned with human preferences for helpfulness, harmlessness, and honesty @xcite . a common practice for aligning llms to human preference is through reinforcement learning with human feedback (rlhf) @xcite , which is based on a reward model trained to score model outputs according to human preference annotations. in rlhf alignment, a high quality reward model is required for policy learning @xcite . however, in practice, learned reward models are typically imperfect approximations of the "true" human reward label function @xcite , because they are trained on a fixed set of human preference data collected offline. when used within the rlhf policy optimization, they may see out-of-distribution (ood) data when annotating the response of the model. aligning according to an imperfect reward model can lead to worse performing language models as policy optimization continues to optimize a mis-specified reward. this can lead to an increased gap between the learned and true reward, a phenomena known as over-optimization and reward hacking @xcite . recently, @xcite proposed direct preference optimization (dpo), and show that any reward model can be implicitly represented by the optimal policy learned by dpo and a reference policy under certain assumptions. similar to rlhf, dpo assumes access to a preference dataset, but directly finetunes a language model policy by minimizing a negative log likelihood objective. due to its simplicity, dpo offers a more stable and convenient alignment process @xcite . while, dpo is often regarded as simpler alternative to rlhf fine-tuning, and models trained with dpo have been shown to empirically match or outperform the rlhf policy performance even on ood data @xcite , its generalization ability still remains under-explored. in this work, we conduct a systematic comparison between explicit reward models (exrm) learned by rlhf and dpo's implicit reward model (dporm) in terms of their generalization ability at distinguishing preferred and rejected answers. we train exrm and dporm on datasets containing chat, instruction following, and summarization, and evaluate them on in-distribution (id) evaluation sets, and ten ood evaluation sets. across five train-test shifts, and three model series: gemma-2b, gemma-7b @xcite and mistral-7b (with instruction tuning) @xcite totaling 35 experiments, we find that dporm underperforms exrm when the validation dataset contains distributional shifts. dporm has a mean drop in accuracy of 3% and a maximum drop of 7%. our results highlight the importance of learning an exrm and justifies more complex online approaches that combine exrm with iterative dpo @xcite . ## background training a reward model r involves training a classifier according to a preference dataset d. the reward model takes as input a prompt @xmath0, and scores the response r(x, y). the reward model is typically parameterized by a language model with an additional linear layer, the output of which is a scalar reward, which is used to compute the preference probability. an overview of the learning procedures rlhf via dpo and rl algorithms are included in figure 1 . rlhf methods train a llm π rm to maximize the reward given by r rm : settings training set id testing ood testing shift type setting i(a) ultra ultra arena, hh, nectar, webgpt, sum mixture setting i(b) arena arena ultra, hh, nectar, webgpt, sum mixture setting i(c) hh hh ultra, arena, nectar, webgpt, sum mixture setting ii(a) ultra ultra ultra-lm3-sft, ultra-lm3-rlhf response shift setting ii(b) sum-reddit sum-reddit sum-dailymail, sum-cnn prompt shift under these assumptions, an aligned llm π dpo can be optimized without explicitly training a reward model. combining equation 2 with 3, the rl optimization can be reduced to the objective: and the implicit reward model can be expressed in terms of the dpo policy as prior work also conjectures that the task of generating preferred responses as in dpo's objective is more challenging than learning a discriminator between responses @xcite . in section 3, we investigate the generalization capability of exrm (equation 1 ) and dporm (equation 5 ). ## experiments to study the generalization ability of exrm and dporm, we investigate both the id performance on a held-out validation set, and the ood generalization performance. we study two distribution shifts summarized in figure 2: (1) prompt shift: the distribution of the testing prompts differs from that of training prompts. this shift could occur when training a reward model on prompts within some domain and using the reward model to annotate the prompts from other domains as in iterative dpo. for all experiments we finetune decoder-only transformer models @xcite at 2b and 7b model parameter scales. the models are trained using the trl foot_0 library. for reward modeling we fine-tune all models for 1 epoch using a learning rate of 5e -6 . for dpo, we train for two epochs using a learning rate of 1e -6 and β = 0.03. all other hyperparameters correspond to the default parameter setting in the trl library. hyperparameters for all model-data combinations were selected using gemma-2b and 7b with ultrafeed-back as the training set. details of our sweep are given in section d. for all experiments, unless otherwise stated we report the mean and standard deviations of the accuracy over three random seeds. ## conclusion and discussion this work takes a step towards explicitly characterizing the generalization ability of the implicit reward function of dpo compared with an explicitly trained reward model. our findings highlight that the implicit reward model consistently underperforms the explicit reward model. for researchers and practitioners aligning llms, our work sheds light on the potential benefits of using reinforcement learning algorithms to fine-tune over the simpler approach of dpo, and substantiates the field of recent work investigating iterative dpo algorithms that combine explicit reward models with dpo.