Reward Difference Optimization For Sample Reweighting In Offline RLHF (2024)

Abstract: With the wide deployment of Large Language Models (LLMs), aligning LLMs with human values becomes increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated and highly resource-intensive. As such, offline RLHF has been introduced as an alternative solution, which directly optimizes LLMs with ranking losses on a fixed preference dataset. Current offline RLHF only captures the ordering relationship between responses, overlooking the crucial aspect of “how much” one is preferred over the others. To address this issue, we propose a simple yet effective solution based on reward difference prediction. Specifically, we introduce reward difference coefficients to reweigh sample pairs in offline RLHF. We then propose a difference model that considers rich interactions between a pair of responses for predicting these difference coefficients. Experiments with 7B LLMs on the HH and TL;DR dataset verify the effectiveness of our method in both automatic metrics and human evaluation, highlighting its potential for aligning LLMs with human values.

## introduction large language models (llms) have recently emerged as a major milestone in modern natural language processing (nlp), offering unprecedented capabilities in understanding, generating, and translating human language @xcite @xcite @xcite @xcite . powered by an extremely large number of parameters, llms encode the wealth of human knowledge via a pretraining process, which is often conducted on a web-scale text corpus with * corresponding authors the next token prediction objective. as llms become more capable, it is essential and demanding for them to follow human preferences such as truthfulness, harmlessness, and helpfulness. unfortunately, the maximum likelihood objective for the next token prediction falls short in capturing such crucial human values @xcite . reinforcement learning with human feedback (rlhf) has been introduced as an effective method for llms alignment @xcite . rlhf first leverages a supervised learning objective to equip llms with basic instruction-following capabilities. subsequently, a reward model is trained on a human preference dataset that contains pairwise comparisons of responses from llms (for the same query). the reward model is exploited for further finetuning llms via reinforcement learning with proximal policy optimization (ppo) @xcite . specifically, training with ppo requires four models: a policy model (the targeted llm), a critic model, a reference model (i.e., a supervisedfinetuning version of the targeted llm), and a reward model. all these models are based on llms with billions of parameters, whereas the policy and the critic models need to be updated online. in light of this, rlhf is highly resource-demanding and rather complicated to be applied in practice. in this context, offline rl has been proposed as an alternative approach to llms alignment @xcite . the basic idea of offline rlhf is to directly train llms on the preference dataset using some ranking loss such as max-margin ranking loss @xcite , or negative log-likelihood loss @xcite . optionally, a reward model can be trained to collect comparison data for sampling new responses from llms @xcite . it should be noted that these offline rlhf methods only exploit the ordinal relationship between responses rather than making use of scalar values for rewarding (or penalizing) responses as in rlhf. the ordinal relationship, however, does not reflect the degree to which the preferred response is better than the dispreferred one. for instance, figure 1 shows an example where the quality difference of the pair (response 1, response 2) is not as significant as that of the pair (response 1, response 3). as such, treating the two pairs equivalently may result in suboptimal performance for offline rlhf. to mitigate the aforementioned issue, we propose a simple yet effective method based on reward difference optimization, shorted as rdo. first of all, the reward difference coefficient is introduced as the degree to which one response is preferred against the other one given the same query. such coefficients can be exploited as sample weights, which are incorporated into diverse offline rlhf methods as additional supervision signals for calibrating the loss function. basically, one can train a reward model similar to that in rlhf and then exploit the reward values for reward difference measurement. this method, however, is not effective enough as the reward model independently assigns scores to responses. inspired by how much easier it is for humans to conduct pairwise judgments, we propose a difference model to directly predict the reward difference between two responses. unlike the reward model, our difference model leverages attention-based interactions between two responses for prediction. our contributions are summarized as follows: ## reward difference coefficients reward difference coefficients as aforementioned, offline rlhf methods exploit the ordinal relationship y w ≻ y l for measuring the alignment loss (l (aln) in eq. 1), without differentiating pivotal pairs and the trivial counterparts. we propose a simple yet effective method to address such issues based on reward differences. specifically, given a well-trained reward model r ϕ : (x, y) → r that assigns a reward scalar to each response y for the query x, we can quantify the reward difference for each pair of responses (y w , y l ) for preference optimization as follows: assuming that the reward model can pinpoint essential differences from minor mistakes, the coefficient term helps steer more gradients towards sample pairs with larger differences. note that, a typical way to learn a reward model is based on the bradley-terry (bt) model (bradley and terry). specifically, the loss function for learning the reward model is as follows: the proposed coefficient can be readily incorporated into diverse offline rlhf methods. the following details how our method can be applied to rrhf and dpo. rrhf+rc rrhf with reward difference coefficients (rrhf+rc) optimizes the following loss: dpo+rc dpo with reward difference coefficients (dpo+rc) optimizes the following loss: ## reward difference prediction the above reward difference coefficients are measured based on a typical (pointwise) reward model r ϕ (x, y). however, doing so is not effective as the model considers responses independently. in this section, we introduce a reward difference model that simultaneously takes the given query x and two responses y w , y l as inputs for predicting the reward difference. we then propose an effective method for training the difference model. ## conclusion this paper proposes a novel approach to address the limitations of existing offline rlhf methods. our method leverages the reward difference coefficient, which quantifies 'how much" one response is preferred over another. these coefficients are integrated as sample weights during training, putting the focus towards more "sure" comparisons. furthermore, we introduce a reward difference model for directly predicting these coefficients, accompanied by an effective training methodology. experimental results consistently show the effectiveness of the reward difference coefficients on two representative offline rlhf methods including dpo and rrhf. in addition, llms-based evaluation and human evaluation validate the advantages of the difference model over the reward model. in the future, we aim to explore several promising directions: (1). scaling law: while this work demonstrates effectiveness with 7b models, scaling to larger architectures is crucial. evaluating performance on progressively larger llms will provide valuable insights into the effectiveness of our approach; (2). generalization: as llms may lose some of their generalization ability after alignment @xcite , exploring techniques to mitigate this phenomenon would ensure llm retention of their core strengths besides improved alignment.