Fine-Tuning Language Models with Reward Learning on Policy (2024)

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences.RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially.Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs’ data distribution.Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize.In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution.Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples.Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs.Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art.Our code is available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp .

## introduction large language models (llms) @xcite have shown great promise in following open-ended user instructions @xcite @xcite . these capabilities are largely attributed to the fine-tuning of pretrained llms using reinforcement learning from human feedback (rlhf) @xcite , which is a prominent technique to align llms with human preferences and greatly en- a typical rlhf procedure is comprised of three interrelated steps: human preference collecting, reward learning, and policy optimization (figure 1 top). the reward learning step fits a reward model to the preference data that elicits evaluations from humans. the policy optimization step uses reinforcement learning (rl) to fine-tune a language model to produce outputs assigned high reward. in practice, the three key steps of rlhf are often performed serially @xcite . since policy optimization shifts the language model's data distribution during the rl phase, the (fixed) reward model will be inaccurate off-distribution which is trained on offline data @xcite . hence, reward model accuracy can quickly degrade and in turn degenerate the policy that exploits differences between the inferred and true reward @xcite . the above issue can be mitigated by gathering new human preference data from an up-to-date version of policy @xcite . however, the resulting system is significantly more complicated and difficult to optimize, involving iterations of data gathering, reward learning, and rl fine-tuning. moreover, significant work is required to maintain high data quality over a long time in this setting. in this paper, we show how to optimize a reward model against the policy to keep it on-distribution, without repeatedly collecting new human preference data. we propose reward learning on policy (rlp), a framework that refines a reward model using policy samples in an unsupervised manner. rlp first trains a reward model and a language model policy from scratch with standard rlhf methods, and then retrains the reward model when exposed to the sample distribution of the trained policy. finally, rlp retrains the policy on the retrained reward model, which attempts to maintain an accurate reward for the latest policy. concretely, rlp uses policy samples to retrain the reward model via two methods: unsupervised multi-view learning (uml) and synthetic preference generation (spg). rlp-uml constructs two views for an input by generating two responses from the policy @xcite , then optimizes a multi-view information bottleneck loss @xcite when fitting the reward model to a dataset of human preferences. this training objective follows the information bottleneck principle @xcite and helps learn robust representations of the policy's data distribution. in addition, rlp-spg simulates preferences on policy generations to supplement the human preference data. rather than producing and scoring two outputs with llms as in reinforcement learning from ai feedback (rlaif) @xcite , rlp-spg generates a set of outputs for an instruction. in this way, rlp-spg can quantify uncertainty and decide when to trust model predictions via measuring the size of the largest semantic equivalence cluster @xcite . thus, rlp-spg selectively generates pairwise preferences for instructions with low uncertainty @xcite , where the preferred output is sampled from the largest cluster of the output set and the non-preferred one is sampled from the rest clusters. this sampling scheme also conforms to the self-consistency assumption, i.e., the most consistent output is selected as the final prediction @xcite . our main contributions are as follows: instruction tuning is a procedure to fine-tune pretrained llms with instructions and human-written completions @xcite , which increases the usability of llms @xcite . recently, rlhf has emerged as the central method for fine-tuning llms based on human preferences and further improves their downstream task performance and alignment with user intent @xcite . generally, rlhf methods first fit a reward model to human preferences, then fine-tune a language model to maximize the inferred reward using rl algorithms. reward models tend to be an imperfect estimate of the true reward due to misspecification @xcite and misgeneralization @xcite , and imperfect in reward models leads to reward hacking @xcite . methods with reward ensemble @xcite and diverse feedback @xcite are proposed to tackle this issue. our method retrains the reward model with policy samples to make it on-distribution and generalize to the policy's data distribution. human feedback simulation aims to generate additional synthetic preference data using weak human supervision and llms @xcite . rlaif approaches obtain pairwise preferences by scoring two outputs from a shared prompt @xcite , whereas rlcd generates outputs from two variants of a prompt @xcite . our method rlp-spg is the first attempt to simulate human preferences using a set of outputs. uncertainty quantification provides confidence scores for generations of llms, helping users decide when to trust these generation results @xcite . supervised methods fine-tune the language model to predict the uncertainty @xcite , while unsupervised methods measure uncertainty by calculating semantic entropy or semantic dispersion amongst generated answers @xcite . in this work, we measure uncertainty to selectively generate preference data. ## preliminaries we start by introducing the instruction following task @xcite . given user instructions @xmath0.g., "generate a definition for artificial intelligence"), we aim to develop a model π θ that generates high-quality responses y ∼ π θ (y|x) as judged by some latent reward model. in this study, we focus on rlhf for this task, due to its central role in instructionfollowing llms @xcite . rlhf usually consists of three steps: human preference collecting, reward modeling, and rl policy optimization @xcite @xcite . step 0, sft: rlhf generally begins with a pretrained model, which is fine-tuned with supervised learning on instruction-following demonstrations (x, y), to produce a model π sft (y|x). ## conclusion in this paper, we propose reward learning on policy (rlp), a novel framework to align llms with human preferences. rlp learns robust representations of the policy's data distribution via optimizing a multi-view information bottleneck loss. rlp also simulates preferences with a set of policy outputs, which enables confidence estimation and selective generation. extensive experiments demonstrate that rlp outperforms sota baselines.