Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023)

Abstract: Reinforcement learning from human feedback serves as a crucial bridge, aligning large language models with human and societal values. This alignment requires a vast corpus of human feedback to learn a reward model, which is subsequently used to finetune language models. However, we have identified that the reward model often finds shortcuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses. The emergence of length bias often induces the model to favor longer outputs, yet it doesn’t equate to an increase in helpful information within these outputs. In this paper, we propose an innovative solution, applying the Product-of-Experts (PoE) technique to separate reward modeling from the influence of sequence length. In our framework, the main expert concentrates on understanding human intents, while the biased expert targets the identification and capture of length bias. To further enhance the learning of bias, we introduce perturbations into the bias-focused expert, disrupting the flow of semantic information. Experimental results validate the effectiveness of our approach, indicating that language model performance is improved, irrespective of sequence length.

## introduction in recent years, the field of natural language processing has witnessed remarkable advancements with the emergence of powerful models like instructgpt @xcite , @xcite , claude @xcite , and others. these models have displayed remarkable proficiency in understanding human queries and providing helpful responses. their success can be attributed to a two-step learning process that involves supervised fine-tuning (sft) followed by the utilization of reinforcement learning from @xcite : what does the phrase "a stitch in time saves nine" mean? one of the key components of rlhf is reward modeling (rm), which involves learning a reward function from human preferences or demonstrations. this allows an rl agent to optimize its behavior based on the feedback received from reward model @xcite . however, the process is not without challenges. human preference data can be noisy and subject to inconsistencies among different annotators, leading to suboptimal results @xcite . for example, reward gaming, a well-known and pervasive issue, refers to the phenomenon where trained models exhibit undesirable patterns and generate low-quality outputs while still receiving high rewards @xcite . these complexities emphasize the need for careful consideration and robust methods to ensure reliable and meaningful reward functions in rlhf. as shown in figure 1 , a similar reward gaming issue arises in nlp reward modeling. we have observed that the reward model tends to reply on simple patterns, such as sentence length, to differentiate between good and bad responses. typically, the reward model assumes that longer responses are better, which hinders its ability to learn the true human intent and preference. addressing this problem is crucial for improving the effectiveness of nlp reward modeling and capturing the true nuances of human language. in this paper, we propose a product-of-experts (poe)-based method @xcite that consists of two expert models to decouple human intent and response length during the reward modeling phase. the first expert operates similarly to a standard reward model and focuses on learning the true human intent behind responses. the second expert, referred to as the bias-only expert, is designed to learn simple patterns, specifically the length of responses. it employs a smaller model capacity and a larger learning rate to capture coarse-grained information of inputs. additionally, stochastic perturbations are introduced into the inputs of the bias-only expert, intentionally disrupting the semantic information present in the input. to summarize, the main contributions of our work are followings: ## related work reinforcement learning from human feedback. using human preference feedback is a popular way for realizing ai alignment @xcite . preferences are often provide numerical value or demonstrations @xcite without requiring expert proficiency or finegrained feedback . alignment bring a potent capability to state-of-the-art generative foundation models, like instructgpt @xcite , sparrow @xcite , claude @xcite , which means this method is of great success in the paradigm of learning with human feedback. some prior work have explored using human feedback to improve various tasks, such as summarization @xcite , dialogue @xcite , translation @xcite , event generation @xcite , semantic parsing (lawrence and riezler, 2019) and instruction following @xcite . these work can be categoried as supervised fine-tuing or reward modeling on well-constructed human annotations information, the latter is also known as a vital phase in rl from human feedback @xcite . our work falls within the realm of rlhf and aims to awaken lm with both harmless and helpful abilities. reward hacking. goodhart's law foot_0 @xcite can be formulated a tough challenge in numerous fields. a few approaches have been proposed for reducing overoptimization in general reinforcement learning @xcite , as well as in reward models @xcite . the overoptimization problem of reward model can generally regard as a special case of reward gaming, also known as reward hacking @xcite . in addition, @xcite proposed to systematically analyze reward misspecification in rl by creating a set of domains where the agent optimizes a hand-engineered proxy reward function. in this study, we consider the length of human preferences as a confounding factor that hinders the reward model from accurately assessing the quality of model responses based on true human intent. products-of-experts. products-of-experts (poe) @xcite has been proposed as an alternative to mixture model to compensate for their poor efficiency in high dimensional space. this technique is often based on the principle of wisdom of crowds, which suggests that aggregating multiple models can lead to better performance than relying on a single model. @xcite firstly use poe to build a paradigm that train a debiased model ensemble with a bias-only model. the goal is to encourage the debiased model to utilize orthogonal information with information from the bias-only model. typically, this kind of method @xcite usually contains two stages. in this work, we adopt the end-to-end manner like @xcite which jointly learn the bias-only model and the debiased main model simultaneously, therefore, our reward model can take advantage of poe to use a weak learner to capture the length shortcuts without any prior information about the length of sentences, and the main model can purely attain the correct knowledge that is suitable for human preference. ## preliminary for the purpose of implement rlhf, we follow the pipeline in @xcite . it is usually made up of three phrases: 1) supervised fine-tuning, 2) reward modeling, 3) reinforcement-learning optimization, our attention is directed towards the last two phases. sft: it begins with a generic pre-trained lm, which is fine-tuned using supervised learning on a high-quality instruction dataset. this allows the model to follow various instructions, perform dialogue and dialogue. as a result, we obtain an lm π sft during this phase. reward modeling: according to the bradley-terry model @xcite , a reward function is hard to describe and needs to be learned from preferences among trajectories. the reward model r θ is trained on human preference dataset to predict which response @xmath0, 1} is better as judged by human, given a query x. if the response preferred by human is y i , the rm loss can be expressed as: ## length bias in reward model in this section, we present the phenomenon of length bias in reward models and utilize causal analysis to examine the underlying reasons for this occurrence. we shed light on why the reward model exhibits a preference for longer sentences and delve into the causal factors contributing to this bias. ## proposed method the section describes the framework and algorithm of our method. we introduce an approach to establish a debias framework that can significantly mitigate the length bias at the rlhf stage. ## analysis and discussion in this section, we uncover valuable insights into the effectiveness and limitations of our approaches, paving the way for future advancements in conversational ai systems. ## conclusion in this study, we investigate the issue of length bias in nlp and propose a poe-based method to mitigate this bias. our work sheds light on the challenges associated with reward modeling, emphasizing that it is not a straightforward task and various difficulties may impede the models from capturing the true human intent. by addressing the problem of length bias, our research highlights the importance of developing techniques that enable models to learn and generate responses aligned with genuine human intentions. further research and advancements in this area are necessary to overcome these obstacles and enhance the performance and reliability of nlp models in realworld applications.