Reward Modeling Requires Automatic Adjustment Based on Data Quality (2024)

Abstract: In Reinforcement Learning from Human Feedback (RLHF), the reward model plays a crucial role in aligning language model outputs with human values. The human preference data used to train the reward model consists of a prompt and a response pair, with humans annotating which response better aligns with human value preferences. Due to the complexity and subjectivity of the annotation task, multiple organizations including OpenAI and Anthropic report significant noise in the human preference datasets, leading to instability and deviation in reward model training from human values. We discover that the difference in scores assigned to response pairs by the reward model effectively indicates the quality of data, and data of varying qualities show significant distinctions in reward model training. We introduce a method that automatically adjusts reward modeling based on data quality, reducing the impact of noise and making full use of dataset. Experiments on multiple human preference datasets demonstrate that our method stabilizes reward model training and significantly enhances the alignment performance of RLHF.

## introduction in the field of artificial intelligence (ai) and large language models (llms), "alignment" is an important topic @xcite . it refers to the process of ensuring that the behavior of ai systems aligns with the intentions of their designers and the expectations of users @xcite . in llms, alignment methods based on reinforcement learning from human feedback (rlhf) are preferable to supervised fine-tuning (sft) for learning generation, as they first require learning to discriminate, which is simpler and more generalizable @xcite . rlhf involves two main steps: first, training a reward model using preference data collected from lots of crowdworkers to discern outputs that are more aligned with human preferences; second, optimizing the language model using reinforcement learning to maximize the reward @xcite . the reward model plays a crucial role in the rlhf process, and our goal is to make it a reliable proxy for human preferences @xcite . however, the human preference datasets used for training the reward model contain substantial noise @xcite . human preference data consist of a prompt and a pair of potential responses, where crowdworkers are tasked to choose the response that better aligns with human preference. on one hand, it is challenging to establish a unified standard of human preference annotation for many crowdworkers @xcite ; on the other hand, the response pairs may be difficult for crowdworkers to distinguish, leading to significant noise in the datasets @xcite . multiple organizations including openai and anthropic have reported this phenomenon, finding that the agreement rate among different crowdworkers is only about 60% to 70% @xcite . the noise in the preference datasets causes the language model to deviate from human value preferences @xcite . we discover that the difference in scores assigned to response pair by the reward model effectively indicates data quality. after being trained, the reward model can score responses based on the prompt. we score the response pair in human preference data using the reward model, subtracting the score of the rejected response from the chosen response. this score difference quantifies the extent to which humans prefer the chosen response over the rejected one, which we term preference strength. we categorize different data groups based on preference strength, which we find to ef- therefore, we propose a method that automatically adjusts reward modeling based on data quality to fully utilize human preference datasets. specifically, we map preference strength to soft labels, which vary for data of different qualities, thereby optimizing differently according to the data quality. compared to the original reward modeling, our approach has two key advantages: (1) it mitigates the impact of noise and ambiguous preferences. after identifying noisy and ambiguous preferences based on preference strength, our method uses label smoothing @xcite to alleviate overfitting to these data. (2) it models preferences more effectively. we introduce an adaptive margin based on preference strength in the reward modeling, explicitly teaching the model to assign more distinct scores to responses with larger differences, which helps the model better learn human intentions @xcite . experimental results show that using our proposed reward modeling method can stabilize the reinforcement learning process and improves the final alignment performance. in summary, our contributions are as follows: ## automatic adjustment of reward modeling based on data quality the introduction of preference strength provides us with a basis for distinguishing among data of varying quality in human preference datasets. based on data quality, we can mitigate the impact of noisy data and fully leverage the dataset. ## related work in addressing the potential risks associated with language models, such as generating false information, biased text, or harmful content, it is crucial to align these models with human values @xcite . this alignment is typically characterized by the principles of being helpful, honest, and harmless (3h) @xcite @xcite . current approaches attempt to incorporate 3h data into sft to guide models towards more ethical and moral responses @xcite @xcite . however, these models still fall short of human-level performance in terms of safety and groundedness, necessitating more effective control strategies @xcite . rlhf emerges as a straightforward method for achieving this alignment. rlhf requires minimal supervision from reward models as human proxies and adapts the agent through numerous iterative trials within the reinforcement learning framework. recent works have explored this direction, demonstrating the potential of rlhf in aligning llms with human preferences @xcite @xcite @xcite @xcite . rlhf, despite its promise, faces several challenges that affect its accuracy and effectiveness. a key issue is the noisy and ambiguous nature of hu-man preferences @xcite @xcite . this uncertainty in the data can significantly impact the reward models' accuracy and effectiveness. additionally, human feedback often contains inherent biases or misalignments influenced by the evaluator's goals or perspectives. this can lead to increased bias in rlhf models, such as chatgpt and claude, potentially due to biases in data collection and evaluator demographics @xcite @xcite . furthermore, interpreting and modeling human feedback is complex. different evaluators may have varying interpretations of the same scenario, leading to inconsistencies in the feedback provided @xcite . this variability poses a significant challenge in accurately capturing and modeling the intended human preferences within the reward model. ## conclusion in this paper, we focus on improving the reward model in rlhf from the perspective of utilizing preference data. we first differentiate data of varying quality in the dataset based on preference strength. then, we investigate the impact of data of different qualities on preference modeling and employ various methods to better utilize these data. finally, we summarize several methods, compare them with the original approach, and find that they lead to more stable training processes and better alignment effects.