Rethinking the Role of Proxy Rewards in Language Model Alignment (2024)

Abstract: Learning from human feedback via proxy reward modeling has been studied to align Large Language Models (LLMs) with human values. However, achieving reliable training through that proxy reward model (RM) is not a trivial problem, and its behavior remained as a black-box. In this paper, we study the role of proxy rewards in the LLM alignment via ‘reverse reward engineering’ by composing interpretable features as a white-box reward function. We aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL). Our findings indicate that successfully emulating the gold reward requires generating responses that are relevant with enough length to open-ended questions, while also ensuring response consistency in closed-ended questions. Furthermore, resulting models optimizing our devised white-box reward show competitive performances with strong open-source RMs in alignment benchmarks. We highlight its potential usage as a simple but strong reward baseline for the LLM alignment, not requiring explicit human feedback dataset and RM training.

## introduction to align large language models (llm) with human values like helpfulness, human feedback-based learning has been studied @xcite @xcite . reinforcement learning from human feedback (rlhf) is a dominant approach to exploiting human feedback. typically, the human feedback is used to train a proxy reward model (rm), and a policy model is optimized over the reward signal from the rm using rl @xcite . however, reliable rl training with the proxy rm is not a trivial problem. @xcite ) study 1 our code is available at github.com/naver-ai/rethinkingproxy-reward. first, we design white-box reward functions with interpretable features such as the length or relevance of the response. then, we conduct rl training using each of the designed functions as a proxy reward and deem it a success in reverse engineering if a monotonic relationship between the proxy and the ground truth (gold) reward scores is observed in the multiple evaluations. the reverse-engineered reward (blue) exhibits such a tendency, whereas the length-only reward (green) does not achieve the monotonic relationship, showing reward overoptimization @xcite . the overoptimization problem of the policy training against the imperfect rm. it is a phenomenon that the proxy rewards from the rm are increased continuously, but the true rewards are saturated or even decreased in fact. similarly, the policy model often finds undesired shortcuts in the reward signal from the imperfect black-box rm, i.e., "reward hacking" @xcite . a line of research has studied the innate limitations of human feedback. @xcite reveal prior knowledge affects the preference judgment in that crowd workers often choose the preferred response considering surface-level properties such as conciseness and specificity, while experts focus on more essential properties like factuality and completeness. similarly, @xcite further categorize the attributions of the preference judg-ments and confirm that human judgments could be biased by stylistic features like assertiveness. moreover, @xcite analyze the correlations between long response and rlhf, demonstrating a verbosity bias in the human feedback. in this work, we show merely combining interpretable features as a proxy reward function can substantially maximize the score of state-of-the-art rm @xcite and further enhance human preferences in various alignment benchmarks @xcite @xcite . specifically, we study the role of proxy rewards learned by human feedback in the rlhf by employing "reverse reward engineering" with interpretable features, such as length or relevance of the responses, to compose the white-box reward function, as shown in figure 1 . we define the goal of reverse engineering as achieving a monotonic relationship between the proxy and ground truth (gold) reward signals after training the model with the proxy reward in an rl manner. this is to verify whether training to maximize the proxy white-box reward during rl training also maximizes the gold reward signal at the test time. experimental results indicate that the key to imitating the gold reward signal lies in generating responses that are sufficiently lengthy yet relevant and faithful for open-ended queries while ensuring consistency in responses to closed-ended queries. contrary to previous study @xcite , our results demonstrate that solely optimizing towards lengthy response makes drastic drops in the gold reward, showing severe overoptimization. also, the reward branching according to query type reliably improves the win rate while not introducing unnecessary verbosity compared to other baseline reward options. furthermore, we find that the reward differentiation brings advantages in improving preference while minimizing alignment tax @xcite , a phenomenon where the increased preference accompanies the degraded performance on other nlp tasks. notably, the engineered white-box reward often results in even more improvements than the strong open-source rm like ultrarm-13b @xcite and also generally works well across llm backbones, demonstrating its potential usage as a baseline reward. our contributions are summarized in three folds: ## related work reinforcement learning from human feedback reinforcement learning from human feedback (rlhf) is the most prevalent approach to leverage the human preference @xcite @xcite @xcite . these approaches include the reward modeling stage to develop a proxy rm predicting the human preference over the response, e.g., typically as a scalar format. then, the trained proxy rm computes reward scores for the sampled responses in the rl stage, i.e., rollout. recently, replacing human feedback with ai or synthetic feedback has been studied @xcite @xcite . there are attempts to leverage multiple rewards beyond the overall reward @xcite @xcite . @xcite devise fine-grained rewards consisting of relevance, factuality, and information completeness to enhance long-form question-answering. pitfalls of reward modeling and human feedback rlhf faces various challenges, especially in reward modeling and human feedback itself @xcite . @xcite present the overoptimization problem by the imperfect proxy reward models. the issue arises when the policy, tailored to enhance the estimated rewards, fails to improve real-world rewards. many studies try to mitigate the overoptimization problem @xcite . similarly, @xcite study reward inconsistency and leverage a sentence encoder to increase the reward consistency. the work leverages sim-cse @xcite to augment the preference dataset and normalize reward scores. the policy models within the rlhf often find and exploit undesired shortcuts or errors in the rms to maximize the proxy rewards. the phenomenon is known as reward hacking @xcite , ## discussion and future works our study does not cover all perspectives on the role of rms, such as safety or reasoning. we try to reverse reward engineering over 'general helpfulness', one of the perspectives on which many alignment learning studies have focused @xcite . in other words, we do not argue that our findings are universally applicable. additionally, the design choice of white-box rewards could be substituted with other features, depending on the target value or domain of interest. exploring a broader range of interpretable features could enhance our understanding @xcite , and investigating aspects of the gold rm that these features do not capture presents another promising direction for future research. on the other hand,