Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024)

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, it is desirable for them to be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.

## introduction in this paper, we explore the role of reward models (rms) within the framework of reinforcement learning from human feedback (rlhf). rms play a crucial role in aligning large language models (llms) as they provide a scalable way to integrate human preferences into the models' training process, guiding the optimization of their policies. to be more specific and provide more context, we first review the most standard and popular rlhf frameworks and the role of rms in this framework. arguably the dominant rlhf approach is a deep reinforcement learning (drl)-based framework, as developed in key studies @xcite @xcite . this framework operates in three stages: 1) preference data collection; 2) reward modeling based on the bradley-terry model @xcite ; 3) policy optimization using proximal policy optimization (ppo) @xcite and the reward model constructed in stage 2. this framework has achieved tremendous success in the posttraining of chatgpt @xcite and claude @xcite . these ideas also extend to other approaches, such as rejection sampling finetuning @xcite and iterative direct preference learning @xcite @xcite . in these approaches, the intermediate policy is typically iteratively deployed to collect new responses, uses the reward model to label the responses, and fine-tunes the model on the newly collected preference data. in all of these rlhf frameworks, the capacity of the reward model is crucial as it directly affects the quality of the aligned llms. the most popular reward modeling approach is based on the maximum likelihood estimation (mle) of the bradley-terry (bt) model @xcite . despite its widespread use, the bt model is rather limited in the capacity of capturing the complicated human preference @xcite @xcite . in addition to the capacity issue, common rms, like the bt model, are typically black-box models that output scores or preferences without providing human-interpretable explanations, making it subject to the widely observed phenomenon of reward hacking @xcite @xcite , where the aligned llms generate high-reward responses (rated by the rm) that do not align with actual human preferences @xcite @xcite . a notable example of this is the verbosity bias, where aligned llms produce longer-than-necessary responses because the rm favors length, regardless of quality @xcite @xcite . in this work, we aim to enhance reward models by making them more interpretable @xcite and steerable @xcite . using the aforementioned verbosity bias as an example, suppose the rm's output is decomposable, meaning that it assigns a high score to a response due to two factors: 40% for its helpfulness and 60% for its length. in this case, we can see that the rm may suffer from the verbosity bias. furthermore, if the rm is steerable, we could adjust its decision-making process to base its scoring 100% on helpfulness. this would be regardless of response length, thus mitigating the verbosity bias. enhancing the interpretability of rms also allows humans to verify whether rms have similar internal decision processes to humans when acting as proxies for human preferences. we believe that this human-ai interaction process could ensure that rms are consistent with human values and preferences, making rm-aligned llms more reliable and robust. at a high level, we propose a two-stage approach that first trains a multi-objective rm and then learns a gating layer that scalarizes reward objectives in a mixture-of-experts way. we then empirically validate its effectiveness by training such an rm with llama-3 8b (meta, 2024), and obtain state-of-the-art performance on rewardbench, a benchmark to evaluate rms. ## conclusion in this work, we addressed the critical issue of interpretability in reward models for rlhf in the context of aligning llms with human preferences. we proposed a novel two-stage approach, consisting of an armorm and a moe strategy with a gating network. our armorm, trained with llama-3 8b, achieved state-of-the-art performance on re-wardbench, demonstrating the effectiveness of our reward modeling approach.