On Diversified Preferences of Large Language Model Alignment (2024)

Abstract: Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs’ interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators’ different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.

## introduction large language models (llms), such as chat-gpt @xcite and llama @xcite , have significantly accelerated the development process toward artificial general intelligence (agi). among the key factors for such great achievement, the alignment technique, which finetunes llms with human feedback @xcite , has played an essential role in training llms' responses to follow human values (e.g., helpfulness and harmlessness) @xcite . among the llm alignment algorithms, reinforcement learning from human feedback (rlhf) @xcite has become the mainstream solution, which first learns a reward model (rm) representing human preferences and then updates llms via the proximal policy optimization (ppo) @xcite toward generating responses with higher rm scores. alternative alignment methods also have been sequentially proposed for better computational complexity and training instability, such as raft @xcite , dpo @xcite , rrhf @xcite , and apo @xcite . the performance of these alignment methods highly depends on the quality of human preference data (x, y w , y l ), where x is the input query to the llm, and response y w is preferred to response y l under the human annotation @xcite . ideally, the preference datasets should uniformly be helpful, harmless, benevolent, and unbiased to guide the llm alignment. however, in real-world scenarios, individuals can have diversified preferences on the same topic based on their different experiences, educational backgrounds, religions, and cultures @xcite . even for the same person, his or her expected model answer to a particular question can vary depending on different scenarios @xcite . the annotation disagreement, which is caused by different annotators or the same annotator in different scenarios @xcite , will significantly hinder the effectiveness of alignment methods @xcite @xcite . to identify the diversified preferences quantitatively, we select five commonly used human feedback datasets, train an rm on each, and then test the performance on the other sets (details in section 3). we plot the observation results in figure 1 . we observe that training rm on a single preference data source may cause inconsistent reward distribution shifts (middle plot), result in diverse reward values (right plot), and compromise the performance of other sets (left plot). the result indicates that different human preference datasets have different preference distributions @xcite . hence, a more comprehensive understanding of the impact of diversified human preference datasets on the reward model becomes crucial, yet it has not received adequate attention and remained unexplored in the llm alignment domain. in our exploration, we found the over-rewarding phenomenon, that is, the vanilla rms tend to output extreme rewards on samples, which damages the rms and llm alignment. to enhance the efficiency of leveraging the diversified preference datasets, inspired by multi-objective optimization methods @xcite , we regard rms as a shared reward additionally with a customized reward drift. the shared reward represents the shared preferences across datasets (or general human preferences) and the reward drift contains individual or domain-specific preference information @xcite . then, we introduce a multi-objective reward training scheme (more) to capture the shared (general) preference information, which adopts a novel reweight techniques to minimize the mean gradient of enlarging reward drifts. with more, rms can capture a broader range of preferences and mitigate the impact of reward drifts. the main contributions of this paper are: ## background large language model alignment parameterized by θ, a reward model (rm) is a mapping r θ : x × @xmath0, which provides a real-valued reward score r θ (x, y) evaluating a textual response @xmath1, y 2 , . . . , y m ) ∈ y corresponding to an input prompt @xmath2, x 2 , . . . , x n ) ∈ x . given a sample (x, y w , y l ) ∼ d from a preference dataset d, r θ is expected to provide a preference score with r θ (x, y w ) > r θ (x, y l ), representing the response y w is preferred. following the bradley-terry model @xcite , the rm learning objective on the preference dataset (x, y w , y l ) ∼ d is defined as: with a well-learned reward r θ (x, y), llm alignment optimizes the generation policy π(y|x) by maximizing the expected reward value: calibration error calibration error is an effective method to estimate the confidence of a model's outputs @xcite . we divide the confidence interval [0, 1] with finite samples into m bins with equal length (1/m ). then, we place model predictions into these bins according to their prediction confidence. let b m be the set of indices of samples that fall into the internal ( m-1 m , m m ]. we calculate the corresponding accuracy and average confidence of each bin as follows: in the context of reward modeling, the prediction confidence @xmath3. for a set of n samples, we can compute the expected calibration error as follows: we set @xmath4. numerous studies have focused on improving the calibration performance of statistical machinelearning systems @xcite @xcite . furthermore, the calibration error of neural networks provides additional information for users to determine whether to trust the model's predictions, especially for modern neural networks that are more challenging to interpret @xcite . in the field of natural language processing, studies have revealed a positive relationship between calibration performance and the reduction of hallucination @xcite , and the evaluation of pre-trained language models @xcite . the calibration error has demonstrated its ability to evaluate the performance of language models. in this paper, we first employ the calibration error to evaluate the rms. subsequently, we investigate the implicit connection between rms and llm alignment under diversified preferences. ## empirical study of diversified preferences we start with an empirical analysis of diversified preferences in reward modeling on multiple sources @xmath5, . . . , d k }, where each data source d k contains the preference comparison pairs from different tasks @xcite , domains @xcite , or individuals @xcite . in this paper, we selected summarize @xcite , webgpt @xcite , helpful&harmless @xcite , and oasst1 @xcite as the different preference sources to empirical analysis the phenomena of diversified preferences. we use pythia-1.4b @xcite as the rm base, and finetuned rms with comparisons from each source. the experiment setup aligns with section 5. the reward distributions across various rms exhibit diversity when applied to the same dataset. we analyze and present the variation in rewards (defined as the difference in reward values assigned by an rm to the winning and losing samples) offered by these rms, as illustrated in figure 1 (additional results in figure 8 and 9 in appendix). compared with the results of raw model rm raw , we observe that training on different datasets results in diverse reward values (right plot) and distribution shift (middle plot). specifically, the reward value distribution of rm harmless shifts from the rm raw in a certain degree. while the reward value distributions of rm helpful , rm webgpt , rm oasst1 and rm summ. shifts to the a different direction. moreover, despite the distribution of rm helpful , rm webgpt , rm oasst1 and rm summ. are similar, the mean-variance of their reward values are quite different. furthermore, when considering the accuracy gains illustrated in figure 1 (left plot), the observed shift in reward distribution indicates that the learned reward values from preference datasets are diversified. to effectively capture the shared reward values across these diversified preferences, it becomes necessary to formulate a new problem approach for reward modeling on diverse preference datasets. ## multi-objective reward learning in this section, we propose our reward modeling on diversified preference datasets, highlighting the implicit reward drift during the reward learning process and its negative impacts. then, we present the more training schemes to mitigate the reward drifts as a feasible solution. to maintain the integrity of our paper, we leave our quantitative analyses of reward modeling on diversified preferences in the next section. ## experiments on reward modeling in this section, we present our experiments and quantitative analyses on reward modeling. the open-source code and data are available at https: //github.com/dunzeng/more . ## experiments on llm alignment in this section, we use the previously obtained rms for llm alignment experiments on alpaca @xcite , which is an instruction-tuned llama-7b model @xcite . we use reject sampling (rjs) @xcite as the alignment algorithms, where we sample 4 responses from alpaca with queries from h&h trainsets. our experiment mainly justifies the correlation between the calibration performance of rms and llm alignment performance. rm more works better than rm multitask for rjs aligning h&h with lower eces we finetune alpaca with the most preferred samples scored by previously obtained rms to align the human preference of h&h, following rjs loss (3). we show the alignment performance in ## additional discussions connections with data composition and ensemble-rm studies @xcite have empirically shown that the llm ability can be improved by adjusting the mixed training data ratio from different sources. however, the mixed proportion can be hard to search in practice. besides, other studies have shown that direct ensemble rms @xcite or merging rms' parameters @xcite during training could also improve the ability of rms. in practice, these approaches induce a large system burden for storing/training multiple rms, especially since the rms can be extremely large. in comparison, this paper focuses on training single rm on diversified datasets. connections with fine-grained reward and multi-dimensional reward existing research, including fine-grained reward @xcite and multi-dimensional reward @xcite , has increasingly focused on the importance of reward diversity. these studies categorize the utilization of diverse reward signals into two main strategies: first, integrating multi-dimensional preferences into a single dimension for aligning large language models (llms); and second, decoupling preferences across dimensions (reward models) to align llms on each dimension. this body of work underscores the necessity of addressing diverse preferences. in contrast, our primary contribution lies in the empirical analysis of diversified preferences within a single reward model, alongside our novel evaluation of expected calibration error (ece) on reward models. additionally, since both fine-grained and multi-dimensional reward methodologies yield scalar rewards, our findings regarding ece are relevant for assessing these approaches. suggestions for reward model training this paper reveals two main suggestions for future reward model training works. first, evaluate rms with reward accuracy and calibration error. reward accuracy is insufficient to evaluate the ability of rms due to model capacity and data quality. our work suggests that the community additionally focuses on the calibration performance of rms. besides, increasing the diversity of preference data samples can ensure the robustness of the reward modeling process. due to the preference information being typically noisy, learning reward information from mixed diversified datasets can be beneficial. ## limitations we only conducted experiments using the conventional rjs algorithm in llm alignment tasks. as a reward modeling algorithm that captures shared preference information, more depends on the quality of the applied data. therefore, the correlation of ece of rms and llm alignment performance in other alignment algorithms requires further exploration. besides, the training datasets we used contain violence, abuse, and biased content that can be upsetting or offensive to particular groups of people.