Not All Voices Are Rewarded Equally: Probing and Repairing Reward Models across Human Diversity (2025)

Abstract: The advancement of Large Language Models (LLMs) has made ensuring their trustworthiness increasingly critical, especially in terms of fairness across diverse human groups. While modern LLMs are aligned with user preferences through Reinforcement Learning from Human Feedback (RLHF), the reward models used for alignment are trained on preference data that may both reflect societal biases and suffer from demographic skewness, as labeler populations are often uneven due to systemic accessibility or participation gaps. In this work, we reveal that reward models can exhibit significant discrepancies across different demographic groups, posing a fundamental challenge to fair and robust alignment. Using real-world datasets, we conduct the most comprehensive study to date, auditing various state-of-the-art reward models across nine sensitive attributes, including age, gender, ethnicity, etc. Our evaluation spans both (1) the agreement level between reward models and specific user groups, and (2) the reward model’s preference toward responses associated with different groups. Based on these findings, we propose the first method to mitigate group disparities in reward modeling. Code is available at https://github.com/Violet24K/FaRM.

## introduction in this work, we focus on one important aspect of llm trustworthiness -fairness, which is about ensuring that models perform equitably across different demographic groups without systematically favoring or disadvantaging any particular population @xcite . it has been demonstrated that llms exhibit notable unfairness across demographic groups @xcite , and even widely deployed models such as gpt-3.5 and gpt-4 (openai, 2023) have been shown to encode substantial performance disparities across sex and race @xcite @xcite . however, while modern llms are trained through unsupervised pretraining, supervised finetuning, and rlhf @xcite @xcite , most existing work on llm fairness has been limited to examining the first two stages @xcite @xcite @xcite or evaluating fairness of only the final deployed llms @xcite @xcite . only recently has a seminal study @xcite revealed that rlhf introduces additional unintended geolocation bias into llms. specifically, the alignment process unequally affects the model's performance across different geolocation groups, leading to unfair overall outcomes after preference tuning. the standard rlhf pipeline @xcite consists of three stages: preference data collection, reward modeling, and model optimization using rl from the learned reward. consequently, the final model behavior is heavily influenced by the quality of the reward model, and, by extension, the collected human preference data, which may itself encode societal biases and demographic skewness. the effectiveness of rlhf fundamentally depends on the quality of reward model, trained from human preference data that may itself reflect societal biases and demographic skewness. as noted by @xcite , it remains challenging to answer whose preferences are being aligned during reward model training. this difficulty arises because reward model developers retain significant control over critical but often undocumented/untraceable alignment variables, such as who provides feedback. nevertheless, systematically understanding how reward models behave across different demographic groups is a crucial step toward identifying and mitigating potential biases. to this end, this work presents the most comprehensive study to date on demographic biases encoded in reward models (see ## preliminary and background we use calligraphic letters (e.g., a) for sets and bold capital letters for matrices (e.g., a). for matrix indices, we use a[i, j] to denote the entry in the i th row and the j th column. preference data and reward modeling. modern reward models @xcite are trained on preference datasets @xmath0, a w , a l }, where a w and a l are two responses to prompt x, with a w preferred over a l by a human annotator. the reward models are expected to assign a higher score to the preferred response, i.e., r(x, a w ) > r(x, a l ), thereby learning to approximate human preferences through pairwise comparisons. the mainstream approach @xcite trains reward models (parametrized by θ) by maximizing the log-likelihood of bradley-terry (bt) model @xcite : figure 2 : overview of probing reward model bias in this study. in section 3, we investigate how user preferences differ across demographic groups partitioned by sensitive attributes such as age and ethnicity. in section 4, we assess how well the reward model aligns with each group by comparing the model's preferences with those of the groups. in section 5, we directly evaluate the reward model's preferences on representative group responses, revealing inherent biases resulting from training on uneven human reward feedback. the trained reward model r θ then provides learning signals during subsequent rl phases. machine learning fairness. fairness has been a long-standing goal in ml, with extensive research from traditional classifiers to modern models including language models @xcite @xcite . various fairness metrics have been proposed to assess disparities across demographic groups @xcite , including statistical parity @xcite , equalized odds @xcite , and calibration within groups @xcite . for instance, given a set of demographic groups g, statistical parity expects that the output of the model parametrized by θ is statistically independent of the sensitive attribute a: in section 4.1, we generalize the fairness concepts to reward modeling settings. ## how preferences vary across demographic groups? we begin by presenting a quantitative analysis of how user preferences vary across demographic groups. this analysis leverages the prism alignment dataset @xcite (details in appendix c.1), which contains real-world interactions from over 1,500 participants worldwide. each participant completed a survey reporting: (1) their demographic attributes, including age, gender, employment status, education level, marital status, english proficiency, religion, ethnicity, and geographic location (see for a sensitive attribute (e.g., gender) with value space s containing all possible attribute values (e.g., male, female, non-binary, etc.), let u s denote all users with sensitive attribute value @xmath1.e., a demographic group), and p @xmath2. given a sensitive attribute which partitions all users into different demographic groups {u s } @xmath3, for each demographic group u s , we compute the group-level average for each response quality dimension as: 1 |us| @xmath4, where @xmath5, helpfulness, safety, ...} corresponds to a response quality axis. we visualize the aggregated preferences across demographic groups using radar charts in figure 3 , which illustrates group-level preferences by the sensitive attribute ethnicity. we observe that some demographic groups exhibit significantly different preference patterns along certain dimensions. for example, in figure 3 , black/african participants tend to assign higher scores to creativity, diversity, and personalization. this may be influenced by historical and cultural factors, including a strong tradition of creative expression in black communities. full figures and analysis can be found in appendix f.1. to assess the statistical significance of preference differences across demographic groups, we perform kruskal-wallis tests (details in appendix e.3) on each response quality dimension and report the resulting p-values in to identify which groups exhibit the most divergent preferences, we compute the max-min score difference for each attribute-dimension pair (s, d) we summarize the key takeaways from this section as follows. based on our analysis of real user survey data, certain demographic groups exhibit notably different preference patterns across various response quality dimensions. these differences suggest that alignment based on aggregated or unbalanced preference data may disproportionately reflect the values of overrepresented groups. given the observed differences in preferences across demographic groups, it is essential to examine how reward models capture, or fail to capture, this diversity. specifically, we aim to further investigate the question posed by @xcite : whose preferences are we aligning llms with, and, more crucially, whose preferences are overlooked? although various fairness metrics have been proposed @xcite , the notion of fairness in the context of preference modeling remains nascent in the existing literature. in @xcite , the authors use the performance gap of the same language model before and after rlhf alignment as an implicit indicator of how well the reward model used for alignment agrees with a demographic group. in this work, we present a more direct measurement of reward model fairness. ## mitigating demographic bias in reward models in the previous sections, we have studied that even state-of-the-art reward models exhibit strong bias across different demographic groups partitioned by various sensitive attributes. in this work, we propose the first method, fair reward modeling (farm) to mitigate such bias in the reward models. ## related works due to page limit, we summary the core related works in the main page, while a more comprehensive review can be found in appendix a. additionally, we discuss promising future directions in appendix b. reward modeling and rlhf. the dominant rlhf approach is the three-staged ppo-based framework we have introduced in the preliminary section. following the release of chat-gpt, researchers have developed alternative approaches, such as iterative rejection sampling fine-tuning @xcite and direct alignment algorithms @xcite . although direct alignment methods were originally proposed to bypass the reward modeling stage, subsequent studies have shown that their iterative, on-policy variants outperform their offline counterparts and are now more commonly adopted in practice @xcite . in these approaches, the intermediate policy is deployed to collect fresh responses and an external reward (preference) model is used to label them. in summary, despite methodological differences, all major rlhf and preference learning frameworks fundamentally depend on the quality of the reward model. ## conclusion in this work, we present a comprehensive study of fairness in reward models used for llm alignment. through both group preference and group perspective analyses, we demonstrate that state-of-the-art reward models exhibit significant disparities across demographic groups. to address this issue, we propose fair reward modeling (farm), a novel method that balances fairness and reward modeling through regularization. empirical results show that farm effectively mitigates demographic bias while almost maintaining model performance.