When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning (2025)

Abstract: While Reinforcement Learning from Human Feedback (RLHF) is widely used to align Large Language Models (LLMs) with human preferences, it typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints.Although personalized preference learning addresses this by tailoring separate preferences for individual users, the field lacks standardized methods to assess its effectiveness. We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence. Through extensive experiments comparing eight personalization methods across three preference datasets, we demonstrate that performance differences between methods could reach 36% when users strongly disagree, and personalization can introduce up to 20% safety misalignment. These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems.

## introduction reinforcement learning from human feedback (rlhf) has been effective in aligning pre-trained large language models (llms) with human preferences, improving their helpfulness, harmlessness, and instruction-following abilities @xcite . however, standard rlhf assumes a homogeneous set of preferences, failing to account for the diverse and sometimes conflicting nature of human values @xcite . this leads to biases toward the perspectives of a western, democratic, postgraduate-educated demographic @xcite , even though llm users represent a wide range of cultural and ideological backgrounds, with a majority being non-u.s. users across the world @xcite . personalized preference learning aims to bridge this gap by adapting llms to the specific preferences of individual users. with the increasing adoption of general-purpose llms, researchers have begun exploring personalization in open-domain contexts @xcite @xcite . however, significant challenges remain, particularly concerning the evaluation of these personalized models. firstly, the evaluation benchmarks are inadequate and incomparable across different studies. existing studies rely either on narrow-domain realworld data @xcite or entirely synthetic general-domain data @xcite , limiting the robustness of evaluation. furthermore, the use of disparate datasets across studies impedes fair and direct comparisons between personalization methods. secondly, the evaluation frameworks fail to address the practical constraints and unintended consequences. existing research often assumes a fixed number of data points per user, neglecting the practical constraints of real-world data availability. how do different personalization algorithms perform under varying levels of data availability? moreover, the potential side ef-the rst step is to consult with a licensed physician. firstly, you need to con rm if your grandmother meets the eligibility criteria. fects of personalization, beyond the scope of (kirk, 2024), remain largely unexplored. does personalization degrade general llm capabilities or introduce safety vulnerabilities? to address these gaps, we introduce a novel, multi-faceted framework for benchmarking opendomain personalized preference learning techniques. our contributions are as follows: personalization in machine learning refers to tailoring systems to generate predictions that align with each individual's preferences and needs. this concept has been extensively studied in recommendation systems @xcite and dialogue systems @xcite . with the widespread adoption of llms, personalization has become even more critical to ensure these models effectively serve diverse global users with varying preferences-a challenge that remains underexplored in current alignment pipelines @xcite . unlike traditional task-specific ml systems, llms are general-purpose models designed to handle a wide range of tasks and domains. this versatility makes personalization both more important and more challenging, as the model must adapt its broad capabilities to each user's specific needs and preferences. several approaches have been proposed, including prompting @xcite , user embedding learning @xcite , latent variable modeling @xcite , meta-learning @xcite , multi-objective reinforcement learning @xcite , preference elicitation @xcite , prompt optimization @xcite , and context compression @xcite . however, these methods have typically been evaluated on different datasets which prohibits a fair comparison between them. evaluation of personalization presents unique challenges beyond traditional preference learning. while domains like recommender systems have established evaluation frameworks using per-user interaction histories @xcite , evaluating natural language outputs and collecting general-domain preference data at scale remains challenging @xcite @xcite . existing survey-based datasets, such as opinionqa @xcite and globalopinionqa @xcite , provide large-scale, real-world general-domain data but are limited to multiple-choice formats, which fail to capture realistic llm usage scenarios. in contrast, generation-based datasets such as @xcite al. (2024); wang et al. (2024); stiennon et al. (2020) contain preferences for open-ended generations but remain restricted to narrow domains. other sources, like personal reddit @xcite and persona-db @xcite , scrape reddit and twitter data but cannot be publicly released due to privacy concerns. prism @xcite offers diverse preference data for llm generations but remains limited in size to effectively model individual annotators. in the absence of large-scale, general-domain preference datasets, recent research has explored synthetic data generation via role-playing agents and llm-as-a-judge evaluations @xcite @xcite @xcite . while these methods may not fully capture real user preferences (hu and collier, 2024), recent works suggest that synthetic benchmarks can serve as viable testbeds for evaluating personalization, even if they don't comprehensively represent all human preference variations @xcite . as noted in @xcite , perfect simulations of human preferences may not be necessary for these simulation to provide valuable insights and help develop better algorithms. ## preliminaries on personalized preference learning preference learning systems can take various forms, including reward models (rms), where a model assigns a numerical preference score; preference ranking models, which make comparative judgments between multiple candidates; and generationbased policy models, where the model explicitly generates preference judgments, sometimes accom-panied by explanations or feedback. in this section, we review previous approaches to learning personalized preferences, with a particular focus on reward models, which constitute the majority of existing methods. ## results personalized rm achieves the best performance across all datasets. as shown in figure 2 , , in terms of reward modeling accuracy, personalized rm consistently outperforms all methods across all datasets. its success over individual reward modeling can be attributed to its collaborative learning -leveraging signals for all users. individual reward models, while serving as simple yet effective baselines, achieve the second-best performance. both of them surpass other baselines by a substantial margin on personal llm and performs even better on p-soups. we attribute this to its superior ability to handle the high interpersonal disagreement nature of p-soups. on tl;dr, all methods-except rag-perform comparably. rag, in contrast, exhibits the weakest performance among all personalization methods across all datasets, with accuracy approaching that of random guessing. this is likely due to the limitations of the 7b model in capturing nuanced user preferences through in-context learning. ## conclusion this work addresses gaps in llm personalization research by introducing a systematic evaluation framework. we establish a principled methodology for characterizing preference datasets through: inter-user disagreement, intra-user consistency, and minority representation. our analysis across p-soups, tl;dr, and personal-llm datasets reveals distinct challenges that personalization methods must address, from high disagreement to varying levels of minority viewpoint representation. our comprehensive evaluation framework extends beyond accuracy to address practical constraints and potential risks. through this lens, we evaluate eight representative personalization methods, finding that individual rm provides a strong baseline while collaborative approaches like prm achieve up to 6% improvement. notably, some methods successfully preserve minority preferences that standard rlhf would overlook. however, we also identify a "personalization tax," where optimizing for individual preferences can degrade model safety and reasoning capabilities. these findings demonstrate both the promise and challenges of personalization. we hope this work's systematic framework and empirical insights will guide the development of more robust, inclusive, and responsible personalization approaches that can better serve diverse global user.