Rethinking Diverse Human Preference Learning through Principal Component Analysis (2025)

Abstract: Understanding human preferences is crucial for improving foundation models and building personalized AI systems. However, preferences are inherently diverse and complex, making it difficult for traditional reward models to capture their full range. While fine-grained preference data can help, collecting it is expensive and hard to scale. In this paper, we introduce Decomposed Reward Models (DRMs), a novel approach that extracts diverse human preferences from binary comparisons without requiring fine-grained annotations. Our key insight is to represent human preferences as vectors and analyze them using Principal Component Analysis (PCA). By constructing a dataset of embedding differences between preferred and rejected responses, DRMs identify orthogonal basis vectors that capture distinct aspects of preference. These decomposed rewards can be flexibly combined to align with different user needs, offering an interpretable and scalable alternative to traditional reward models. We demonstrate that DRMs effectively extract meaningful preference dimensions (e.g., helpfulness, safety, humor) and adapt to new users without additional training. Our results highlight DRMs as a powerful framework for personalized and interpretable LLM alignment.

## introduction reinforcement learning from human feedback (rlhf) @xcite @xcite @xcite has proven to be a powerful approach for fine-tuning large language models (llms) to serve as better general assistants @xcite , ai agents @xcite @xcite , and clinical decision support systems * equal contribution. @xcite @xcite . typically, llms are optimized using a scalar reward model trained on human preference data, acting as a proxy for overall user satisfaction. however, this approach has two key limitations: (1) it often reflects the preferences of the majority, potentially marginalizing underrepresented groups @xcite and failing to capture the full diversity of human preferences, and (2) it struggles to represent the complex, multifaceted, and sometimes conflicting nature of human preferences with a single scalar reward @xcite @xcite . as demand grows for more personalized llms @xcite , researchers have explored ways to capture fine-grained, multidimensional human preferences. some studies have introduced datasets that evaluate multiple aspects such as relevance, correctness, completeness, helpfulness, and harmlessness @xcite @xcite . building on these datasets, others have proposed multiobjective optimization methods @xcite to accommodate diverse user needs @xcite . however, these approaches face significant challenges: collecting fine-grained human annotations is expensive, and using gpt-generated labels can introduce biases. considering that large-scale binary preference datasets-where users simply compare two responses-are easier to collect and more widely available, we ask the question: can we infer multidimensional human preferences directly from large-scale binary comparisons? to address this question, we propose decomposed reward models (drms), a framework that extracts fine-grained human preferences using binary comparison data. unlike traditional probabilistic models such as bradley-terry (bt) @xcite or pairwise preference figure 1 : illustration of the decomposition pipeline in drms. in the original single-dimensional head, a prompt-response pair can be predicted incorrectly. in contrast, drms capture preferences along multiple dimensions, aligning more effectively with the complex and multifaceted nature of human preferences. models @xcite , our approach represents human preferences as d-dimensional vectors, corresponding to the learned weights of the final linear layer in a reward model. we show that this vector-based representation establishes a connection between principal component analysis (pca) and preference learning. our method constructs a dataset of embeddings for all prompt-response pairs and applies pca to the differences between chosen and rejected response embeddings. this process identifies orthogonal basis vectors, each capturing a distinct human preference direction. these basis vectors can be combined with their corresponding feature extractor to construct reward models. the framework is shown in figure 1 . drms are particularly effective for personalized preference learning. given a small adaptation dataset from a new user, we compute coefficients for the basis vectors and form a linear combination that best aligns with the user's preferences. this approach allows for flexible, scalable, and interpretable preference modeling without requiring additional training. in our experiments, we first show that the learned drm basis vectors capture diverse human preference attributes. for instance, one vector may strongly correlate with safety, while another may reflect humor. next, we evaluate drm's effectiveness in adapting to user preferences at test time. our results show that drms outperform both the single-head reward model and previous test-time alignment methods based on reward ensembles. these findings emphasize drm's advantages in modeling and adapting to diverse human prefer-ences. in addition, we also provide an explainability study to better understand human preferences. in summary, our work makes three key contributions. ( 1 ) we propose a vector-based representation of human preferences, providing a more structured and interpretable approach to preference learning. ## preliminary the bradley-terry (bt) model @xcite is a probabilistic framework commonly used in preference learning. given a prompt x and two responses, y 1 and y 2 , the bt model defines the probability of y 1 being preferred over y 2 as: ## methodology the standard approach to preference learning relies on a scalar-valued reward model, which may not adequately capture the full complexity of human preferences. to address this limitation, we first introduce a vector representation of human preference and establish its connection to pca. building on this, we propose a pca-based method that decomposes human preference into multiple basis vectors, allowing any preference to be represented as a linear combination of these vectors. this approach enables a novel paradigm for diverse reward modeling without additional training. ## experiment in this section, we conduct extensive experiments to evaluate the effectiveness of drms, focusing on the diversity and interpretability of the decomposed heads, as well as their adaptivity to downstream human preferences. ## conclusion in this paper, we establish the connection between preference learning and pca, introducing decomposed reward models (drms). drms represent diverse human preferences as a set of orthogonal basis vectors using a novel vector-based formulation of preference. this approach enables efficient test-time adaptation to user preferences without requiring additional training, making it both scalable and practical. beyond the efficiency, drms provide a structured way to understand human preferences. by decomposing complex preferences into interpretable components, they reveal how preferences are formed and interact. we hope this work inspires further research into the fundamentals of human preference learning while promoting more transparent and personalized ai systems.