Uncovering Factor-Level Preference to Improve Human-Model Alignment (2025)

Abstract: Large language models (LLMs) often exhibit tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. While crucial for improvement, identifying the factors driving these misalignments remains challenging due to existing evaluation methods’ reliance on coarse-grained comparisons and lack of explainability.To address this, we introduce PROFILE, an automated framework to uncover and measure factor-level preference alignment of humans and LLMs.Using PROFILE, we analyze preference alignment across three key tasks: summarization, instruction-following, and document-based QA. We find a significant discrepancy: while LLMs show poor factor-level alignment with human preferences when generating texts, they demonstrate strong alignment in discrimination tasks. We demonstrate how leveraging the identified generation-discrimination gap can be used to improve LLM alignment through multiple approaches, including fine-tuning with self-guidance.Our work highlights the value of factor-level analysis for identifying hidden misalignments and provides a practical framework for improving LLM-human preference alignment.

## introduction human preference for text is inherently multifaceted, influenced by an interplay of factors such as fluency, helpfulness, and conciseness. the relative importance of these factors is not static; it shifts depending on the specific task and context. for instance, a desirable summary should be concise and to the point, while creative writing might prioritize novelty and an engaging narrative. as large language models (llms) generate increasingly human-like text, a critical question arises: do llms prioritize these quality factors in ways that align with human expectations? this question becomes particularly pressing given recent findings highlighting discrepancies between llms' generation and discrimination abilities @xcite . we extend this line of inquiry to investigate whether such discrepancies also manifest at the factor level of preference alignment: do models prioritize individual quality factors consistently with human judgment during generation, and does this prioritization differ when discriminating between responses? despite significant advances in preference alignment @xcite @xcite , existing approaches take a coarse-grained view, measuring overall preferences while overlooking the underlying factors that drive them. recent work has begun examining finergrained preference aspects @xcite @xcite , yet systematic factor-level comparisons between human and model priorities remain limited. moreover, existing research has predominantly focused on generation settings, leaving open whether models exhibit consistent factor prioritization across different task settings. to address these gaps, we introduce profile, an automated framework designed to decompose and quantify how individual factors (e.g., fluency, helpfulness) contribute to overall preference decisions. profile quantifies each factor's contribution as a factor score. by comparing how factor scores of responses correlate with overall preference decisions, profile captures the factor-level preferences of humans and models. this enables systematic comparison between human and model priorities across both generation and discrimination settings. using this framework, we investigate three key research questions: we conduct comprehensive experiments across three preference alignment tasks-summarization, instruction-following, and document-based qaevaluating eight prominent llms. our analysis reveals systematic misalignments: models often do not prioritize quality factors in line with human expectations during generation. for instance, models frequently exhibit a strong preference for length regardless of the task, whereas human preferences for factors such as conciseness or detail vary contextually (figure 1 ). interestingly, we observe that these same llms demonstrate notably better factor-level alignment during discrimination tasks, specifically in evaluation settings where the model selects which of two outputs is better. this disparity between generation and discrimination alignment presents an opportunity, and we demonstrate ways to leverage the stronger alignment in discrimination to enhance the factor-level preference alignment during generation. our work underscores the importance of factor-level analysis for a deeper understanding of llm alignment and offers a pathway toward more genuinely human-aligned generative models. ## profile: framework for analyzing human-model alignment we introduce profile (probing factors of influence for explainability), a framework that provides a systematic way to decompose overall response level preferences into their underlying factors. profile makes explicit the notion of factorlevel preference-the extent to which specific factors such as fluency, factual accuracy, or conciseness influence which response is preferred. by uncovering these factor-level preferences, profile enables a more interpretable analysis of human and model choices and provides a principled basis for comparing and improving their alignment. profile centers on the computation of factor scores, which quantify the influence of individual factors on overall preferences. to compute these scores, profile proceeds in three steps: (1) we define overall response-level preferences for humans and models ( § 2.2); (2) we quantify the manifestation of each factor in every response based on our factor taxonomy ( §2.3); (3) we compare these pairwise factor manifestations with preference labels to measure concordance, thereby deriving factor scores that reveal the influence of each factor and the alignment of model preferences with human values ( §2.1 & §2.4). figure 2 illustrates this overall process, which is detailed below: ## uncovering factor-level preference of llms in this section, we analyze how models and humans differ in their factor-level preferences during text generation. using human preference datasets across summarization, instruction-following, and qa tasks, we apply profile to model-generated responses and compare the relative importance of each factor with human judgments. ## conclusion we introduce profile, a framework for granular factor level analysis of llm alignment with human preferences. our analysis using profile reveals that llms tend to over-prioritize factors like output length, misaligning human preferences during generation. however, these models exhibit stronger alignment in evaluation tasks, indicating the potential for leveraging evaluative insights to improve generative alignment. profile facilitates a nuanced understanding of the alignment gaps between human and model preferences. these insights underscore the necessity for more sophisticated, factor-level alignment strategies that can guide the development of llms to better align with human expectations, ultimately fostering more reliable aligned ai systems.