HEAL : A Hypothesis-Based Preference-Aware Analysis Framework (2025)

Abstract: Preference optimization methods like DPO have achieved remarkable performance in LLM alignment. However, the evaluation for these methods relies on a single response and overlooks other potential outputs, which could also be generated in real-world applications within this hypothetical space. To address this issue, this paper presents a Hypothesis-based PrEference-aware AnaLysis Framework (HEAL), a novel evaluation paradigm that formulates preference alignment as a re-ranking process within hypothesis spaces. The framework incorporates two complementary metrics: ranking accuracy for evaluating ordinal consistency and preference strength correlation for assessing continuous alignment. To facilitate this framework, we develop UniHypoBench, a unified hypothesis benchmark constructed from diverse instruction-response pairs. Through extensive experiments based on HEAL, with a particular focus on the intrinsic mechanisms of preference learning, we demonstrate that current preference learning methods can effectively capture preferences provided by proxy models while simultaneously suppressing negative samples. These findings contribute to preference learning research through two significant avenues. Theoretically, we introduce hypothesis space analysis as an innovative paradigm for understanding preference alignment. Practically, HEAL offers researchers robust diagnostic tools for refining preference optimization methods, while our empirical results identify promising directions for developing more advanced alignment algorithms capable of comprehensive preference capture.

## introduction direct preference optimization (dpo) has emerged as the predominant method for aligning large language models (llms) with human preferences * corresponding author. @xcite . recent research on dpo has also explored various variants, including simpo @xcite , orpo @xcite , and kto @xcite . to evaluate the effectiveness of these preference alignment methods, researchers typically rely on benchmarks such as alpacaeval @xcite and mt-bench @xcite . in the evaluation process using these benchmarks, we typically follow a standard procedure: given a prompt, we first generate a response from the aligned model using a temperature-based sampling method @xcite . next, we employ a proxy model (such as to compare this response with the reference response, evaluating whether the model is effectively aligned. however, this procedure faces a fundamental limitation because this sampling-based evaluation approach only examines single responses sampled from target llms. but in practice, the llms are commonly expected to generate a wide spectrum of diverse responses, which cannot be sufficiently covered by several sampled responses. this misalignment between evaluation and real-world llm development prohibits researchers and developers from assessing the llms' performance accurately. furthermore, this paradigm also neglects the relative comparison of responses, which is fundamentally modeled in preference learning scenarios. to address these limitations in the evaluation, we propose heal (hypothesis-based preferenceaware analysis), a novel framework that evaluates llms through the lens of ranked hypothesis spaces. inspired by the ranking-based evaluation approaches such as rewardbench @xcite , heal conceptualized preference alignment as a dynamic re-ranking process within the hypothesis space, enabling comprehensive assessment through two complementary quantitative methods: we evaluate mainstream preference learning methods using heal to address three key research questions. (rq1): do these methods effectively capture preference information? while ranking accuracy confirms that llms acquire preferences through optimization, results reveal incomplete learning. to elaborate, current methods struggle to absorb all preference signals fully. (rq2): can llms discern proxy model-specific preferences? experiments demonstrate that llms successfully learn distinct preference patterns from different proxy models, showcasing heal's sensitivity to subtle inter-model variations. (rq3): how do learned preferences vary across methods? all tested methods achieve strong in-distribution alignment with proxy models, but out-of-distribution performance degrades significantly, except for simpo, which exhibits notable generalization. these results validate the partial efficacy of preference alignment while underscoring critical limitations, particularly in robustness and completeness of learned preferences. our main contributions are: ## heal: a hypothesis-based analysis framework we propose heal (hypothesis-based analysis), illustrated in figure 1 . the framework models preference patterns as ranked hypothesis spaces and evaluates them through two complementary metrics: (1) ranking accuracy for ordinal consistency and (2) preference strength correlation for continuous alignment, which we detail in this section respectively. ## conclusion this paper presented heal, a novel, hypothesisbased framework designed to move beyond traditional evaluation of preference learning. by employing two complementary metrics, heal provides a more holistic assessment of model alignment. our evaluation using heal yields three key observations in current models, revealing suboptimal ranking performance, an inability to capture preference nuances, and unique, model-specific learning behaviors. ultimately, heal offers the community a more powerful and nuanced tool for diagnosing and improving preference learning, paving the way for the development of more reliably aligned models. name commercial open-source