DCRM : A Heuristic to Measure Response Pair Quality in Preference Optimization (2025)

Abstract: Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. In this work, our observation is that the differences between the preferred response y^+ and dispreferred response y^- influence what LLMs can learn, which may not match the desirable differences to learn. Therefore, we use distance and reward margin to quantify these differences, and combine them to get Distance Calibrated Reward Margin (DCRM), a metric that measures the quality of a response pair for PO. Intuitively, DCRM encourages minimal noisy differences and maximal desired differences. With this, we study three types of commonly used preference datasets, classified along two axes: the source of the responses and the preference labeling function. We establish a general correlation between higher DCRM of the training set and better learning outcome. Inspired by this, we propose a best-of- N^2 pairing method that selects response pairs with the highest DCRM. Empirically, in various settings, our method produces training datasets that can further improve models’ performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.

## introduction preference optimization (po) methods such as dpo @xcite have shown success in improving llms' performance in various tasks @xcite . these methods usually involve a contrastive learning objective that encourages llms to generate a preferred response y + with higher probability and a dispreferred response y -with lower probability, given a query x. prior research @xcite has shown the importance of selecting suitable response pairs for po training. in particular, the contrastive training signals sent to llms are partly derived from the differences between y + and 1 our code is at https://github.com/hcy123902/dcrm . figure 1 : top: ideal response pairs should have fewer noisy differences (small distances) and more desired differences (large reward margins). dcrm measures response pair quality with this intuition; bottom: common preference datasets (ss-rm, ds-rm, ds-fix; see § 2.2) have varying locations in the distance-reward margin landscape, but none achieves an ideal combination. y -. however, the latter often includes a mix of desirable differences we want the model to learn (e.g., y + is more helpful than y -in factoid question answering) as well as noisy differences. for instance, y + and y -can differ in features that are irrelevant to quality or correctness, e.g. different writing styles for factoid question answering. ideally, we want a preference optimization dataset that minimizes the fraction of these noisy signals in the overall dataset, thereby prioritizing useful signals (see figure 1 ). although prior research @xcite has investigated the correlation between certain proxies of "differences" (e.g., edit distance) and po learning outcome, it does not distinguish noisy and desired differences, and therefore cannot accurately model the relationship. in this paper, we develop a metric called distance calibrated reward margin (dcrm) to measure the density of desired differences among the overall differences in a preference optimization dataset. dcrm computes the the ratio between the reward margin, which we use as a proxy for desired differences, and two distance metrics (edit distance, probability difference), which we use as proxies for the total difference. we find that dcrm scores effectively capture the quality of a preference optimization dataset; language models trained on datasets with higher scores report in better learning outcomes after training. concretely, we study dcrm's impact on quality for three types of preference dataset construction strategies, categorized based on the (1) source of the positive (y + ) and negative (y -) pairs, and the (2) preference labeling scheme (reward model v/s heuristics). combined, these cover the common preference dataset curation techniques used in prior works @xcite @xcite . for all strategies, we use ultrafeedback @xcite as the seed to construct the datasets. we train three base models (llama-2-7b-chat, llama-3.2-1b-instruct, gemma-2b-it) on these datasets and use alpacaeval @xcite , mt-bench @xcite , and arena-hard @xcite for evaluation. for all models and dataset combinations, we find that different dataset construction strategies result in datasets with very different dcrm scores. interestingly, we find that higher dcrm scores of training datasets correlate with better training outcomes for the trained models. we operationalize this correlation and propose a strategy called best of n foot_0 that curates preference optimization training data that maximizes the dataset-wide dcrm score. our strategy allows us to flexibly construct better quality preference datasets for any given model and input questions combination. our results show that llms trained on datasets that maximize dcrm leads to a substantial boost in performance compared to the original datasets. our contribution is summarized as follows. ## comparing different types of preference datasets in this section, we compare models that are trained on different types of preference datasets, and establish a correlation between the dataset-level dcrm value and downstream performances. we report the results in ## operationalizing dcrm in § 4, we observe that higher dcrm is correlated with better training outcomes. can we use this correlation to guide training dataset selection? approach an answer is to sample responses from π ref . however, this can be expensive with a large model or dataset. instead, we want to investigate how to select the best response pair from an existing pool of responses, formally, given n responses {y 1 , • • • , y n } (and also {y n +1 , • • • , y 2n } from a second model in the ds setting), we propose to select the pair (y i , y j ) with the highest dcrm. we denote this as best of n 2 pairing (bon 2 ), since we select the best pair from n × n candidates. our method is different from the conventional method (used in ss-rm), which chooses the pair with the highest reward margin by setting y + and y -to the response with the highest and lowest reward scores. ## qualitative analysis (feature-analysis) we analyze the feature differences between y + and y -. we define relevant features (correctness, helpfulness, etc.) as those that the llms should learn, and irrelevant features (writing style, sarcasm, tone, etc.) as those not targeted by the task. features to align with the reward signals, we use the 11 features (de-duplicated) from the armorm reward model as the relevant features. these include helpfulness, truthfulness, etc. we manually define 21 irrelevant features that are roughly orthogonal to these relevant features (see the full lists in appendix c.1). the useful training signals come from differences between y + and y -that are along relevant features and are pointing in the correct direction (y + is better than y -for a relevant feature), which we call desired feature differences. metrics we define f ∆ as the number of features along which y + and y -differ. to measure the fraction of desired feature differences, we define f des ∆ as the fraction of features in f ∆ that are (a) relevant and (b) contrasted in the correct direction (i.e. y + is "better" than y -for that feature). fraction of features that only satisfy condition (a) is denoted by f rel ∆ . similar to dcrm, f des ∆ indicates the ratio of useful contrast signals among noisy signals. to compute these, we prompt gpt-4o-mini-0718 to (1) identify the three most prominent features that differ between the two responses (setting f ∆ =3) and ( 2 ) indicate a contrast direction for each feature if applicable (i.e., whether y + is better). referring to the list of relevant features, we can then compute f rel ∆ and f des ∆ . note that we can use this to study the training dataset (i.e. y + -y -), and the learned differences after training (y trained -y ref ). ## related work preference optimization preference optimization is an alternative to traditional rlhf methods @xcite such as ppo @xcite . it avoids the need for an explicit reward model. popular po algorithms includes dpo @xcite , @xcite , kto @xcite , r-dpo @xcite , simpo @xcite , cpo @xcite , orpo @xcite , and so on. many papers report performance increases on alpacaeval when training llms using po methods on chat datasets @xcite . ## conclusion we propose a metric called dcrm that measures the density of useful training signals in response pairs and show its correlation with the po training outcome. inspired by this correlation, we design a best of n 2 pairing method, which can curate highquality datasets to train llms with po effectively. in addition, we provide a feature analysis to inspect the characteristics of various common datasets with varying dcrm values.