Compare without Despair: Reliable Preference Evaluation with Generation Separability (2024)

Abstract: Human evaluation of generated language through pairwise preference judgments is pervasive. However, under common scenarios, such as when generations from a model pair are very similar, or when stochastic decoding results in large variations in generations, it results in inconsistent preference ratings. We address these challenges by introducing a meta-evaluation measure, separability, which estimates how suitable a test instance is for pairwise preference evaluation. For a candidate test instance, separability samples multiple generations from a pair of models, and measures how distinguishable the two sets of generations are. Our experiments show that instances with high separability values yield more consistent preference ratings from both human- and auto-raters. Further, the distribution of separability allows insights into which test benchmarks are more valuable for comparing models. Finally, we incorporate separability into ELO ratings, accounting for how suitable each test instance might be for reliably ranking LLMs. Overall, separability has implications for consistent, efficient and robust preference evaluation of LLMs with both human- and auto-raters.

## introduction as large language models' (llm's) capabilities have rapidly improved in recent years, evaluation of these capabilities has become increasingly reliant on human preference judgments that compare pairs of model generations. while these judgments offer freedom from gold-standard references @xcite @xcite , they are far from perfect @xcite . in particular, human evaluation faces issues including low rater agreements @xcite , spurious correlations with factors like length @xcite , lack of measurement validity @xcite , and inconsistent usage and interpretation of inter-rater agreement @xcite . furthermore, for annotation efficiency, human judgments are sometimes replaced with llm judgments, which have shown high correlation with crowdworker ratings @xcite @xcite @xcite ; however, it remains unclear whether such auto-evaluations are a step in the right direction or exacerbate existing biases @xcite @xcite . in this work, we focus on the problem of unreliable preference judgments from human raters, illustrated in figure 1 . we show that output pairs from any two modern llms can often be hard to distinguish from each other; such high cross-alignment between models can cause preference judgments to be highly arbitrary. we identify another understudied factor affecting judgments: the variability within one llm's generations for the same input, owing to the stochasticity of popular decoding techniques such as temperature sampling @xcite . such low self-alignment, in addition to high cross-alignment between models, may lead to inconsistent ratingshighly dependent on the exact sampled pair chosen for preference judgments. as a concrete example, on 100 news articles from cnn/dailymail @xcite , our human evaluations show that when comparing five different summary pairs for each input, raters picked the same model only 46% of the time ( §3). these findings raise the question: when can we rely on pairwise judgments to compare generations from two llms? we argue that some test instances might be better suited for human evaluation than others, mirroring insights from prior work @xcite . we introduce separabil-ity, a meta-evaluation measure that determines, for a single instance, how distinguishable two sets the group discusses their upcoming presentations and decides to split into two teams, with the girls and guys working separately. they also joke about gender roles and being gentlemen. ethan mentions that solstafir is playing on november 21st and asks who will be attending. noah expresses interest but is unsure due to work, while leo and ethan encourage him to come. archie jokes that there are better concerts, but the others are still excited to see solstafir. ethan, noah, archie, and leo discuss who will be attending a solstafir concert on november 21st. noah is unsure if he can make it due to work, but the others try to convince him to come. the group discusses attending a concert by the band solstafir on november 21st, with noah expressing interest but also considering the possibility of missing it due to work commitments. leo is insistent that noah attend, and the others agree. the group discusses seeing solstafir in concert, with noah expressing interest and leo confirming he will go. archie and ethan mention better concerts, but noah and leo insist on going. ## separability as a meta-evaluation we address the problem of consistency in modern generative evaluation: namely, how suitable a test input x @xmath0, m a and m b . our approach is based on the key intuition that it is harder to collect consistent preference ratings between m a and m b if their output generations are, on average, harder to distinguish for a (human) rater. for instance, the distinction is hard when the generations focus on similar content (see summaries in figure 1 , left), or have similar styles; we call this high crossalignment between generations from m a and m b . another factor that may make distinguishing two models' generations harder is large variability within each model's sampled generations, due to stochastic llm decoding approaches such as temperature and nucleus sampling. such variability, which we refer to as low self-alignment, makes it hard to characterize each model's specific tendencies, which in turn makes it hard to have a consistent preference for a single model. under the above two conditions, the choice of which generations to use for pairwise comparison influences the preference rating outcome (figure 1 ). both kinds of alignments, while orthogonal, play a key role in determining how consistent human ratings for an instance might be ( §2.1). we introduce separability, a meta-evaluation measure that estimates how suitable a test instance is for preference rating by consolidating cross-and selfalignment ( §2.2). while separability does not determine which generation is better or more preferable, it helps us understand how much we can trust each preference rating for a given input instance. at a very high level, there are four common scenarios which may occur in comparing generations from two models, which we highlight in figure 2 . 1 scenarios 1, 2 and 3 all depict output sets where any sample from model a is expected to be very distant from any sample from model b (i.e. low cross-alignment). it is easy to distinguish the two models under scenarios 1 and 3. in scenario 1, if two generations are very different, they must be from different models. for scenario 3, the difference in self-alignments is a clear distinguishing factor between the two sets of generations. under scenario 2, generations from the same model are also far from each other (i.e. the self-alignment is also low), which makes the overall output sets hard to distinguish. in contrast, scenario 4 depicts a situation where both self-and cross-alignments are high; all generations, regardless of the model they came from, are similar, making it hard for the rater to distinguish the models' output sets. formally, given a set of test inputs {x i } d @xmath1.g. scenario 1 low cross-alignment high selfalignment high separability ⇒ high selfalignment high separability ⇒ scenario 3 low cross-alignment low selfalignment high selfalignment low separability ⇒ low cross-alignment low selfalignment low selfalignment low separability ⇒ scenario 4 high cross-alignment high selfalignment high selfalignment scenario 2 figure 2 : four scenarios illustrating the intuition behind separability. blue and gold circles represent generations from models m a and m b respectively, and euclidean distances represent (dis)similarities between them. for a given input, at least one of the two models needs to have higher similarity among its own generations (high self-alignment) to have high separability for that input. high similarity across generations from different models (high cross-alignment) leads to lower separability. high self-alignment corresponds to low spread of a set of same-colored circles and viceversa. high cross-alignment corresponds to low spread of the entire set of circles and vice-versa. from this distribution, we can sample k generations using a common stochastic decoding algorithm such as temperature sampling, yielding sets {ŷ i,j a } k @xmath2,l b } k @xmath3. ## applying separability to elo as another concrete application of separability, we investigate extending a popular novel method for ranking llms: elo ratings @xcite . in particular, we weight how much a new preference comparison affects a model's elo rating using the separa-bility of the test instance for that comparison. elo ratings have emerged as a popular method of scoring and comparing llms @xcite . originally developed to score and rank chess players, elo ratings model the expected win probability of a model in a pairwise comparison. after observing the outcome of a comparison between two models, both models' ratings are updated. the elo updated rating for a model m a is given by we propose incorporating separability into the elo update in equation 5 by modifying the weight k i for each new comparison based on its separability value. for an update elo ′ a after a comparison on an instance x i with separability δ i a,b , we use the weighting factor: when the input's separability value is at the chosen threshold t , k is not updated. we compute elo and separability-weighted elo (sep-elo; equation 6 ) using data from our 600 human evaluation hits as preference judgments ( §3). to calculate these ratings, we sample one rating for each input from our pool of ratings. we compute confidence intervals with the bootstrap method with 100 trials. figure 9 shows that sep-elo has narrower gaps in model ranking, suggesting that models are more similar under adjustments to consistency of judgments (or, separability). we acknowledge that our results are a pilot due to the limited number of ratings we could use in our computation. however, we expect sep-elo can reveal reliable trends even when applied to larger sets of preference data such as lmsys 12 . further testing with larger sets of ratings can also help users more optimally tune the hyperparameters used in the separability-elo score calculation. however, we note that hyperparameter selection involves a subjective judgment on how much a practitioner wishes to incorporate separability values into the score update. alternative applications of separability beyond sep-elo, separability values could be used for adversarially filtering test sets @xcite . not only would this lead to fine-grained comparisons between models, but could also lead to obtaining cost-and time-efficient human ratings. however, some caution is to be urged since such filtering may lead to biases @xcite , since low separability instances can still contain valuable information. instead, we recommend importance weighting instances by separa-bility when sampling instances for human judgments, in a similar manner as it is used in elo ratings. that is, we recommend that high sepa-rability instances be used for human preference judgment collection. alternatively, a stratified sampling approach from different separability ranges could ensure a more robust preference evaluation scheme. model output variability giulianelli et al. (2023) comprehensively characterize llm vs human output variability, with a focus on comparing it to human output variability. suzgun et al. (2022); bertsch et al. (2023) take advantage of production variability to select more optimal generations using minimum bayes risk (mbr) decoding. in a 12 https://lmsys.org/ similar vein, our work incorporates variability in generations into our meta-evaluation measure. prioritizing test instances rodriguez et al. (2021); vania et al. (2021) evaluate test instances on a variety of dimensions such as difficulty and discriminability (similar to our notion of separabil-ity) using item response theory (irt), albeit in a text classification setting. boubdir et al. (2023a); ashury-tahan et al. (2024) study prioritizing test instances for human evaluation for efficiency-related purposes. however, the approach of boubdir et al. (2023a) relies on access to model logits which are not necessarily available to llm users. moreover, we take a more task-centered approach. ashury-tahan et al. (2024) use embeddings of model outputs and a clustering-based method to select a subset of test instances that are most illustrative of model differences. 6 conclusion we present separability, a meta-evaluation measure that estimates how suitable a test instance is for pairwise preference elicitation. we show that instances with high separability yield more consistent human judgments. we show that the test distribution of separability can be used to analyze how useful a benchmark may be for the comparison of two llms. we show that separability can be incorporated into elo scores. our work shows that separability can help llm developers and users determine and prioritize evaluation instances and benchmarks. future work will look at applying separability in building quality filters for preference tuning data for learning from human feedback. models for multiple-choice tasks are not robust under variation of scoring methods. arxiv preprint arxiv:2403.00998. clara vania, phu mon htut, william huang, dhara mungra, richard yuanzhe pang, jason phang, haokun liu, kyunghyun cho, and samuel r. bowman. 2021. comparing test sets with item response theory. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1141-1158, online. association for computational linguistics. peiyi wang, lei li, liang chen, zefan cai, dawei zhu, binghuai lin, yunbo cao, qi liu, tianyu liu, and zhifang sui. 2023. large language models are not fair evaluators. arxiv preprint arxiv:2305.17926. minghao wu and alham fikri aji. 2023. style over substance: evaluation biases for large language models. arxiv preprint arxiv:2307.03025. zhiyuan zeng, jiatong yu, tianyu gao, yu meng, tanya goyal, and danqi chen. 2023. evaluating large language models at evaluating instruction following. in the twelfth international conference on learning representations. tianyi zhang, varsha kishore, felix wu, kilian q weinberger, and yoav artzi. 2019. bertscore: evaluating text generation with bert. in international conference on learning representations. tianyi zhang, faisal ladhak, esin durmus, percy liang, kathleen mckeown, and tatsunori b hashimoto. 2023. benchmarking large language models for news summarization. arxiv preprint arxiv:2301.13848. lianmin zheng, wei-lin chiang, ying sheng, siyuan zhuang, zhanghao wu, yonghao zhuang, zi lin, zhuohan li, dacheng li, eric xing, et al. 2024. judging llm-as-a-judge with mt-bench and chatbot arena. advances in neural information processing systems, 36.