Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation (2024)

Abstract: Generative AI systems have become ubiquitous for all kinds of modalities, which makes the issue of the evaluation of such models more pressing. One popular approach is preference ratings, where the generated outputs of different systems are shown to evaluators who choose their preferences. In recent years the field shifted towards the development of automated (trained) metrics to assess generated outputs, which can be used to create preference ratings automatically. In this work, we investigate the evaluation of the metrics themselves, which currently rely on measuring the correlation to human judgments or computing sign accuracy scores. These measures only assess how well the metric agrees with the human ratings. However, our research shows that this does not tell the whole story. Most metrics exhibit a disagreement with human system assessments which is often skewed in favor of particular text generation systems, exposing a degree of favoritism in automated metrics. This paper introduces a formal definition of favoritism in preference metrics, and derives the Favi-Score, which measures this phenomenon. In particular we show that favoritism is strongly related to errors in final system rankings. Thus, we propose that preference-based metrics ought to be evaluated on both sign accuracy scores and favoritism.

## introduction with the rise of generative ai for text, images, code, music, and other modalities, the question of how to evaluate the quality of the outputs of these systems is becoming more and more important. preference ratings have become a ubiquitous strategy for this task @xcite @xcite @xcite . the general idea is that the outputs of two generative systems are shown to a (typically human) rater who decides which output they prefer, or whether both outputs are of equal quality. the ratings are then aggregated over a test set to compute in how many cases the outputs of one system were preferred over the other and whether the difference is significant. since human-based evaluations are expensive, there has been a growing movement towards training automated metrics to assess generated content. @xcite @xcite . so instead of showing the outputs of two generative systems to human raters, one uses an automatic metric and hopes that the evaluation will resemble the ratings of the humans. 1 this raises the question of how these automated metrics themselves ought to be evaluated. currently, the most popular method is measuring the correlation of the metrics' ratings to human judgments, or in the case of preference ratings, sign-accuracy on both the sample and system level is computed (i.e., on how many preference ratings do human and metrics agree, or on how many system-pairs do human and metrics agree) @xcite . for some tasks, such as machine translation, we have seen that according to these, metrics are steadily improving @xcite . while both measures are important in the assessment of automated metrics, they only count how many mistakes the metric typically makes, ignoring how these mistakes are distributed. that is, whether the mistakes are systematically skewed in favor of one generative system under investigation or another. to underscore the significance of this information, let us consider the assessment of two generative systems, denoted as π a and π b , using a metric that exhibits only a 10% error rate, which are all skewed towards π a . this means that in 10% of cases, the metric's ratings will deviate from human judgments and in all cases the deviation is in favor π a . furthermore, assume that according to humans π a is slightly worse than π b , then the favoritism of the metric will cause π a to be rated as being better than π b according to the metric. thus, the research question that we tackle in this work is how we can measure favoritism, and investigate its impact on evaluation. contributions. this work has three main contributions: ## impact of favoritism on system rankings to demonstrate the need for the favi-score, we use the data of the wmt-22 metrics shared task @xcite . this contains ratings for 9 machine translation systems. based on this data, we produce one ranking using human ratings and one ranking using comet-22 @xcite , a wellestablished automated metric for machine translation outputs. we visualize their assessments using directed acyclic graphs (dags), as depicted in figure 1. for this, the preference ratings are computed for each pair of systems using the human ratings and the corresponding comet-22 ratings. then, we put an edge between two systems if one system is statistically significantly preferred to the other system according to a sign test @xcite . for better visibility, we omit transitive edges. note that the system-level agreement lies at 61.1%, 2 note that the formal definition of the favi-score could also be applied to other modalities. thus, being agnostic to the task/domain at hand (e.g., music, images, or code generation). 3 we provide an implementation of the favi-score: https: //github.com/vodezhaw/faviscore the ranking of a set of systems is depicted as a directed acyclic graph, where an edge from system a to system b states that system a "wins" against system b. here, a win is determined by a sign test @xcite at a 95% confidence threshold. note that only considering the dag does not reveal the whole story. for example, if a metric unduly favors a system that humans also rank highly, the overall ranking might not change. yet, the favored system could appear to win by a much larger margin than it deserves, exaggerating its performance. for a concrete example, if we consider the outcome percentages between online-w and jdexploreacademy, then under human evaluation, online-w is rated better in 24.1% of cases and worse in 18.6% of cases (giving a margin of 5.5%). under comet-22, it is rated better in 57.5% of cases and worse in 42.2% (giving a margin of 15.3%). even though online-w is rated better than jdexploreacademy in both cases, the perceived difference is much more pronounced under comet-22, which is unfaithful to the human evaluation even though there is sign agreement. a concrete consequence is that, for example, a sign test will report much higher significance (lower p-value) for evaluation under comet-22 than for human evaluation. this example shows the need for a mechanism that measures the inherent favoritism of a metric. informally, the favi score is a relative measure between the human ratings and the metric ratings. instead of measuring how much the humans and metrics agree on their preference, it is a function of those samples where the humans and metrics disagree. the main idea is to count and weigh the cases in which the metric disproportionately assigns more samples in favor of one system than the human judges. thus, the favi score yields a more fine-grained view compared to the dag, as it also can measure the cases where there are no changes to the dag, but one system is disproportionately favored (or disfavored) by the metric (see examples in appendix d). the favi score can be used as a diagnostic tool to discover those cases. in the subsequent sections, we will formally introduce the favi score. ## related work we are not aware of any prior work that defines or analyses favoritism in preference ratings. most work on the analysis of metrics for text generation tasks is concerned with the overall performance, which is measured in terms of correlation to human judgments or sign accuracy @xcite @xcite . the main conclusion of these analyses is that automated metrics do not yet match human ratings, and that the performance of the metrics vary depending on the domain and task. moreover, there is a growing body of work that analyses individual metrics for generative tasks and their problems @xcite @xcite , but these studies do not compare or assess different metrics. @xcite go a step further and analyse specific weaknesses of various metrics for datato-text. they showed that the metrics had difficulties distinguishing medium from good-quality outputs. similarly, @xcite showed that in the case of automated text summarization, automated metrics behave differently depending on whether the systems are extractive or abstractive. thus, there is evidence in the literature that automated metrics are biased toward certain types of outputs and have difficulties with certain features. however, we are not aware of a fundamental analysis and discussion on the topic of structurally preferring the outputs of one system over others. a different line of work is concerned with adversarial attacks on metrics. @xcite showed that adem @xcite , a trained metric, is susceptible to specific changes in the inputs. @xcite devised a method based on reinforcement learning to find nonsensical adversarial samples that elicit perfect scores from metrics for conversational dialogue systems. these findings indicate a lack of robustness and raise the question why certain outputs are preferred. finally, the last type of work related to ours is concerned with a more fundamental analysis of metrics. @xcite apply the bias-variancenoise decomposition from @xcite to assess statistical bias of automated metrics used in the evaluation of machine translation and summarization. their analysis produces a ranking of automated metrics based on their statistical bias, but it does not indicate the direction of the bias. also, their approach does not investigate the relation between the bias strength and the systems under scrutiny. while they find that automated metrics for natural language generation (nlg) are biased, we extend this notion and show that the strengths of the biases vary on the system level and affects system rankings. @xcite analyse the bias and variance of automated metrics. they find that automated metrics correlate poorly with human judgments, and that there is a correlation bias, i.e. the metrics often agree with humans in judging bad outputs but correlate poorly with humans on average or good quality outputs. to summarize, there is strong evidence that automated metrics exhibit behavior which can benefit or harm certain systems. to our knowledge, our work provides a definition and a measure for quantifying the amount of this benefit or harm for the first time. ## preference ratings in this section, we formalize the notion of a preference rating and define the confusion matrix which shows the amount of disagreement of two preference rating schemes (e.g. human judgments vs. an automated metric. we first start by formally defining a generative system (gs) as a function from an input space to an output space. definition 1 (generative system (gs)). let i denote the set of all possible inputs to a gs, and o the set of all possible outputs. then, a gs is a function π that takes an input and generates an output note that the input and output spaces are defined quite abstractly and are task-specific. for instance, in machine translation, the input space might be the set of utterances in the source language and the output space the set of utterances in the target language, whereas in news summarization, the input space would be the set of all news articles and the outputs is the set of all texts. next, we define the preference rating as a function of a triple of one input and two outputs to a sign. definition 2 (preference rating). let i denote the set of all possible inputs to a gs, and o the set of all possible outputs. then, we define a preference rating as: r : i × o × @xmath0, =, -} thus, for two generative systems π 1 and π 2 , r[i] = r(i, π 1 (i), π 2 (i)) is the preference rating that compares the outputs of system π 1 and system π 2 for the input @xmath1. in the following, we denote a human-based preference rating as r h [i] = r h (i, π 1 (i), π 2 (i)) and an automated metric-based preference rating as r a [i] = r a (i, π 1 (i), π 2 (i)), respectively. in order to run an evaluation and apply a preference rating r to decide whether π 1 performs better than π 2 , we first define the dataset that defines the evaluation setting. definition 3 (evaluation setting). given a test-set t ⊆ i, a pair of text generation systems π 1 and π 2 , and two preference ratings r h and r a , we define the evaluation setting as: thus, the evaluation setting e is composed of the inputs of the test set, the outputs of both tg systems, and the two preference ratings (e.g. according to humans and according to an automated metric). for a given evaluation setting e, we can compute the confusion matrix between r a and r h . ## measuring favoritism in this section, we define favoritism and derive a score to quantify it. our view of favoritism is fundamentally relative, since it is based on comparing the behaviour of a metric with respect to the ground-truth provided by humans. in theory, one could compute favoritism compared to some other set of ground-truth labels (e.g., comparing two sets of human ratings). however, to keep our discussion simple, we treat human ratings as the unbiased ground truth, and leave deviations of this assumption to future work. our measure of favoritism, called favi-score, is based on two observations about errors of an automated rating with respect to the human rating. first, every error is necessarily in favor of a specific system. for example, changing a + to a =, is favoring π 2 . second, not every error has the same severity. switching a preference from one system to another (i.e., + to a -) is worse than classifying a draw as a preference (i.e., = to a -), since the former has a larger impact on the outcome margin than the latter. this means that the favoritism measure should depend both on the proportion of errors in favor of one system and the severity of those errors. to better illustrate the interplay of these criteria, let us consider the confusion matrices depicted in figure 2 for an outcome of @xmath2, 100, 100) alongside the corresponding outcome according to the metric d. we can see the error counts in the offdiagonal entries of c i , where the lower triangular sub-matrix, indicated in magenta and red, corresponds to errors in favor of π 1 and the upper triangular sub-matrix, indicated in cyan and blue, corresponds to errors in favor of π 2 . let us first compare c 1 and c 2 . in both cases there are 10 errors that are all in favor of π 1 . nevertheless, c 1 shows more favoritism since its errors are more severe, favoring π 1 more strongly. the consequence is visible in the outcome margin, where mar( d1 ) = 110-90 = 20 and mar( d2 ) = 100 -90 = 10. c 3 has more total errors, namely 20, but they are equally distributed in favor of each system and of equal severity. therefore, c 3 does not favor either system, which is also visible in the outcome margin mar( d3 ) = 0. finally, c 4 also has 20 errors distributed equally in favor of either system. but since the errors in favor of π 1 are more severe (i.e., mistaking + and -) than the ones in favor of π 2 (i.e., mistaking + @xmath3, c 4 favors π 1 overall. thus, favoritism depends on how the errors are distributed, and how the severity of the mistakes are counted. the severity can be quantified in terms of how much they change the outcome according to the metric. mistaking + for leads to a change of 2 in the outcome margin, whereas mistaking + @xmath4. thus, we weigh these mistakes accordingly and define the error cost: definition 8 (directed error cost). let the following matrix be the error cost matrix: thus the cost of an error reflects its impact on the outcome margin, and the errors in favor of π 1 have a positive weight and errors in favor of π 2 have a negative weight. in cases where @xmath5. based on these observations, we will define a measure of favoritism, called favi-score φ, that corresponds to the expected weight of the errors of an automated rating with respect to the human rating. definition 9 (favi-score). given an evaluation setting e and its confusion matrix c the favi-score is the final score ranges from -2 to 2, where positive values indicate favoritism of π 1 and negative values indicate favoritism of π 2 . the term cmn e corresponds to the probability of a given type of error in cases where m ̸ = n, meaning thus, this brings us to the following interpretation of the favi-score: the favi-score is the expected error cost. there is an equivalent definition of the favi-score based on the outcome margins. lemma 1. the following equality holds: thus, the favi-score can also be interpreted as the average change in outcome margin per error. 4 coming back to our examples, the favi-scores for the confusion matrices in figure 2 are therefore: φ(c 1 ) = 2, φ(c 2 ) = 1, φ(c 3 ) = 0, and φ(c 4 ) = 1 2 . in figure 3 we show a more complex example. in this case more errors are in favor of π 2 (280 vs. 200), but more of the severe errors are in favor of π 1 (90 vs. 60). the overall favi-score φ(c 5 ) = -0.104 shows slight favor for π 2 . the score can be interpreted as each error contributing around 0.104 increase in margin in favor of π 2 . ## favi-score vs. sign accuracy in this section we explore the relation of the favi-score to sign accuracy. we show that the favi-score and sign accuracies measure different properties of a metric, and that the scores are complementary to each other. first, we show the relationship between the favi-score and the system level sign accuracy. for this, 4 the proof is in appendix a. ## application of the favi score for the empirical application, we require data in the form highlighted by the evaluation setting that is where we have a set of text generation systems with paired ratings from humans and automated metrics. for this, we use the data collected by @xcite , which consists of three different tasks (machine translation, summarization, and dialogue systems). we extended their dataset with data from the wmt-22 metrics shared task (see appendix c for details). here we provide an overview of the data (see ## conclusion we motivated and defined the favi-score, a simpleto-implement and easy to interpret score, which measures how much an automated preference metric favors the outputs of a generative ai system. our investigations show that the favi-score differs from the sign accuracy scores, and is to be used complementary to them. the empirical application of the favi-score to real-world setting reveals that all metrics exhibit some degree of favoritism, which needs to be accounted for when applying these metrics, to better interpret results. we envision that the favi-score is used as a diagnostic tool to understand which systems are favored by a metric.