Correction of Errors in Preference Ratings from Automated Metrics for Text Generation (2023)

Abstract: A major challenge in the field of Text Generation is evaluation: Human evaluations are cost-intensive, and automated metrics often display considerable disagreements with human judgments. In this paper, we propose to apply automated metrics for Text Generation in a preference-based evaluation protocol. The protocol features a statistical model that incorporates various levels of uncertainty to account for the error-proneness of the metrics. We show that existing metrics are generally over-confident in assigning significant differences between systems. As a remedy, the model allows to combine human ratings with automated ratings. We show that it can reduce the required amounts of human ratings to arrive at robust and statistically significant results by more than 50%, while yielding the same evaluation outcome as the pure human evaluation in 95% of cases. We showcase the benefits of the evaluation protocol for three text generation tasks: dialogue systems, machine translation, and text summarization.

## introduction the field of text generation (tg) has witnessed substantial improvements over the past years. the gain in performance is mainly due to the application of large-scale pre-trained language models @xcite based on the transformer architecture @xcite , which allows fast processing of large amounts of data. this has spawned myriads of new systems for tg. the most prominent example is gpt-3 @xcite , which showcases impressive performance on a variety of tasks in a zero-shot learning regime. one major hurdle for further progress is the evaluation of tg systems. currently, the most reliable * these authors contributed equally. approach to evaluating tg systems is a humanbased evaluation @xcite , which is time-consuming and cost-intensive. furthermore, human evaluation suffers from a set of problems such as low annotator agreements @xcite , and they need to be designed with care to be reproducible @xcite . these problems motivated the development of automated evaluation metrics, which take the input of a tg system and the generated text (and potentially one or multiple reference texts) as their input, and return a rating. generally, there are two types of automated metrics: trained and untrained metrics @xcite . the most prominent untrained metrics are the bleu score @xcite and the rouge score @xcite , developed for the evaluation of machine translation and automated summarization systems, respectively. the more recent metrics that were proposed are trained metrics. one of the first such approaches is the paradise framework by @xcite for task oriented dialogue systems, which learns to match interaction statistics to user satisfaction scores. current approaches are based on large pre-trained language models. for conversational dialogue systems, there are adem @xcite , usr @xcite , fed @xcite or maude @xcite , among others (for a more complete overview, we refer the reader to @xcite ; @xcite ). for machine translation, the most prominent trained metrics are comet @xcite and @xcite . for a more in-depth treatment of different automated metrics for tg systems, we refer the reader to @xcite . some metrics already achieve correlations with human judgements of 50% and above @xcite @xcite . for this reason, it is a tempting to use automated metrics to rate and rank tg systems. a typical approach to compare two systems is to use preference ratings, where the generated output of two systems for the same input is given, and the metric is used to decide which output is preferred or if they are of similar quality @xcite . such preferences are then aggregated for several sample inputs to decide which system is "better". one important open question, which we will tackle in this paper, is how erroneous ratings from an automated metric on the sample level influences the system level evaluation. motivating example. assume that we are given a set of tg systems, and the goal is to rank them according to some criterion (e.g. relevance of generated summaries for some text summarization systems). assume that we are given an automatic preference metric. the naive application of this metric to determine which of two tg systems is better is to apply the metric to the outputs of the two systems for a test set of a fixed size. then one would apply a statistical significance test @xcite to determine if one system is preferred significantly more often than the other by the metric. this process is repeated for each pair of systems, and then a partial ordering can be derived from the pairwise decisions. to compare the outcome of the automated evaluation, the same procedure is repeated with a human evaluation. a good metric is one that recreates the same system level preference ranking or the same pairwise results as a human evaluation would generate. in this setting, there are four types of outcomes with respect to a human evaluation at the system level: figure 1 highlights the error-proneness of the analysed automated metrics. the main findings are: contributions. this paper has two main contributions: first, we propose a novel bayesian statistical model of tg evaluation which integrates the various sources of uncertainty mentioned above. the model yields a more robust evaluation and has the flexibility to combine human and automated evaluations. the model can be used to determine whether two systems are significantly different or if they are of equal quality. the second contribution is an evaluation protocol that leverages the statistical model and reduces the amount of human ratings required. we investigate the performance of the evaluation protocol in a case-study in three different tg tasks: chatbots, text summarization, and machine translation. our case-study shows that using our contributions, we can almost completely correct for the errors emerging in the naive application of the metrics and that the amount of human ratings bl2_3b bl400distill dialogptlarge polyencoder seekerdial3b human bl2_3b bl400distill dialogptlarge polyencoder seekerdial3b human (a) chatbot-deb facebook-ai huaweitsc nemo online-w uedin volctrans-at volctrans-glat etranslation ref-a ref-b ref-d facebook-ai huaweitsc nemo online-w uedin volctrans-at volctrans-glat etranslation ref-a ref-b ref-d (b) wmt21-comet m0 m1 m10 m11 m12 m13 m14 m15 m17 m2 m20 m22 m23 m5 m8 m9 m0 m1 m10 m11 m12 m13 m14 m15 m17 m2 m20 m22 m23 m5 m8 m9 (c) summeval coh. rouge m0 m1 m10 m11 m12 m13 m14 m15 m17 m2 m20 m22 m23 m5 m8 m9 m0 m1 m10 m11 m12 m13 m14 m15 m17 m2 m20 m22 m23 m5 m8 m9 (d) summeval cons. bertscore needed to produce robust evaluation outcomes is reduced by more than 50%. ## definition of preference metrics in this section, we formally define preference metrics, their errors, and how to mitigate them. we then use this formalism to derive an effective evaluation protocol that can handle error-prone metrics. for the remainder of the paper, we define i as the set of all possible inputs (e.g., for machine translation all sentences in the source language), and o as the set of all possible outputs (e.g., all sentences in the target language). we start by defining a tg system as a function that takes an input and generates an output π : @xmath0. on an abstract level, we can define a preference metric as a function that takes as an input a triple consisting of: the input to a tg system (e.g., a sentence in the source language to be translated), the output of a system a, and the output of a system b (e.g., the translated sentences in the target language), and returns the preference rating. this is formalized as follows: definition 1 (preference metrics). we call functions of the form m : i × o × @xmath1, =, <} preference metrics. we call an outcome of " > " a win, " = " a draw, and " < " a loss. note that the semantics of m(i, o 1 , o 2 ) is to find out whether output o 1 is preferred over o 2 . at this point, the notion of an output being preferred is to be taken abstractly, in a real-world application this would be realized by a concrete feature (better fluency, higher relevance, etc.). next, we introduce an oracle, which constitutes the ground-truth. when constructing an oracle in a real-world application, we would usually resort to human annotations. definition 2 (preference oracle). the preference oracle is a function ω : i × o × @xmath2, =, <} such that now we define the notion of an error-prone metric. definition 3 (error-prone metric). a preference metric with independent confusion errors m is an error-prone metric where the probability of a given outcome is only dependent on the comparison oracle rating foot_1 . its confusion probabilities are defined as: the errors made by an error-prone preference metric can be represented by a confusion matrix with normalized columns, such that each entry in the matrix is a probability. the matrix µ is called the mixture matrix of m: note that the mixture matrix of an error-free metric is the identity matrix. ## statistical model for preference metrics in this section, we introduce the statistical model that is used to compare two tg systems π a and π b . the model encompasses three main sources of uncertainty. input :πa, π b input :set of metric judgments m input :set of human judgments a output :return preference rating 1 begin 2 n cc ′ ← countconfusions (a, m , c, c') we build up the statistical model step-by-step by discussing each source of uncertainty. we apply the bayesian approach which allows us to describe the process in terms of probability distributions that can be sampled by using markov chain monte carlo (mcmc) sampling (refer to appendix a and b for additional details). for the rest of the chapter, assume that we have access to a set of inputs {i 1 , . . . , i n } ⊆ i, and the corresponding system outputs of π a and π b , i.e., o a @xmath3, and o b @xmath4. ## evaluation protocol in this section, we present an evaluation protocol combining human and automated metric ratings assuming a limited budget for human annotations. more formally, given a set of tg systems π 1 , ..., π s , we want to create a partial order, where π @xmath5. the evaluation protocol is depicted in algorithm 2. the protocol leverages the statistical framework to reduce the amount of human annotations needed by leveraging the metric judgments. this works as follows: we are given a set of inputs i (which corresponds to a test set), 3 the posterior in equation 3 can be sampled directly. ## case studies -setup in this section, we present three case studies where we apply the evaluation protocol outlined in algorithm 2. as showcases, we use three domains: the wmt 21 metrics task data @xcite for machine translation, the summeval data @xcite for summarization, and data collected for conversational dialogue systems (see appendix c). ## case studies -results in this section, we discuss the results of the case study. we use the error-measures that we presented in list 1 in the introduction. furthermore, for each system pair we compute the kullback-leibler divergence (kld) between the mode of the posterior in equation 3 based on all human annotations p hum and the mean estimated by running algorithm 2 p prot . we then report the average over all pairs of systems: note that in tables 2 and 3 , we only report the relevance part of the summeval data due to space limitations. the results for the other features are in appendix d. the naive application of metrics yields many errors. when applying the metrics naively, i.e. by simply checking whether @xmath6, then this introduces many cases where systems that are not significantly different are rated as such. in the evaluation protocol is able to recreate the original results. since the ratings are based on our decision function, which takes into account different sources of uncertainty, the insertions are not necessarily wrong. in fact one reason to use automated evaluation is to find differences between system that would be too expensive to discover with human annotations. a different view for comparing the outcomes of the evaluations is given by the kld score, which reports how close the distribution p prot is to the original human evaluation. this view removes the significance test from the equation, and better showcases the disagreement between the protocol and the original human evaluation. in all cases the kld scores are very low, which shows that the protocol yields results comparable to the original human evaluation. in terms of the number of annotations needed, there are two measures. first, we compare the number of annotations needed by the protocol to the one needed by the full human evaluation. here, the application of our protocol reduces the amount of humans annotation by more than half in most cases. for the wmt21 we can even reduce it by two thirds. the second view is comparing the number of annotations needed by the protocol to the annotations needed when the protocol is applied to human ratings only. for the chatbot and summeval domain, leveraging automated metrics results in less data needed (up to 10% for the chatbot domain, and 5% for the summeval domain). for the wmt21 domain, only 1% difference is measured. we assume that this are due to the fact that the metrics are not yet of high enough quality to yield the boost needed to have a large impact. summary. figure 2 summarizes the main outcomes of this work. the figure shows the number of annotations (x-axis) in relation to the negative log-kld score achieved (y-axis). the full human evaluation is set as reference, that is, using 100% of annotations, and a kld score of 0 (thus, not shown). the figure shows that on average using the full protocol on real-world metrics yields using 40% of annotations, and achieving a kld score of 0.08. on the other hand, not using metric ratings in the protocol needs 43% of annotations and achieving a worse kld score of 0.6. the naive application of the metrics does not need any annotations but yields high kld scores (1.6 on average). to showcase an upper limit, we also added the kld divergence for an ideal metric, which we simulated using the bayesian model, where we use a fixed µ (see appendix e for details). the ideal metric only needs 38% of annotation, and achieves a kld score of 0.02. an ideal metric would also achieve a low kld when applied naively. ## related work we here focus on approaches that discuss theorydriven analysis of metrics-based evaluation of tg systems that involve human annotations. @xcite propose an approach to combine human and metrics-based evaluation using control variates to reduce biases in automated metrics and save annotation costs. they explore automated scalar metrics in the summarization and qa domain and find that their approach can lead to marginal reductions of the required human annotations. they conclude that further improvement of automated metrics and exploration of rankingbased evaluation are potential future directions. one interesting take-away from this work is the influence of the quality of the human annotations. currently, we approximate the oracle preference ratings through human annotations without explicitly modeling uncertainty stemming from annotator disagreement. @xcite pick up this point and apply a statistical approach to identify a setting in which automated metrics for machine translation and summarization are more precise than human annotations: when the qualitative difference is small and there are only few human annotations. they argue that the reason is that while human annotations are unbiased, they have a high variance. conversely, automatic metrics tend to have a high bias but low variance. furthermore, they apply the bias-variance-noise decomposition from @xcite to analyse sources of errors in evaluation and asses bias levels in automated metrics. our analysis, in comparison, is more fine-grained in terms of categorizing metric errors, and we propose how to combine human and metric evaluation under a budget constraint. similar to this work, von @xcite propose a model that captures uncertainties stemming from imperfect metrics and insufficiently sized test sets. with their framework, the required size of a test set that is needed to distinguish a given difference in performance between two systems with a given automated metric can be calculated. their investigation is limited to the case when scalar metrics are converted to binary metrics, however. @xcite also analyse the required data set sizes that enable the detection of significant differences between systems, but they do not account for metric errors explicitly. @xcite propose an evaluation approach that combines human and automated evaluation in a statistical framework to estimate diversity and quality of nlg systems jointly. their focus is the creation of a novel metric, while our goal is to evaluate existing ones and to combine them with human annotations to obtain robust evaluations. ## conclusion in this work, we introduced a novel bayesian model that explicitly handles errors from automated metrics at the sample level. we then proposed an evaluation protocol that leverages this statistical model to reduce the amount of human annotations needed while yielding similar evaluation outcomes. we applied the protocol to three tasks in a case study. namely, dialogue systems, summarization, and machine translation. the results show that the bayesian model is able to successfully include various types of uncertainty, which leads to more trustworthy applications of automated metrics. when applying the protocol, we achieve similar results as a purely human evaluation with only half the annotations needed. and internal funding by the zurich university of applied sciences.