On the Effectiveness of Automated Metrics for Text Generation Systems (2022)

Abstract: A major challenge in the field of Text Generation is evaluation, because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.

## introduction the field of text generation is a subfield of natural language processing @xcite . we define text generation tasks as those where many different texts may constitute an optimal solution to a given problem. examples are automated summarization, machine translation, dialogue systems, paraphrasing, caption generation, or natural language generation. one unsolved issue in the field of text generation is the evaluation, be it human or automated evaluation. human evaluation is more reliable but more cost and time intensive, and automated evaluation is erroneous but performed in a fraction of time and cost @xcite @xcite . one of the main issues is the lack of theoretically founded guidelines when running an evaluation. for instance, how many samples are needed to be able to significantly distinguish the performance of two systems? or how do we handle the errors made by automated metrics? under which circumstances is it still possible to run an evaluation campaign that yields significant results? in this work, we make a first step towards developing such a theoretical foundation, which can be used as a guideline to answer the above questions. for this, we consider what we call binary metrics. these are metrics that classify the output of a text generation system as being either adequate or inadequate. this allows us to measure the performance of a text generation system as the ratio of adequate responses it generates. furthermore, it allows us to reason about the performance of the metric in terms of true positives and true negatives. ## definitions in this section, we introduce the basic definitions that we need for the derivations. first, we define the general setting of text generation, then we cover binary metrics, and finally we describe text generation systems. ## theory: estimating α with binary metrics in this section, we show how binary metrics can be used to estimate the performance α of text generation systems. for the remainder of the text, assume that t φ = {(i j , o j , r * j )|1 ≤ @xmath0, where i j are inputs, o @xmath1, and r * @xmath2, o j ) denotes the error-free rating of the j th input-output-pair. analogously, let t @xmath3, o j , r j )|1 ≤ @xmath4, where r @xmath5,η b (i j , o j ) denotes the rating of an errorprone (ρ, η)-optimal binary metric. we consider three different cases: 1) the errorfree case, 2) the error-prone metric case, and 3) the mixed case. the error-free case is where we have access to r * j . for instance, we can interpret human evaluation as an example of the error-free case. in the error-prone metric case, we have access only to an (ρ, η)-optimal binary metric. finally, the mixed case is a novel approach that leverages errorfree ratings, which are usually costly to obtain, with error-prone ratings, which are cheaper but are needed en-masse for automated metrics with low ρ and η values, as we will see. usually, in evaluation campaigns, either the first or second setting is applied. we apply a bayesian approach to estimate α by treating it as a random variable, which allows us to model various sources of uncertainty stemming from α, ρ and η, which all need to be estimated from data. the full derivations are given in appendix a. ## minimal number of samples needed to make reliable distinctions between tg systems we now come back to the main question of this paper: how many samples are needed to be able to significantly distinguish the performance of two text generation systems? the intuition is that the closer the performance of the two tg systems is, the more samples are needed. thus, we investigate the setting where their difference in performance |α π 1α π 2 | = ϵ is small. using the formulas from section 3, we can compute the estimates shown in by applying normal approximations for p(α 1 ) and p(α 2 ), and using simulations we can compute the minimal distinguishable difference ϵ for a given set of fixed parameters. the details of the simulations are given in appendix b. ## showcases: application in practise in order to show that the theoretical findings translate to practical applications, we apply our theory to two real-world settings: the wmt21 metric shared task @xcite and the spot-the-bot data @xcite . since the two tasks have significantly different settings (e.g., machine trans-lation and dialogue systems, different types of human annotations, and different types of metrics) this shows that our theory is applicable to a variety of text generation tasks. the showcases highlight the different dimensions that can be manipulated when designing an evaluation. in showcase 1, we highlight the number of ratings needed, whereas, in showcase 2, we focus on the influence of the metric performance. ## conclusion we introduced a theoretical framework for binary metrics that can be used to extract guidelines for designing an evaluation of text generation systems. the framework estimates the performance of a text generation system from a mix of human and automated ratings giving guarantees of which level of significance can be achieved. using the formulas, one can design the evaluation setup and compute estimates of how many human and automated samples are needed for a significant evaluation. we applied the theory to two very different real-world cases and exemplified how the theory can be leveraged to improve the significance of the results. we provide a tool that allows the computation of the formulas so that different settings can be tested. the current theory is limited to binary metrics, but in future work, we will extend the theory to more types, such as comparative or scalar metrics. furthermore, we will apply the theory to a wider range of tasks and domains. in general, we hope to have set in motion efforts to arrive at a sound formalization of the evaluation of text generation systems to increase the robustness, reliability, and significance of future evaluation campaigns.