Better than Average: Paired Evaluation of NLP systems (2021)

Abstract: Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which system is best, since the average, as well as alternatives such as the median, ignores the pairing arising from the fact that systems are evaluated on the same test instances. We illustrate the importance of taking the instancelevel pairing of evaluation scores into account and demonstrate, both theoretically and empirically, the advantages of aggregation methods based on pairwise comparisons, such as the Bradley–Terry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, we show that the choice of aggregation mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facilitate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appropriate statistical testing.

## introduction research is driven by evaluation results, with attention and resources being focused on methods identified as state of the art (sota). the proper design of evaluation methodology is thus crucial to ensure progress in the field. in nlp, evaluation usually consists in comparing the averaged scores of competing systems over a common set of test instances. indeed, averaging scores independently for each system and declaring the one with the highest average to be best is particularly all systems have the same mean. c is better than a on all instances but one, so bt declares @xmath0, b is better than a on all instances but one, so bt declares @xmath1, whereas the median of a is greater, and the means are the same. overall, mean and median fail to capture the complex instance-level pairing. here, we critically assess the specific choice of the average to aggregate evaluation scores. in particular, we emphasize that there is a natural instance-level pairing between the evaluation scores of systems, which aggregation mechanisms such as the mean or median fail to take into account: as they produce a score for each system independently, systems that have the same set of scores (but potentially in different order) cannot be distinguished. consider the three systems a, b, and c compared on five test instances in fig. 1 . despite a complex pairing structure, they all have the same mean score across test instances. moreover, even though b is better than a on all test instances but one, the median of a is greater than the median of b. in this work, we discuss an alternative aggregation mechanism: the bradley-terry (bt) model @xcite . bt compares systems for each test instance and estimates the latent strength of systems based on how frequently one system scores higher than another. such paired mechanisms have already been successfully used to aggregate human judgments @xcite ; for example, wmt evaluation protocols regularly employ trueskill @xcite , a bayesian variant of bt @xcite . contributions. we contribute the first comprehensive analysis of the bt model (especially vis-à-vis mean and median) as an aggregation mechanism for comparing system scores in nlp. code and data for replicating our analyses and experiments is available online. ## aggregation of evaluation results in this section, we briefly present the three aggregation mechanisms we consider. ## comparison of assumptions since the roles played by a and b are symmetrical, we now assume without loss of generality that system a is better, i.e., λ @xmath2. the proof is given in appendix b. note that, whereas the expectation is linear ( robustness to ouliers. the mean is not robust to outliers: e a-b can be swayed above or below the threshold of 0 by a small number of test instances for which the difference between system scores is large. on the contrary, the median is a robust statistic that cannot be easily influenced by outliers. similarly, bt is robust to outliers because its decision is based on the median of differences m a-b . ## simulations with synthetic data next, we perform simulations to extend the analysis of the previous section to (i) @xmath3, (ii) finitely many test samples, (iii) a practical implementation of bt (for @xmath4, bt is an iterative optimization algorithm, as discussed in appendix e). we synthesize evaluation scores with various properties starting with systems of predefined implicit strengths λ i . to create situations where the pairing of evaluation scores matters, we introduce multiple test instance types. for each type, systems perform differently but still have the same relative strength (p(@xmath5, differing only by an added offset. for example, the evaluation scores obtained by a and b could be sampled from n (λ a , σ) and n (λ b , σ) for one test instance type, and by n (λ a + , σ) and n (λ b + , σ) for another type, with being the offset. we sample evaluation setups by varying the following properties: the number of systems, the number of test instances, the percentage of outliers, the numbers of test instance types, and the level of noise. this results in 2,880 simulated evaluation setups. in appendix a.2, we give the detailed algorithm and parameters used to generate the data. in fig. 3 , we report kendall's τ between the latent scores λ i and the aggregated scores estimated by mean, median, and bt. when the evaluation setup does not present any difficulty (fig. 3(a , b )), all aggregation mechanisms work equally well (within each other's 95% error bounds), improving with more samples (fig. 3(b) ) and deteriorating with more systems (fig. 3(a) ). unsurprisingly, mean fails in the presence of outliers, whereas median and bt are unaffected (fig. 3(c , e , f )). when several types of test instances are considered, median begins to fail (fig. 3(d )), which is made worse when outliers are also present (fig. 3(f) ). overall, bt is more robust and does not fail when the pairing matters fig. 3(g , h ). ## analysis of empirical data in this section, we perform large-scale experiments using real evaluation scores from four nlg tasks. for summarization, we use the tac-08, tac-09, tac-11 and cnn/dm @xcite datasets. for machine translation, we use the shared tasks of wmt-17 @xcite , wmt-18 @xcite , and wmt-19 @xcite . for image captioning, we use the mscoco @xcite dataset, and for dialogue, we use the personachat and topicalchat (mehri and eskenazi, 2020) datasets. the evaluation scores are obtained with a total of 18 different evaluation metrics: bleu-[1,2,3,4] @xcite , rouge-[1,2,l] @xcite , rouge-we-[1,2] (ng and abrecht, 2015), js-[1,2] @xcite , s3-[pyr, resp] @xcite , cider @xcite , chrfpp @xcite , me-teor @xcite , moverscore @xcite , and bertscore @xcite . some metrics are only available for some task; e.g., cider, meteor are only available for the image captioning task. we provide details about datasets, metrics, and their statistics in appendix a.3. overall, across datasets and metrics we have 296 evaluation setups, 73,471 pairs of systems, and 91,197 test instances. we also experiment with sub-sampling different sizes of test sets (see appendix a.3) to simulate varying train/dev/test splits or cross-validation. ## related work several studies have made a critical assessment of the standard evaluation methodologies. for example, @xcite demonstrate the advantages of carefully choosing which references to use for nlg evaluation. @xcite show that outliers matter in practice. recently, graham et al. (2020) draws attention on test set size. several works have emphasized the importance of careful statistical testing @xcite @xcite . they recommend paired statistical tests. finally, @xcite report that "relative rankings yield more discriminative results than absolute assessments", which further motivates aggregation mechanisms like bt. aggregations. pairwise comparison mechanisms date back to @xcite . subsequently, the bradley-terry (bt) model has become a standard pairwise comparison model (bradley and @xcite . in nlp, bt-inspired mechanisms have sometimes been used to aggregate human assessments. for instance, @xcite ranked chatbots regarding their ability to mimic conversational behavior of humans. item response theory (irt) has a similar formulation as bt, but also estimates the difficulty of each test instances using a latent-variable bayesian model @xcite . irt has been applied to perform dataset filtering @xcite , evaluate chatbots from human assessments @xcite , and aggregate human assessments in machine translation @xcite . elo @xcite and trueskill @xcite are famous extensions of the bt model commonly used to rate players in the context of gaming or sports events. elo views player strengths as normally distributed random variables. trueskill is a bayesian variant of elo. since 2015, the workshop on machine translation (wmt) has been using trueskill to rank models based on human assessments following the methodology of sakaguchi et al. (2014). we provide a detailed presentation and comparison of bt, elo, and trueskill in appendix g, and make both elo and trueskill available as alternatives to bt in the released tool. the arguments in favor of bt made in this work transfer to its variants, including irt, elo, and trueskill, and the conclusions drawn from the experiments of sec. 5 still hold when replacing bt by elo or trueskill (appendix g). our work extends previous works that has considered bt variants by analyzing the potential causes for disagreement with mean and median and by measuring the disagreement in real nlp evaluation setups. ## discussion we briefly discuss some possible questions raised by the use of bt-like metrics, with more details provided in appendix e, f, g, and h. ## conclusion we performed a critical assessment of the standard nlp evaluation methodology based on averaged scores, which ignores the natural instance-level pairing of evaluation scores when comparing systems. we showed the importance of the pairing and demonstrated the advantages of paired mechanisms such as bradley-terry (bt) over more standard aggregation schemes such as the mean or median. the choice of aggregation mechanism matters in real evaluation setups, and we therefore recommend bt as a robust aggregation mechanism. to facilitate adoption, we release pairformance, a new tool to perform full analyses of system scores using bt and two of its variants, elo and trueskill.