Tangled up in BLEU : Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020)

Abstract: Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

## introduction automatic metrics are an indispensable part of machine translation (mt) evaluation, serving as a proxy to human evaluation which is considerably more expensive and time-consuming. they provide immediate feedback during mt system development and serve as the primary metric to report the quality of mt systems. accordingly, the reliability of metrics is critical to progress in mt research. a particularly worrying finding was made in the most recent conference on machine translation (wmt), as part of their annual competition findings to benchmark progress in translation and translation evaluation. wmt has established a method based on pearson's correlation coefficient for measuring how well automatic metrics match with human judgements of translation quality, which is used to rank metrics and to justify their widespread use in lieu of human evaluation. their findings @xcite showed that if the correlation is computed for metrics using a large cohort of translation systems, typically very high correlations were found between leading metrics and humans (as high as @xmath0.9). however, if considering only the few best systems, the correlation reduced markedly. this is in contrast to findings at sentence-level evaluation, where metrics are better at distinguishing between high-quality translations compared to lowquality translations @xcite . when considering only the four best systems, the automatic metrics were shown to exhibit negative correlations in some instances. it would appear that metrics can only be relied upon for making coarse distinctions between poor and good translation outputs, but not for assessing similar quality outputs, i.e., the most common application faced when assessing incremental empirical improvements. overall these findings raise important questions as to the reliability of the accepted best-practises for ranking metrics, and more fundamentally, cast doubt over these metrics' utility for tuning highquality systems, and making architecture choices or publication decisions for empirical research. in this paper, we take a closer look into this problem, using the metrics data from recent years of wmt to answer the following questions: to test this we consider a range of system quality levels, including random samples of systems, and show that the problem is widely apparent. overall we find that current metric evaluation methodology can lend false confidence to the utility of a metric, and that leading metrics require either untenably large improvements to serve a gatekeeping role, or overly permissive usage to ensure good ideas are not rejected out of hand. perhaps unsurprisingly, we conclude that metrics are inadequate as a substitute for human evaluations in mt research. foot_0 ## related work since 2007, the conference on machine translation (wmt) has organized an annual shared task on automatic metrics, where metrics are evaluated based on correlation with human judgements over a range of mt systems that were submitted to the translation task. methods for both human evaluation and meta evaluation of metrics have evolved over the years. in early iterations, the official evaluation measure was the spearman's rank correlation of metric scores with human scores @xcite . however, many mt system pairs have very small score differences, and evaluating with spearman's correlation harshly penalises metrics that have a different ordering for these systems. this was replaced by the pearson correlation in 2014 @xcite . to test whether the difference in the performance of two metrics is statis-tically significant, the william's test for dependent correlations is used @xcite , which takes into account the correlation between the two metrics. metrics that are not outperformed by any other metric are declared as the winners for that language pair. pearson's r is highly sensitive to outliers (osborne and overbay, 2004): even a single outlier can have a drastic impact on the value of the correlation coefficient; and in the extreme case, outliers can give the illusion of a strong correlation when there is none, or mask the presence of a true relationship. more generally, very different underlying relationships between the two variables can have the same value of the correlation coefficient (anscombe, 1973). 2 the correlation of metrics with human scores is highly dependent on the underlying systems used. bleu @xcite has remained mostly unchanged since it was proposed in 2002, but its correlation with human scores has changed each year over ten years of evaluation (2006 to 2016) on the english-german and german-english language pairs at wmt @xcite . the low correlation for most of 2006-2012 is possibly due to the presence of strong rule-based systems that tend to receive low bleu scores @xcite . by 2016, however, there were only a few submissions of rule-based systems, and these were mostly outperformed by statistical systems according to human judgements @xcite . the majority of the systems in the last three years have been neural models, for which most metrics have a high correlation with human judgements. bleu has been surpassed by various other metrics at every iteration of the wmt metrics shared task. despite this, and extensive analytical evidence of the limitations of bleu in particular and automatic metrics in general @xcite @xcite , the metric remains the de facto standard of evaluating research hypotheses. ## re-examining conclusions of metrics task 2019 4.1 are metrics unreliable when evaluating high-quality mt systems? in general, the correlation of reference-based metrics with human scores is greater than @xmath1.8 for all language pairs. however, the correlation is dependent on the systems that are being evaluated, and as the quality of mt increases, we want to be sure that the metrics evaluating these systems stay reliable. to estimate the validity of the metrics for highquality mt systems, @xcite sorted the systems based on their direct assessment scores, and plotted the correlation of the top n systems, with n ranging from all systems to the best four systems. they found that for seven out of 18 language pairs, the correlation between metric and human scores decreases as we decrease n , and tends towards zero or even negative when @xmath2. there are four language pairs (german-english, english-german, english-russian, and english-chinese) where the quality of the best mt systems is close to human performance @xcite . if metrics are unreliable for strong mt systems, we would expect to see a sharp degradation in correlation for these language pairs. but as 3 esim's submission to wmt shared task does not include scores for the language pairs en-cs and en-gu. in this paper, we use scores obtained from the same trained model that was used in the original submission. is there another explanation for these results? pearson's r between metrics and da scores is unstable for small samples, particularly when the systems are very close in terms of quality. the low correlation over top-n systems (when n is small) could be an artefact of this instability. to understand this effect, we instead visualise the correlation of a rolling window of systems, starting with the worst n systems, and moving forward by one system until we reach the top n systems. the number of systems stays constant for all points in these graphs, which makes for a more valid comparison than the original setting where the sample size varies. if the metrics are indeed less reliable for strong systems, we should see the same pattern as with the top n systems. for the german-english language pair (figure 1 b), the correlation of most metrics is very unstable when @xmath3. both bleu and chrf perfectly correlate with human scores for systems ranked 2-5, which then drops to -1 for the top 4 systems. on the other hand, esim exhibits the opposite behaviour, even though it shows an upward trend when looking at the top-n systems. even worse, for english-german, yisi-2 obtains a perfect correlation at some values of n , when in fact its correlation with human scores is negligible once outliers are removed (section 4.2). we observe similar behaviour across all lan-guage pairs: the correlation is more stable as n increases, but there is no consistent trend in the correlation that depends on the quality of the systems in the sample. if we are to trust pearson's r at small sample sizes, then the reliability of metrics doesn't really depend on the quality of the mt systems. given that the sample size is small to begin with (typically 10-15 mt systems per language pair), we believe that we do not have enough data to use this method to assess whether metric reliability decreases with the quality of mt systems. a possible explanation for the low correlation of subsets of mt systems is that it depends on how close these systems are in terms of quality. in the extreme case, the difference between the da scores of all the systems in the subset can be statistically insignificant, so metric correlation over these systems can be attributed to chance.