To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (2021)

Abstract: Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system’s quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on – to the best of our knowledge – the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.

## introduction automatic evaluation metrics are commonly used as the main tool for comparing the translation quality of a pair of machine translation (mt) systems @xcite . the decision of which of the two systems is better is often done without the help of human quality evaluation which can be expensive and time-consuming. however, as we confirm in this paper, metrics badly approximate human judgement @xcite , can be affected by specific phenomena @xcite @xcite or ignore the severity of translation errors @xcite , and thus may mislead system development by incorrect judgements. therefore, it is important to study the reliability of automatic metrics and follow best practices for the automatic evaluation of systems. significant research effort has been applied to evaluate automatic metrics in the past decade, including annual metrics evaluation at the wmt conference and other studies @xcite @xcite . most research has focused on comparing sentence-level (also known as segment-level) correlations between metric scores and human judgements; or system-level (e.g., scoring an entire test set) correlations of individual system scores with human judgement. @xcite emphasize that this scenario is not identical to the common use of metrics, where instead, researchers and practitioners use automatic scores to compare a pair of systems, for example when claiming a new state-of-the-art, evaluating different model architectures, deciding whether to publish results or to deploy new production systems. the main objective of this study is to find an automatic metric that is best suited for making a pairwise ranking of systems and measure how much we can rely on the metric's binary verdicts that one mt system is better than the other. we design a new methodology for pairwise systemlevel evaluation of metrics and use it on -to the best of our knowledge -the largest collection of human judgement of machine translation outputs which we release publicly with this research. we investigate the reliability of metrics across different language pairs, text domains and how statistical tests over automatic metrics can help to increase decision confidence. we examine how the common use of bleu over the past years has possibly negatively affected research decisions. lastly, we re-evaluate past findings and put them in perspective with our work. this research evaluates not only the utility of mt metrics in making pairwise comparisons specifically -it also contributes to the general assessment of mt metrics. based on our findings, we suggest the following best practices for the use of automatic metrics: ## data in this section, we describe test sets, the process for collecting human assessments, and mt systems used in our analysis. we publish all human judgements, metadata, calculated metrics scores, and the code with replication of our findings and promoting further research. we cannot release the proprietary test sets and so system outputs for legal reasons. the collection is available at https: //github.com/microsofttranslator/ toshipornottoship. moreover, we plan to evaluate new metrics emerging in the future. ## automatic metrics in this study, we investigate metrics that were shown to provide promising performance in recent studies (see section 6) and currently most widely used metrics in the mt field. 3 we focus on language-agnostic metrics, therefore we do not include metrics supporting only a small set of languages. the full list of evaluated metrics and their main features is presented in two categories of automatic machine translation metrics can be distinguished: (1) string-based metrics and (2) metrics using pretrained models. the former compares the coverage of various substrings between the human reference and mt output texts. string-based methods largely depend on the quality of reference translations. however, their advantage is that their performance is predictable as it can be easily diagnosed which substrings affect the score the most. the latter category of pretrained methods consists of metrics that use pretrained neural models to evaluate the quality of mt output texts given the source sentence, the human reference, or both. they are not strictly dependent on the translation quality of the human reference (for example, they can better evaluate synonyms or paraphrases). however, their performance is influenced by the data on which they have been trained. moreover, the pretrained models introduce a black-box problem where it is difficult to diagnose potential unexpected behavior of the metric, such as various biases learned from training data. for all metrics, we use the recommended implementation. see appendix a for implementation details. most metrics aim to achieve a positive correlation with human assessments, but some error metrics, such as ter, aim for a negative correlation. we simply negate scores of metrics with anticipated negative correlations. pretrained metrics usually do not support all languages, therefore to ensure comparability, we evaluate metrics on a set of language pairs supported by all metrics. ## meta analysis we analyze findings from past research to put our results in the broader context. we focus on the results on the system-level evaluation, however, a large part of the research studied a sentence-level evaluation. the largest source of metrics evaluation is yearly wmt metric shared task occurring over more than the past ten years @xcite , where various methods are evaluated with human judgement over the set of submitted systems and language pairs in wmt news translation shared tasks. recently, @xcite reevaluated two translation directions from wmt 2020 with the multidimensional quality metric framework and raised a concern that general crowd-sourced annotators used in into-english evaluation in wmt prefer literal translations and have a lower quality than some automatic metrics. past studies evaluate system-level correlations with pearson's correlation calculated for each translation direction separately. we are interested in how metrics correlate with human judgement in general across different language pairs. thus, to generalize the past findings, we use the hunter-schmidt method @xcite , which allows combining already calculated correlations with various sizes. we use it to generalize correlations within each study across all language pairs. for this purpose, hunter-schmidt is effectively a weighted mean of the raw correlation coefficients. although past studies evaluated a larger number of methods and their variants, we have selected a subset of metrics that are evaluated in more than one study or showed promising performance over other metrics in a given study. when a study evaluated several variants of a metric with various parameters, we selected the setting closest to either the recommended setting in the recent years, such as sacrebleu, or a setting that is used in the later evaluation study, mainly in @xcite . meta-analysis in in the study by @xcite , comet under-performed other pretrained metrics. we found out that submitted comet scores failed to score one english-chinese system with tokenized output. however, we obtain valid comet scores on that system output when replicating the results. moreover, we have not seen any problems with comet on chinese. as this one system largely skews pearson's correlation, we also present analysis without english-chinese systems in ## discussion we corroborate results from past studies that pretrained methods are superior to string-based ones. however, pretrained methods are relatively new techniques and we can potentially discover significant drawbacks, for example, they could resemble biases from training data, fail on particular domains, or prefer fluency over adequacy. another problem could arise if an mt system would be trained on the same data as the metric was or if it incorporates the same pretrained model, for example, xlm-r @xcite used by comet. pretrained methods support only a selected set of languages and the quality can differ for each of them. thus, we argue that the string-based method should be used as a secondary metric. an interesting solution to dissipate potential drawbacks of any metric would be if different research groups preselect a different primary pretrained metric in advance to lead their research decisions and to discover improvements not apparent under other metrics. however, we fear that it could lead to "metric-hacking", i.e., picking a metric that confirms results. therefore, we recommend using comet as the primary metric. and to use chrf, the best performing string-based method, as a secondary metric and for unsupported languages. a surprising results is the high accuracy of comet-src, a reference-free metric. it allows automatic evaluation over monolingual domain-specific testsets as suggested by @xcite . limitations of bleu are well-known @xcite . @xcite argued that mt community is overly reliant on it, which @xcite confirmed by showing that 98.8% of mt papers use bleu. we present indirect evidence that the over-use of bleu negatively affects mt development and support deprecation of bleu as the evaluation standard. we show that the reliability of metrics decisions can be increased with statistical significance tests. however, @xcite point out the assumption of statistical significance tests that data samples are independent and adequately distributed is rarely true. also, statistical significance tests do not account for random seed variation across training runs. thus, one should be cautious when making conclusions based on small metrics improvements. @xcite give recommendations for a better use of statistical significance testing. @xcite have shown that almost 40% of mt papers from 2020 copied score from different papers without recalculating them, which is a concerning trend. also, new and better metrics will emerge and there is no need to permanently adhering to a single metric. instead, the simplest and most effective solution to avoid the need to copy scores or stick to obsolete metric is to always publish translated outputs of test sets along with the paper. this allows anyone to recalculate scores with different tools and/or metrics and makes comparisons with past (and future) research easier. there are some shortcomings in our analysis. we have only a handful of non-english systems, therefore we cannot conclude anything about the behaviour of the metrics for language pairs without english. similarly, the majority of our language pairs are high-resource, therefore, we cannot conclude the reliability of metrics for low-resource languages. lastly, many of our translation directions are from translationese into authentic, which as @xcite showed is the easier direction for systems to score high by human judgement. these are potential directions of future work. lastly, we assume that human judgement is the gold standard. however, we need to keep in mind that there can be potential drawbacks of the method used for human judgement or human annotators fail to capture true assessment as @xcite observe. for example, humans cannot explicitly mark critical errors in da and instead they usually assign low assessment scores. ## conclusion we show that metrics can use a different scale for different languages, so pearson's correlation cannot be used. we introduce accuracy as a novel evaluation of metrics in a pairwise system comparison. we use and release a large collection of the human judgement confirming that pretrained metrics are superior to string-based. comet is the best performing metric in our study, and chrf is the best performing string-based method. the surprising effectiveness of comet-src could allow the use of large monolingual test sets for quality estimation. we do not see any drawbacks of the metrics when investigating various languages or domains, especially, for methods pretrained on human judgement. we present indirect evidence that the overuse of bleu negatively affects mt development. we show that statistical testing of automatic metrics largely increases the reliability of a pairwise decision based on automatic metric scores. we endorse the recommendation for publishing translated outputs of research systems to allow comparisons and recalculation of scores in the future. which metrics have equal or higher accuracy than metric α in a given resample. if metric α outperforms metric x by less than 95% of the time, we draw the conclusion that metric x performs on par with 95% statistical significance to the winning metric α.