Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers (2021)

Abstract: This paper presents the first large-scale meta-evaluation of machine translation (MT). We annotated MT evaluations conducted in 769 research papers published from 2010 to 2020. Our study shows that practices for automatic MT evaluation have dramatically changed during the past decade and follow concerning trends. An increasing number of MT evaluations exclusively rely on differences between BLEU scores to draw conclusions, without performing any kind of statistical significance testing nor human evaluation, while at least 108 metrics claiming to be better than BLEU have been proposed. MT evaluations in recent papers tend to copy and compare automatic metric scores from previous work to claim the superiority of a method or an algorithm without confirming neither exactly the same training, validating, and testing data have been used nor the metric scores are comparable. Furthermore, tools for reporting standardized metric scores are still far from being widely adopted by the MT community. After showing how the accumulation of these pitfalls leads to dubious evaluation, we propose a guideline to encourage better automatic MT evaluation along with a simple meta-evaluation scoring method to assess its credibility.

## introduction new research publications in machine translation (mt) regularly introduce new methods and algorithms to improve the translation quality of mt systems. in the literature, translation quality is usually evaluated with automatic metrics such as bleu (papineni et al., 2002) and, more rarely, by humans. to assess whether an mt system performs better than another mt system, their scores given by an automatic metric are directly compared. while such comparisons between mt systems are exhibited in the large majority of mt papers, there are no well-defined guideline nor clear prerequisites under which a comparison between mt systems is considered valid. consequently, we assume that evaluation in mt is conducted with different degrees of thoroughness across papers and that evaluation practices have evolved over the years. what could be considered, by the research community, as a good evaluation methodology ten years ago may not be considered good today, and vice versa. this evolution has not been studied and whether mt evaluation has become better, or worse, is debatable. on the other hand, several requirements for mt evaluation have been well-identified. for instance, the limitations of bleu are well-known (callison-burch et al., 2006; reiter, 2018; mathur et al., 2020) and the necessity to report automatic metric scores through standardized tools, such as sacre-bleu, has been recognized (post, 2018). moreover, a trustworthy evaluation may adopt statistical significance testing (koehn, 2004) and strong baselines (denkowski and neubig, 2017). however, to what extent these requirements have been met in mt publications is unclear. in this paper, we propose the first large-scale meta-evaluation of mt in which we manually annotated 769 research papers published from 2010 to 2020. our study shows that evaluation in mt has dramatically changed since 2010. an increasing number of publications exclusively rely on bleu scores to draw their conclusions. the large majority of publications do not perform statistical significance testing, especially since 2016. moreover, an increasing number of papers copy and compare bleu scores published by previous work while tools to report standardized metric scores are still far from being extensively adopted by the mt community. we also show that compared systems are often trained, validated, or even evaluated, on data that are not exactly the same. after demonstrating how the accumulation of these pitfalls leads to dubious evaluation, we propose a general guideline for automatic evaluation in mt and a simple scoring method to meta-evaluate an mt paper. we believe that the adoption of these tools by authors or reviewers have the potential to reverse the concerning trends observed in this meta-evaluation. ## a survey on mt evaluation we manually annotated the mt evaluation in research papers published from 2010 to 2020 at *acl conferences. @xcite to identify mt papers, we searched the acl anthology website 2 for the terms "mt" or "translation" in their titles 3 and analyzed among them the 769 papers that make comparisons of translation quality between at least two mt systems. for each year between 2010 and 2020, we respectively annotated the following numbers of papers: 53, 40, 59, 80, 67, 45, 51, 62, 94, 115, and 103. we annotated each paper as follows: a1. all the automatic metrics used to evaluate the translation quality of mt systems. we did not list variants of the same metric: e.g., chrf3 and chrf++ are labeled chrf (popović, 2015). moreover, we did not consider metrics which only target specific aspects of the translation quality, such as pronoun translation and rare word translation. a2. whether a human evaluation of the translation quality has been conducted: yes or no. if the human evaluation only targets specific types of errors and did not evaluate the translation quality of the entire text, we answered "no." 4 a3. whether any kind of statistical significance testing of the difference between automatic metric scores has been performed: yes or no. potentially, some papers did perform significance testing without mentioning it, but due to the lack of evidences such papers have been annotated with "no" for this question. @xcite we considered only *acl main conferences, namely acl, naacl, eacl, emnlp, conll, and aacl, as they are the primary venues for publishing mt papers. a4. whether it makes comparisons with automatic metric scores directly copied from previous work to support its conclusion: yes or no. most papers copying scores (mostly bleu) clearly mention it. if there is no evidence that the scores have been copied, we annotated these papers with "no" for this question. a5. whether sacrebleu has been used: yes or no. if there is no mention or reference to "sacrebleu," we assume that it has not been used. note that "yes" does not mean that the paper used sacrebleu for all the mt systems evaluated. a6. if previous work has not been reproduced but copied, whether it has been confirmed that all the compared mt systems used exactly the same pre-processed training, validating, and testing data: yes or no. except for a6, the annotation was straightforward since most papers present a dedicated section for experimental settings with most of the information we searched for. answering a6 required to check the data exploited in the previous work used for comparison. note that answering "yes" to the questions from a2 to a6 may only be true for at least one of the comparisons between mt systems, while we did not evaluate how well it applies. for instance, answering "yes" to a5 only means that at least one of the systems has been evaluated with sacrebleu but not that the sacrebleu signature has been reported nor that sacrebleu scores have been correctly compared with other bleu scores also computed with sacrebleu. our annotations are available as a supplemental material of this paper. to keep track of the evolution of mt evaluation, we will periodically update the annotations and will make it available online. @xcite ## pitfalls and concerning trends this section discusses the four main pitfalls identified in our meta-evaluation of mt: the exclusive use of bleu, the absence of statistical significance testing, the comparison of incomparable results from previous work, and the reliance on comparison between mt systems that do not exploit exactly the same data. we report on how often they affected mt papers and recent trends. based on previous ## conclusion our meta-evaluation identified pitfalls in the mt evaluation in most of the annotated papers. the accumulation of these pitfalls and the concerning trends we observed lead us to propose a guideline for automatic mt evaluation. we hope this guideline, or a similar one, will be adopted by the mt community to enhance the scientific credibility of mt research. this work also has its limitations since it does not cover all the pitfalls of mt evaluation. for instance, we noticed that mt papers regularly rely on the same language pairs to claim general improvements of mt. they also almost exclusively focus on translation from or into english. another, more positive observation, is that mt papers tend to use stronger baseline systems, following some of the recommendations by denkowski and neubig (2017), than at the beginning of the last decade when baseline systems were mostly vanilla mt systems. for future work, we plan to extend our meta-evaluation of mt to publications at conferences in other research domains, such as machine learning and artificial intelligence. as a final note, we would like to encourage nlp researchers to perform a similar meta-evaluation in their respective area of expertise. as we showed, it can unveil pitfalls and concerning trends that can be reversed before becoming prevalent.