Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics (2023)

Abstract: While many new automatic metrics for machine translation evaluation have been proposed in recent years, BLEU scores are still used as the primary metric in the vast majority of MT research papers. There are many reasons that researchers may be reluctant to switch to new metrics, from external pressures (reviewers, prior work) to the ease of use of metric toolkits. Another reason is a lack of intuition about the meaning of novel metric scores. In this work, we examine “rules of thumb” about metric score differences and how they do (and do not) correspond to human judgments of statistically significant differences between systems. In particular, we show that common rules of thumb about BLEU score differences do not in fact guarantee that human annotators will find significant differences between systems. We also show ways in which these rules of thumb fail to generalize across translation directions or domains.

## introduction despite mounting evidence over the course of many years @xcite @xcite @xcite demonstrating that bleu @xcite has fundamental flaws in accurately reflecting translation quality, it has remained the de facto standard automatic mt evaluation metric for both scientific research and practical deployment @xcite . numerous research efforts @xcite @xcite @xcite have focused on the correlations between human judgments of translation quality and automatic metric scores; year after year, these have shown new metrics correlating better with human judgments than bleu does. there are certainly some other obstacles beyond correlation with human judgment on translation quality that hinder the adoption of newer and better human-correlating automatic mt evaluation metrics in practice. @xcite outlined four objectives in the search for new and improved automatic mt evaluation metrics: 1) "high correlation with human assessments of translation quality"; 2) "applicable to multiple target languages"; 3) "ability to differentiate between systems of varying quality" and finally, 4) "intuitive interpretation", i.e. whether the scores are meaningful and easy to understand on their own, with values and differences that are interpretable and clear in practice. as the first three objectives can be addressed by the correlation analysis of mt metrics with human judgment on translation quality but not the last one, we believe that gaining an intuitive understanding of the properties and behavior of the metrics is one of the remaining challenges that mt researchers are facing when they are considering adopting a new metric. one way to do this is by designing metrics to be easily interpretable; another way is to examine whether we can build up reliable and useful intuitions about existing metrics. in order for metrics to be widely adopted, a combination of these-making new metrics that are more interpretable and simple to understand or debug, foot_0 as well as forming intuitions about them-may be necessary. our focus in this work is on the latter, examining existing metrics to understand the meaning of the score differences they present. in this work, we focus in particular on whether it is possible to get a sense of what kinds of metric score differences may correspond to significant improvements as judged by human annotators. 2 we examine whether this is consistent across target languages and across translation domains, within a specific metric. we do not suggest that this means that mt researchers can forego running significance tests or doing the appropriate human evaluation; as marie (2022) notes, "a rule of thumb may yield correct results but can't be scientifically credible." however, having these rules of thumb and intuitive senses of metric score meanings may indeed be necessary to encourage broader adoption, so we present this work solely focusing on whether it is possible to build such rules of thumb about some of the modern metrics. nevertheless, there remain some common "gut feelings" among researchers and reviewers about what constitutes "significant" improvement on the basis of metric score differences alone, without running human evaluation or significance tests. as @xcite note, the majority of mt papers since 2018 do not use significance tests and instead rely on score differences. one number commonly tossed about informally is that a score difference of around 2 bleu points can typically be expected to be significant. but where does this assumption come from, and does it hold? one possible source for this is @xcite , which found, specific to the particular test scenario that "even for small test sets of size 300 sentences (about 9000 words), we can reliably draw the right conclusion, if the true bleu score difference is at least 2-3%." 3 in that setting, the "right conclusion" is the one that matches the conclusion drawn from a very large test set (30,000 sentences) about which of two systems is better, based on automatic metric scores. the goal of that work is to identify how small of a dataset can still be reliably used (along with bootstrap resampling for statistical significance) to draw conclusions about the automatically measured differences between two systems. however, sometimes this kind of bleu difference is used informally as a proxy for whether a human annotator will find the difference notable, something that does not follow from that particular paper. @xcite found that for systems from wmt21 and 22, almost all system pairs with a bleu difference greater than 2.0 were significantly different with p-@xmath0.001, though this significance judgment relates only to the metric scores and not to any human annotation. in this work, we focus on a question more closely related to this and to @xcite , rather than marie (2022): whether there exist rules of thumb about metric score differences and their correspondence to significant differences in human judgments. regardless of the exact source of these rules of thumb (which may never be known) or the exact bleu score difference (or exact relative improvement) of a particular rule of thumb, some researchers feel that they have a sense of metric score differences, and we examine how that may correspond (or not) to judgments of mt quality across a range of metrics in this work. similar to @xcite , we are interested in the relationship between metric score differences and significant differences in human scores. that work is interested both in type i errors (where an insignificant metric difference might correspond to an actually significant difference under human evaluation) and type ii errors (where the metric score difference is significant, but the human evaluation does not find a significant difference). we take a related but slightly different approach to examining this relationship. we examine the "rules of thumb" about which metric score differences are meaningful. using the large number of system pairs from wmt evaluations, we look at how the metric difference between two systems is related to the probability that the human annotations find the systems to be statistically significantly different. we select a threshold for this probability and examine the metric difference that corresponds. we then examine whether this is consistent across different test sets, domains, and target languages. that is, are there consistent rules of thumb about metric score differences? or is there too much variation? in casual discussion and sometimes even formal work or reviewing, there is often a conflation of several (somewhat) orthogonal topics, which may be the source of these intuitions and rules of thumb. sometimes "significant" is used simply to mean some value of "large", unrelated to precise definitions of statistical significance testing. @xcite note the use of this convention and suggest that it indicates some level of consensus among researchers on bleu differences, albeit a consensus that is not necessarily well-founded; they address a number of other pitfalls in mt evaluation as well. in particular, in their meta-evaluation of 769 mt papers, they note that the majority of recent papers do not perform statistical significance tests, relying instead just on the "amplitude of the differences between metric scores to state whether they are significant or not"; in fact they note that even a bleu score difference of around 1 may be used by most mt papers as "significant evidence of the superiority of an mt system and as an improvement in translation quality" @xcite . these are assumptions that sufficiently large metric score differences guarantee significant differences in metric scores; when combined with the assumption that metric scores and human scores are well-correlated, this often leads to the assumption that a certain metric score difference guarantees a statistically significant difference in human scores. we examine this relationship between statistically significant differences in human scores and the magnitude of metric differences in this work. first, we investigate whether the more generous rule of thumb surrounding the significance of 2 bleu improvement has a basis in fact. while @xcite has shown that (at least for wmt21 and 22) such a bleu difference tends to be a significant difference (@xmath1.001) in metrics score, does that mean that human annotators will judge the pair of systems to be meaningfully different? that is to say, we assess the probability that an mt system pair would be judged by humans as having a statistically significant difference in quality, if bleu showed a difference of 2 or more points for that pair. ## discussion another unaddressed problem of the rules of thumb is that the mt community may sometimes treat them as though they are language and domain independent, applying rules of thumb across different target languages and domains without considering their differences. this is the case despite the fact that it is widely known that language typology affects bleu scores (for example, highly inflected languages may see their bleu scores penalized due to single-character differences in affixes). in addition, recent wmt metrics shared tasks @xcite has moved on to using multidimensional quality metric (mqm) @xcite as the human annotation method for translation quality for more consistent and reliable annotations @xcite . we now investigate into the consistency of the cutoff of metrics' score differences at 80% confidence level for different target languages, evaluation domains and human annotation methods. ## conclusions we presented an empirical study of the relationship between statistically significant differences in human scores and the magnitude of metric differences. we showed that the rules of thumb surrounding the significance of bleu improvement does not hold according to human judgment on translation quality (regardless of whether the rule of thumb is exactly 1 or 2 or even slightly larger bleu differences). we provided an intuitive interpretation between bleu differences and the differences in some of the modern metrics. however, we found that for some metrics, the score differences corresponding to significant improvements as judged by human annotators may not be transferable across target languages or translation domains. we have to emphasize again that we do not suggest that this means that mt researchers can forego running significance tests or doing the appropriate human evaluation. this work only supports an intuitive senses of metric score meanings to encourages broader adoption of new automatic mt evaluation metrics.