DEMETR : Diagnosing Evaluation Metrics for Translation (2022)

Abstract: While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words. The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlations with human quality judgments than BLEU, are opaque in comparison. In this paper, we shed light on the behavior of these learned metrics by creating DEMETR, a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories. All perturbations were carefully designed to form minimal pairs with the actual translation (i.e., differ in only one aspect). We find that learned metrics perform substantially better than string-based metrics on DEMETR. Additionally, learned metrics differ in their sensitivity to various phenomena (e.g., BERTScore is sensitive to untranslated words but relatively insensitive to gender manipulation, while COMET is much more sensitive to word repetition than to aspectual changes). We publicly release DEMETR to spur more informed future development of machine translation evaluation metrics

## introduction automatically evaluating the output quality of machine translation (mt) systems remains a difficult challenge. the bleu metric @xcite , which is a function of n-gram overlap between system and reference outputs, is still used widely today despite its obvious limitations in measuring source (de) : murray verlor den ersten satz im tiebreak, nachdem beide männer jeden einzelnen aufschlag im satz gehalten hatten. ## diagnosing mt evaluation metrics most existing mt evaluation metrics compute a score for a candidate translation t against a reference sentence r. 3 these scores can be either a simple function of character or token overlap between t and r (e.g., bleu), or they can be the result of a complex neural network model that embeds t and r (e.g., bleurt). while the latter class of learned metrics foot_3 provides more meaningful judgments of translation quality than the former, they are also relatively uninterpretable: the reason for a particular translation t receiving a high or low score is difficult to discern. in this section, we first explain our perturbation-based methodology to better understand mt metrics before describing the collection of demetr, a dataset of linguistic perturbations. ## performance of mt evaluation metrics on demetr we test the accuracy and sensitivity of 14 popular mt evaluation metrics on the perturbations in demetr. we include both traditional stringbased metrics, such as bleu or chrf, as well as newer learned metrics, such as bleurt and comet. within the latter category, we also include two reference-free metrics, which rely only on the source sentence and translation and open possibilities for a more robust mt evaluation. the rest of this section provides an overview of the evaluation metrics before analyzing our findings. detailed results of each metric on every perturbation are found in table a3 . ## sensitivity analysis while the accuracy of a metric on demetr is useful to know, it also obscures the sensitivity of a metric to a particular perturbation. are metrics more sensitive to critical errors than minor ones? are different learned metrics comparatively more or less sensitive to a particular perturbation? in this section, we explore these questions and highlight interesting observations, focusing primarily on the behavior of learned metrics. measuring sensitivity: since each of our metrics has a different score range, we cannot naïvely just compare their score differences to analyze sensitivity. instead, we compute a ratio that intuitively answers the following question: how much does score drop on this perturbation compared to the catastrophic error of producing an empty string? we choose the empty string as a control since it is the perturbation that results in the largest score drop for most metrics. concretely, for a given reference translation r i , machine translation t i , and perturbed translation t ′ i , we compute a ratio z i as: bertscore is relatively more sensitive to some minor errors than it is to critical errors: although we observe that bertscore drops only by a small absolute number for most perturbations, it is actually quite sensitive to many perturbations, especially when passing an unrelated translation and a shuffled version of the existing translation -two of the most drastic perturbations. it also shows higher sensitivity to untranslated words (i.e., codemixing) than to the remaining perturbations, which is to be expected as bertscore uses a multilingual model. however, its sensitivity to incorrect numbers (0.044), gender information (0.067), or aspect change (0.099) is lower than sensitivity to less severe errors, such as tokenized sentence (0.26) or lower-cased sentence (0.33) -a trend visible in other metrics, though not to such an extent. comet-qe, a metric adapted to mqm scoring, does not perform well on demetr: comet-qe trained on mqm ratings (i.e., on the identification of mistakes similar to those included in demetr) varies in its sensitivity to perturbations. while it is sensitive to a sentence with shuffled words, it is not sensitive to a different, unrelated translation (an observation in line with its accuracy). comet-qe also seems to be insensitive to minor errors such as the removal of the final punc-tuation, but also to some major or critical errors such as gender and number replacement. 21 furthermore, comet-qe is much more sensitive to word repetition (0.46-0.72) and word swap (0.41) than to some critical or major errors, such as named entity replacement (0.16) or sentence negation (0.16). overall, comet-qe behaves very differently from most of the other metrics, and in ways that are difficult to explain. overall, all metrics struggle to differentiate between minor and critical errors: while all metrics other than comet-qe are very sensitive to the two baselines (different translation and shuf-fled words) when compared to other perturbations (0.44-2.20), they struggle to differentiate the severity of some critical errors, such as an addition of a plausible but meaning-changing word (0.032-0.12) or incorrect number (0.0038-0.07). these ratios are lower than of some minor errors such as a word repeated four times (0.086-0.72). in fact, bertscore, comet, and comet-qe are more sensitive to word repetition than to an addition of a word which ultimately critically changes the meaning. our work builds on the previous efforts to analyze the performance of mt evaluation metrics, as well as efforts to curate diagnostic datasets for nlp. specia (2019) show that metric performance varies significantly across different levels of mt quality. @xcite demonstrate the importance of reference quality during evaluation. @xcite investigate the performance of pretrained and string-based metrics, and conclude that learned metrics outperform string-based metrics, with comet being the best-performing metric at the time. however, amrhein and sennrich (2022) explore comet models in more depth finding, just as in the current study, that the models are not sensitive to number and named entity errors. @xcite , on the other hand, find that bertscore is more robust to errors in major content words, and less so to small errors. finally, @xcite introduce a leaderboard for generation tasks that ensembles many of the metrics used here. ## conclusion we present demetr, a dataset designed to diagnose mt evaluation metrics. demetr consists of 31k semi-automatically generated perturbations that cover 35 different linguistic phenomena. our experiments showed that learned metrics are notably better than any string-based metrics at distinguishing perturbed from unperturbed translations, which confirms results reported in other studies @xcite . we further explore the sensitivity of learned metrics, showing that even the best-performing metrics struggle to distinguish between minor errors such as word repetition and critical errors such as incorrect number, aspect, and gender. we will publicly release demetr to spur more informed future development of machine translation evaluation metrics.