Exploring Robustness of Machine Translation Metrics: A Study of Twenty-Two Automatic Metrics in the WMT 22 Metric Task (2022)

Abstract: Contextual word embeddings extracted from pre-trained models have become the basis for many downstream NLP tasks, including machine translation automatic evaluations. Metrics that leverage embeddings claim better capture of synonyms and changes in word orders, and thus better correlation with human ratings than surface-form matching metrics (e.g. BLEU). However, few studies have been done to examine robustness of these metrics. This report uses a challenge set to uncover the brittleness of reference-based and reference-free metrics. Our challenge set1 aims at examining metrics’ capability to correlate synonyms in different areas and to discern catastrophic errors at both word- and sentence-levels. The results show that although embedding-based metrics perform relatively well on discerning sentence-level negation/affirmation errors, their performances on relating synonyms are poor. In addition, we find that some metrics are susceptible to text styles so their generalizability compromised.

## introduction automatic metrics compare machine-translated results with human-translated references or/and sources, and give scores accordingly. such metrics offer a quick and inexpensive approach for researchers to evaluate model performances. among these metrics, bleu @xcite has dominated the area for twenty years since its birth in 2002. however, its limitations are obvious: (1) it weighs each word equally but in fact the entropy of each word varies; (2) it only counts n-grams that are exact in the reference and thus synonyms and elaborations are wrongly punished @xcite . consequently, the correlation between bleu and human evaluation is relatively low, which sometimes puzzles researchers. 1 we open-source our challenge set at: in recent years, embedding-based approaches have been introduced to design new automatic metrics. these metrics, e.g. bertscore @xcite , comet @xcite , and bleurt @xcite , claim better ability to capture synonyms and changes in word order, and thus better performance than bleu. apart from ref-based metrics, researches on quality estimation (qe) have been rising, as qe is an cheaper and more convenient approach considering no need of human-translated references. in the wmt metric task, correlation with human annotators is the major indicator to evaluate metric performance @xcite . however, in addition to that, a good metric should meet the following requirements @xcite : (1) sensitivity to nuances in quality among systems or outputs of the same system in different stages of its development so it can be used to direct system performance optimization; (2) consistency and reliability of scores; (3) usability in a great range of fields; (4) speed; (5) low cost. we believe the first three aforementioned requirements are crucial for judging metric performance as well. so we build a @xmath0. section 2 offers a brief description of metrics to be evaluated. details of our challenge set are described in section 3. section 4 presents experiment results and section 5 discusses our findings. ## conclusion this paper presents our submitted challenge set to the wmt22 metrics challenge sets subtask and various metrics' performances on our set. our set focuses on five categories of errors and the result shows that while most metrics are able to identify catastrophic sentence-level affirmation/negation errors, some metrics fail at discerning word-level keyword errors and capturing synonyms of such words. the results show that references are not always useful for a metric to identify errors. in addition, generalizability of metrics should be considered as some metrics are susceptible to test sets styles. the majority of metrics fail to meet the requirements @xcite we discuss in the introduction section. they fail to identify nuances in quality and provide reliable scores, and suffer from domain issues as well. the limitation of this research is that all of the perturbations are human-crafted, and these errors may seldom occur in neural machine translations. to further analyze metric performance in real settings, we will try to annotate and categorize real machine translation errors and evaluate metric performance accordingly. src: 投票存在两极分化的情况，29% 的受访者认为澳大利亚应该尽 快成立共和国，31% 的人则认 为澳大利亚永远不应该成立共 和国。 ref: at the extremes of the poll, 29 per cent of those surveyed believe australia should become a republic as soon as possible, while 31 per cent believe australia should never become a republic. good: the vote was polarised, with 29% of respondents saying australia should become a republic as soon as possible and 31% saying it should never become a republic. bad: the vote was polarised, with 31% of respondents saying australia should become a republic as soon as possible and 29% saying it should never become a republic. phenom: number (swapped number) source: flores src: 除了大件，让傅昆宝两口子头 疼的还有家里1000多斤粮食和 新买的一些家具。 ref: apart from the large items, the over 1,000 jin (500 kg) of grain and newly bought furniture was also a headache fu kunbao and his wife. good: in addition to big items, fu baokun and his wife don't know how to deal with more than 1000 jin of grain and some newly bought furniture in the home. bad: in addition to big items, fu baokun and his wife don't know how to deal with more than 1.