Linguistically Motivated Evaluation of Machine Translation Metrics Based on a Challenge Set (2022)

Abstract: We employ a linguistically motivated challenge set in order to evaluate the state-of-the-art machine translation metrics submitted to the Metrics Shared Task of the 7th Conference for Machine Translation. The challenge set includes about 20,000 items extracted from 145 MT systems for two language directions (German-English, English-German), covering more than 100 linguistically-motivated phenomena organized in 14 categories. The best performing metrics are YiSi-1, BERTScore and COMET-22 for German-English, and UniTE, UniTE-ref, XL-DA and xxl-DA19 for English-German.Metrics in both directions are performing worst when it comes to named-entities & terminology and particularly measuring units. Particularly in German-English they are weak at detecting issues at punctuation, polar questions, relative clauses, dates and idioms. In English-German, they perform worst at present progressive of transitive verbs, future II progressive of intransitive verbs, simple present perfect of ditransitive verbs and focus particles.

## introduction automatic evaluation metrics have been valuable tools for machine translation (mt), allowing quick evaluation and suggesting directions for further development. many metrics have been suggested throughout the years, which in turn sets the requirement for their evaluation. whereas mt metrics so far have been evaluated based on the agreement of their scores with human judgments on test sets drawn from broad text, little research has taken place on investigating whether the performance of the metrics generalizes enough when evaluating particular cases. a more target way of evaluating metrics is using challenge sets. these are targeted test sets, which have been devised in such a way, so that they benchmark the ability of metrics to score particular translation phenomena. in this paper we present empirical results on the performance of mt metrics, using an extensive challenge set, which includes thousands of test items aiming to test the performance over more than one hundred linguistically-motivated phenomena in two language directions. it is based on thousands of manually created test items, their translation outputs from dozens of mt systems and semi-automatically evaluated with the supervision of linguists. through this analysis we attempt to reveal strengths and weaknesses of several state-ofthe-art mt metrics considering their background methods with regards to linguistic aspects. the rest of the paper is structured as follows. in section 2 related work is briefly described. in section 3 we describe the construction of the challenge set and the evaluation protocol. the empirical results are outlined in section 4, followed by a conclusion is section 5. ## related work the need for a thorough evaluation of natural language processing (nlp) tools has lately received increased interest in the research community, indicated by a big amount of publications, among them several which received best paper awards @xcite @xcite . when focusing on mt, first efforts were made in the 1990s with the introduction of test suites @xcite , which were revived after the latest advances in the field @xcite . to the best of our knowledge, the first efforts relevant to the application of challenge sets on mt metrics was presented as an analysis at the findings paper of the metrics shared task of the 6th conference of machine translation @xcite , based on our test suite @xcite that we are using on this paper. hereby we are advancing as to that preliminary analysis by (a) increasing the number of challenge items to about 9,000-10,000, including outputs from state-of-the-art systems from 2021, (b) adding a second language direction (english-german) (c) presenting a more fine-grained analysis, not only in the category level but also on the phenomenon level. this way we can get more confident and more generalisable empirical conclusions. ## results the results are displayed in detail in tables 1 and 3 in the category level and in tables 4 and 5 for the phenomenon level, for both language directions german-english and english-german respectively. ## conclusion in this paper we analyzed the performance of several state-of-the-art metrics with regards to particular linguistically-motivated phenomena for two language pairs, german-english and english-german. the analysis gave a multitude of observations, re- for the evaluation of portuguese-english machine translation. in computational processing of the portuguese language, pages 15-25, cham. springer international publishing. eleftherios avramidis, vivien macketanz, ursula strohriegel, aljoscha burchardt, and sebastian möller. 2020. fine-grained linguistic evaluation for state-of-the-art machine translation. in proceedings of the fifth conference on machine translation. association for computational linguistics. eleftherios avramidis, vivien macketanz, ursula strohriegel, and hans uszkoreit. 2019. linguistic evaluation of german-english machine translation using a test suite. in proceedings of the fourth conference on machine translation. conference on machine translation (wmt-2019), pages 644-653, florence, italy. association for computational linguistics. aljoscha burchardt, vivien macketanz, jon dehdari, georg heigold, jan-thorsten peter, and philip williams. 2017. a linguistic evaluation of rule-based, phrase-based, and neural mt engines. the prague bulletin of mathematical linguistics, 108:159-170. niccolò campolungo, federico martelli, francesco saina, and roberto navigli. 2022. dibimt: a novel benchmark for measuring word sense disambiguation biases in machine translation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 4331-4352, dublin, ireland. association for computational linguistics. markus freitag, ricardo rei, nitika mathur, chi-kiu lo, craig stewart, george foster, alon lavie, and ondřej bojar. 2021. results of the wmt21 metrics shared task: evaluating metrics with expert-based human evaluations on ted and news domain. in proceedings of the sixth conference on machine translation, pages 733-774, online. association for computational linguistics. liane guillou and christian hardmeier. 2016. protest: a test suite for evaluating pronouns in machine translation. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 636-643, portorož, slovenia. european language resources association (elra). margaret king and kirsten falkedal. 1990. using test suites in evaluation of machine translation systems. in proceedings of the 13th conference on computational linguistics, volume 2, pages 211-216, morristown, nj, usa. association for computational linguistics. yilun liu, xiaosong qiao, zhanglin wu, su chang, min zhang, yanqing zhao, shimin tao song peng, hao yang, ying qin, jiaxin guo, minghan wang, yinglu li, peng li, and xiaofeng zhao. 2022. partial could be better than whole: hw-tsc 2022 submission for the metrics shared task. in proceedings of the seventh conference on machine translation, abu dhabi. association for computational linguistics. chi-kiu lo. 2019. yisi -a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 507-513, florence, italy. association for computational linguistics. vivien macketanz, eleftherios avramidis, aljoscha burchardt, and hans uszkoreit. 2018. fine-grained evaluation of german-english machine translation based on a test suite. in proceedings of the third conference on machine translation (wmt18), pages 578-587, brussels, belgium. association for computational linguistics. vivien macketanz, eleftherios avramidis, aljoscha burchardt, he wang, renlong ai, shushen manakhimova, ursula strohriegel, sebastian möller, and hans uszkoreit. 2022. a linguistically motivated test suite to semi-automatically evaluate german-english machine translation output. in proceedings of the thirteenth language resources and evaluation conference, pages 936-947, marseille, france. european language resources association. vivien macketanz, eleftherios avramidis, shushen manakhimova, and sebastian möller. 2021. linguistic evaluation for the 2021 state-of-the-art machine translation systems for german to english and english to german. in proceedings of the sixth conference on machine translation. (wmt21), punta cana, dominican republic. association for computational linguistics. ananya mukherjee and manish shrivastava. 2022. reuse: reference-free unsupervised quality estimation metric. in proceedings of the seventh conference on machine translation, abu dhabi. association for computational linguistics. kishore papineni, salim roukos, todd ward, and wei-jing zhu. 2002. bleu: a method for automatic evaluation of machine translation. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 311-318, philadelphia, pennsylvania, usa. association for computational linguistics. stefano perrella, lorenzo proietti, alessandro scirè, niccolò campolungo, and roberto navigli. 2022. machine translation evaluation as a sequence tagging problem. in proceedings of the seventh conference on machine translation, abu dhabi. association for computational linguistics. ricardo rei, josé g. c. de souza, duarte alves, chrysoula zerva, ana c farinha, taisiya glushkova, alon lavie, luisa coheur, and andré f. t. martins. 2022. comet-22: unbabel-ist 2022 submission appendix german-english idiom src ich glaube, tim hat ein auge auf lena geworfen. ref i think tim has a crush on lena. 3 i think tim has cast an eye on lena. 7 i think tim has an eye on lena. polar question src willst du mit mir ins kino gehen? ref do you want to go to a movie with me? 3 do you want to go with me into the cinema? 7 you want to go to the cinema with me? measuring unit src ein ausgewachsener afrikanischer elefant wiegt etwa sechs tonnen. ref an adult african elephant weighs about six tons. 3 a fully grown african elephant weighs about six tons. 7 an adult african elephant weighs about six tonnes. comma src er fragte sich, welches auto er kaufen sollte. ref he wondered what car to buy. 3 he wondered which car to buy. 7 he asked himself, which car he should buy. when are we going to meet? they wanted to know.