Linguistic Evaluation for the 2021 State-of-the-art Machine Translation Systems for G erman to E nglish and E nglish to G erman (2021)

Abstract: We are using a semi-automated test suite in order to provide a fine-grained linguistic evaluation for state-of-the-art machine translation systems. The evaluation includes 18 German to English and 18 English to German systems, submitted to the Translation Shared Task of the 2021 Conference on Machine Translation. Our submission adds up to the submissions of the previous years by creating and applying a wide-range test suite for English to German as a new language pair. The fine-grained evaluation allows spotting significant differences between systems that cannot be distinguished by the direct assessment of the human evaluation campaign. We find that most of the systems achieve good accuracies in the majority of linguistic phenomena but there are few phenomena with lower accuracy, such as the idioms, the modal pluperfect and the German resultative predicates. Two systems have significantly better test suite accuracy in macro-average in every language direction, Online-W and Facebook-AI for German to English and VolcTrans and Online-W for English to German. The systems show a steady improvement as compared to previous years.

## introduction evaluation in nlp and particularly in machine translation (mt) is an essential process for identifying flaws and leading further system improvements. nevertheless, the exact method of evaluation to be used varies, given the quality requirements of the particular use case. whereas the vast majority of the evaluation methods reside on metrics or direct assessment by humans to produce a single quality score given an entire test set, a recent trend has opted to evaluating the details of the produced translations, with major focus on their correctness from a linguistic perspective. for this reason, the translation systems are not tested based on generic test-sets, but they are given input which is particularly crafted to trial their performance. most commonly, this is done with the help of a test suite (cf. @xcite @xcite . the paper at hand describes the use of a test suite in order to evaluate 18 german to english and 18 english to german mt systems that participated at the shared task of the sixth conference on machine translation (wmt21) foot_0 . the evaluation is performed by an extensive test suite that tests a wide range of linguistically motivated phenomena. in addition to our contributions in the previous years, which focused only on german to english, this year we are presenting for the first time results with an extensive test suite with a similar logic for the opposite direction english to german. our german to english test set contains 5,560 test sentences, covering 107 linguistic phenomena that are organized in 14 categories. the english to german test set contains 4,443 test sentences, covering 111 linguistic phenomena that are organized in 12 categories. ## related work test suites have already been used since the beginnings of mt in the 1990s @xcite @xcite . with the rise of deep learning, the quality of mt outputs has improved significantly, which in turn lead to a recent revival of test suites that focus on the evaluation of specific linguistic phenomena (e.g., pronoun translation @xcite , or on the comparison of different @xcite , and quality estimation methods @xcite . within the scope of the test suite track of the conference on machine translation, several test suites for multiple language directions have lexical ambiguity er las gerne novellen. he liked to read novels. fail he liked to read novellas. pass phrasal verb warum starben die dinosaurier aus? why did the dinosaurs die? fail why did the dinosaurs die out? pass why did the dinosaurs become extinct? pass ditransitive perfect ich habe tim einen kuchen gebacken. i have baked a cake. fail i baked tim a cake. ## methods our test suite consists of two test sets (one per language direction) that have been created manually with the aim of testing the performance of mt systems. they cover a wide variety of linguistic phenomena which are grouped in different categories. while there is a big overlap between the linguistic categories and phenomena in the two test sets, there are also many differences as the categories and phenomena are language-specific. some exemplary test sentences can be seen in ## results the accuracies resulting from the application of the test suite on the system outputs can be seen in the tables in the appendix. we first present the results aggregated in categories (tables 4 and 5 ) so one can have a broad overview of the systems performance, whereas afterwards a yearly comparison with last years ( ## comparison with previous years the progress of the systems performance through the last four years for german-english can be seen in individual systems show some small improvements in general, but the fine-grained evaluation is able to indicate some significant deterioration in particular categories. for example, online-b, online-g and volctrans, despite their overall improvement, show a significant deterioration regarding verb tense/aspect/mood, which reaches a -9% in the case of volctrans. other deteriorations occur for several systems regarding false friends and function words. this shows that the overall improvement in translation quality may occur at the expense of particular qualitative aspects. ## conclusions and further work we presented the result of applying a fine-grained linguistically motivated test suite on the outputs of 36 state-of-the-art machine translation systems, as submitted in the sixth conference on machine translation. we presented detailed accuracies of translations of 18 german to english as well as 18 english to german mt systems based on more than 3,000 test items each, organized in various linguistic categories and fine-grained phenomena. additionally, we drew a comparison to previous years' evaluations. in both language directions, the systems achieve good accuracies in most phenomena or categories and there is some advancement as compared to last year, although there is space for about 10% improvement on the average accuracy. a few phenomena still suffer considerably, such as the idioms, the modal pluperfect and the german resultative predicates, although there is notable improvement as compared to previous years. as discussed, the very high accuracies for some categories or phenomena raise the question whether the difficulty of the respective test items should be increased. in future work, we plan to investigate this by constructing more test items. further work includes the development of similar test suites for other language pairs. ondřej bojar, jiří mírovský, kateřina rysová, and magdaléna rysová. 2018. evald reference-less discourse evaluation for wmt18. in proceedings of the third conference on machine translation, pages 545-549, belgium, brussels. association for computational linguistics. aljoscha burchardt, vivien macketanz, jon dehdari, georg heigold, jan-thorsten peter, and philip williams. 2017. a linguistic evaluation of rule-based, phrase-based, and neural mt engines. the prague bulletin of mathematical linguistics, 108(1):159-170. franck burlot, yves scherrer, vinit ravishankar, ondřej bojar, stig-arne grönroos, maarit koponen, tommi nieminen, and françois yvon. 2018. the wmt'18 morpheval test suites for english-czech, english-german, english-finnish and turkish-english. in proceedings of the third conference on machine translation, pages 550-564, belgium, brussels. association for computational linguistics. silvie cinkova and ondřej bojar. 2018. testsuite on czech-english grammatical contrasts. in proceedings of the third conference on machine translation, pages 565-575, belgium, brussels. association for computational linguistics. liane guillou and christian hardmeier. 2016. protest: a test suite for evaluating pronouns in machine translation. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 636-643, portorož, slovenia. european language resources association (elra). liane guillou, christian hardmeier, ekaterina lapshinova-koltunski, and sharid loáiciga. 2018. a pronoun test suite evaluation of the english-german mt systems at wmt 2018. in proceedings of the third conference on machine translation, pages 576-583, belgium, brussels. association for computational linguistics. ulrich heid and elke hildenbrand. 1991. some practical experience with the use of test suites for the evaluation of systran. in the proceedings of the evaluators' forum, les rasses. citeseer. pierre isabelle, colin cherry, and george foster. 2017a. a challenge set approach to evaluating machine translation. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2486-2496, copenhagen, denmark. association for computational linguistics. pierre isabelle, colin cherry, and george foster. 2017b. a challenge set approach to evaluating machine translation. margaret king and kirsten falkedal. 1990. using test suites in evaluation of machine translation systems. in proceedings of the 13th conference on computational linguistics, volume 2, pages 211-216, morristown, nj, usa. association for computational linguistics. tom kocmi, tomasz limisiewicz, and gabriel stanovsky. 2020. gender coreference and bias evaluation at wmt 2020. in proceedings of the fifth conference on machine translation, pages 357-364, online. association for computational linguistics. maja popović. 2019. evaluating conjunction disambiguation on english-to-german and french-togerman wmt 2019 translation hypotheses. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 464-469, florence, italy. association for computational linguistics. alessandro raganato, yves scherrer, and jörg tiedemann. 2019. the mucow test suite at wmt 2019: automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 470-480, florence, italy. association for computational linguistics. annette rios, mathias müller, and rico sennrich. 2018. the word sense disambiguation test suite at wmt18. in proceedings of the third conference on machine translation, pages 594-602, belgium, brussels. association for computational linguistics. kateřina rysová, magdaléna rysová, tomáš musil, lucie poláková, and ondřej bojar. 2019. a test suite and manual evaluation of document-level nmt at wmt19. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 455-463, florence, italy. association for computational linguistics. yves scherrer, alessandro raganato, and jörg tiedemann. 2020. the mucow word sense disambiguation test suite at wmt 2020. in proceedings of the fifth conference on machine translation, pages 365-370, online. association for computational linguistics. tereza vojtěchová, michal novák, miloš klouček, and ondřej bojar. 2019. sao wmt19 test suite: machine translation of audit reports. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 481-493, florence, italy. association for computational linguistics. andrew way. 1991. developer-oriented evaluation of mt systems. in proceedings of the evaluators' forum, pages 237-244, les rasses, vaud, switzerland. issco. vilém zouhar, tereza vojtěchová, and ondřej bojar. 2020. wmt20 document-level markable error exploration. in proceedings of the fifth conference on machine translation, pages 371-380, online. association for computational linguistics. appendix category count onl-w faceb onl-b volct onl-a smu onl-g huawe borde nemo uedin water p3ai icl onl-y manif happy avg ambiguity 74 87.8 90.5 86.5 86.5 81.1 83.8 85.1 89.2 83.8 83.8 83.8 75.7 79.7 86.5 82.4 81.1 60.8 82.8 composition 43 97.7 97.7 100.0 100.0 97.7 95.3 97.7 95.3 95.3 93.0 97.7 97.7 97.7 95.3 93.0 93.0 74.4 95.2 coordination & ellipsis 57 89.5 89.5 89.5 89.5 87.7 86.0 86.0 87.7 89.5 87.7 87.7 86.0 87.7 77.2 87.7 89.5 80.7 87.0 false friends 36 86.1 80.6 75.0 75.0 83.3 83.3 80.6 63.9 77.8 72.2 66.7 80.6 80.6 72.2 75.0 69.4 63.9 75.7 function word 40 92.5 92.5 92.5 92.5 90.0 85.0 95.0 92.5 85.0 92.5 92.5 92.5 85.0 87.5 90.0 72.5 80.0 88.8 ldd & interrogatives 103 91.3 91.3 91.3 91.3 91.3 91.3 90.3 93.2 90.3 91.3 89.3 92.2 89.3 91.3 88.3 57.3 74.8 87.9 mwe 66 90.9 86.4 83.3 83.3 86.4 86.4 84.8 86.4 86.4 86.4 83.3 80.3 84.8 86.4 81.8 84.8 69.7 84.2 named entity & terminology 71 95.8 94.4 93.0 93.0 94.4 93.0 94.4 95.8 91.5