Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions (2022)

Abstract: This document describes a fine-grained linguistically motivated analysis of 29 machine translation systems submitted at the Shared Task of the 7th Conference of Machine Translation (WMT22). This submission expands the test suite work of previous years by adding the language direction of English–Russian. As a result, evaluation takes place for the language directions of German–English, English–German, and English–Russian. We find that the German–English systems suffer in translating idioms, some tenses of modal verbs, and resultative predicates, the English–German ones in idioms, transitive-past progressive, and middle voice, whereas the English–Russian ones in pseudogapping and idioms.

## introduction neural machine translation has seen enormous progress and reached a quality that is helpful for many everyday use cases. however, several methods for evaluating mt suggest that there is still plenty of room for improvement. an evaluation method for revealing the translation flaws in a more structured way refers to the use of test suites or challenge sets. contrary to the classical evaluation, where test sets are drawn from random everyday texts, test suites consist of manually devised or selected sentences that focus on testing the ability of the mt systems to translate a particular phenomenon. here, we are presenting test suite results while analyzing the state-of-the-art systems with regard to many linguistically-motivated phenomena. the test suites foot_0 were applied to the mt systems submitted at the 7th conference of machine translation (wmt22) for the language directions german-english, english-german, and english-russian. the test suites for the first two language directions have also been showcased during the previous years, whereas english-russian is published for the first time. this paper is structured as follows: section 2 goes through related papers, whereas section 3 explains how the test suite was created and applied. section 4 outlines the setup of this year's experiment, whose results are detailed in section 5. section 6 concludes the paper with an outlook to future research. ## related work the first test suites were introduced as early as the first mt systems in the 1990s @xcite @xcite . recent years saw the rise of deep learning and the drastic improvement of the quality of mt outputs, which has led to the current revival of test suites. most of these test suites, however, focus on evaluating specific linguistic phenomena, e.g., @xcite , or on the comparison of different mt technologies @xcite , and quality estimation methods @xcite . over the last few years, several test suites for multiple language directions have emerged as a part of the conference on machine translation test suite track. these test suites, however, focus on one or a few different phenomena, including the works of @xcite . our test suite, on the other hand, performs a systematic evaluation of more than one hundred phenomena per language direction @xcite . similar to our work, the test suite approach and human evaluation are also used to evaluate mt quality metrics @xcite . ## method we have created a large-scale test suite with the goal of testing and comparing the performance of mt systems. currently, the test suite covers four different language pairs. we will present three in this paper: german to english, english to german, and english to russian (the fourth language pair being portuguese to english). the test suite is based on a number of linguistic categories which are in turn divided into more fine-grained linguistic phenomena. the categories and phenomena are language-specific; however, there is a significant overlap between many of the categories and phenomena across the different language pairs. each linguistic phenomenon in the test suite is represented by multiple test sentences. all categories, phenomena, and test sentences are the result of extensive research and knowledge of the syntax and morphology of the languages under inspection. the categories and phenomena do not follow a specific linguistic theory, but they were created by linguistic experts who are native speakers or highly proficient speakers of the languages. furthermore, the set of categories and phenomena was reviewed internally by linguists and experienced translators to achieve objectivity in the classification. the number of test sentences, categories, and phenomena for each language pair can be found in in order to allow for a semi-automatic evaluation of the test sentences, we have created a set of rules which determine whether a test sentence is translated correctly or incorrectly. the rules consist of hand-crafted regular expressions and fixed strings of translation outputs. they can be applied with the help of an internal evaluation tool @xcite . the workflow of the preparation and application of the test suite is depicted in figure 1 . ## experiment setup in this paper, we present the evaluation of 29 systems with our test suite. the systems are part of the news translation task of the seventh conference on machine translation (wmt22). the systems cover three different language pairs: nine systems for german-english, nine systems for english-german, and 11 systems for english-russian. this year is the second time that the english-german systems are being evaluated and the first time that the english-russian systems are being evaluated with our test suite. every year, manual work is involved upon receiving the system translations as there are usually a number of translation outputs that are not yet covered by the existing rules in the database (the warnings). this year, there were on average 7.8 % of warnings for german-english, 9.7 % for english-german, and 20,6 % for english-russian. it is not surprising that the english-russian test set had a comparably bigger amount of warnings as this was the first time the test set was evaluated and therefore, the database of evaluation rules for this language pair was still rather small. it was also expected that english-german would have a higher amount of warnings than german-english as the german-english test set has the largest rules database since this language pair has been evaluated five years in a row. two annotators with extensive linguistic knowledge of the three languages under investigation conducted the manual evaluation of the warnings. no inter-annotator agreement was calculated; however, problematic cases were discussed with several linguistic experts to exclude subjectivity. the manual evaluation took around four weeks and involved around 50 person-hours. after the manual evaluation, there were on average 1.2 % of warnings left for german-english, 3.2 % for english-german, and 0.7 % for english-russian. as mentioned above, test sentences with at least one warning by one system were excluded from the analysis to achieve a fair comparison between the systems under inspection. as a result, our analysis was conducted on 5049 (91 %) test sentences for german-english, 3723 (83 %) test sentences for english-german, and 300 (97 %) test sentences for english-russian. ## results all result tables can be found in the appendix. ## conclusions and outlook this paper presents a fine-grained, linguistically motivated test suite to evaluate machine translation outputs. the test suite was applied to evaluate and compare the outputs of 29 machine translation systems in three different language pairs: german-english, english-german, and (for the first time) english-russian. altogether, almost 7,000 test sen-tences, structured in various linguistic categories and phenomena, were evaluated altogether across the three language pairs. additionally, a comparison to the evaluation in previous years for the language pairs german-english and english-german was drawn. the average accuracy for most categories and phenomena is relatively high for german-english and english-german, with only about 5 % room for improvement. as compared to last year, this is an improvement of around 5 %. for english-russian, the average accuracy is not as high, yet still around 80 %.