Fine-grained linguistic evaluation for state-of-the-art Machine Translation (2020)

Abstract: This paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Machine Translation (WMT20). The analysis covers 107 phenomena organized in 14 categories based on about 5,500 test items, including a manual annotation effort of 45 person hours. Two systems (Tohoku and Huoshan) appear to have significantly better test suite accuracy than the others, although the best system of WMT20 is not significantly better than the one from WMT19 in a macro-average. Additionally, we identify some linguistic phenomena where all systems suffer (such as idioms, resultative predicates and pluperfect), but we are also able to identify particular weaknesses for individual systems (such as quotation marks, lexical ambiguity and sluicing). Most of the systems of WMT19 which submitted new versions this year show improvements.

## introduction fine-grained evaluation has recently had increasing interest on several natural language processing (nlp) tasks. focusing on particular issues gives the possibility to analyse the automatic output in ways that cannot be seen by generic metrics. this is of particular importance in the era of deep learning, which has led to high performances and differences that are relatively difficult to distinguish. additionally, detailed evaluation can provide indications for the improvement of the systems and the data collection, or allow focusing on phenomena of the long tail that might be of particular interest for certain cases (e.g. social biases; @xcite . the most common method for fine-grained or focused evaluation are the test suites (also known as challenge sets or benchmarks; @xcite . these are test sets engineered in a particular way, so that they can test the performance of nlp tasks on concrete issues @xcite . this paper is presenting the use of such a test suite for the evaluation of the 11 @xmath0; @xcite . the evaluation applies the dfki test suite on german-english, via 5,514 test items which cover 107 linguistically motivated phenomena organized in 14 categories. after a reference in related work (section 2), we explain shortly the structure of the test suite (section 3) and present the results (section 4) and the conclusions (section 5). ## related work the use of test suites was introduced along with the early steps of mt in the 1990's @xcite @xcite . with the emergence of deep learning, recent works re-introduced test suites that focus on the evaluation of particular linguistic phenomena (e.g. pronoun translation; @xcite or more generic test suites that aim at comparing different mt technologies @xcite and quality estimation methods @xcite . the test suite track of the conference of machine translation has already taken place two years in a row, allowing the presentation of several test suites, focusing on various linguistic phenomena and supporting different language directions. these include work in grammatical contrasts @xcite , discourse @xcite , morphology @xcite , pronouns @xcite and word sense disambiguation @xcite . when compared to the vast majority of the previous test suites, the one presented here is the only one lexical ambiguity er las gerne novellen. he liked to read novels. fail he liked to read novellas. pass phrasal verb warum starben die dinosaurier aus? why did the dinosaurs die? fail why did the dinosaurs die out? pass why did the dinosaurs become extinct? pass ditransitive perfect ich habe tim einen kuchen gebacken. i have baked a cake. fail i baked tim a cake. ## method the test suite is a test set that has been devised manually with the aim to allow testing the mt output for several linguistic phenomena. the entire test suite consists of subsets that test one particular phenomenon each, through several test items. each test item of the test suite consists of a source sentence and a set of correct and/or incorrect mt outputs. at the evaluation time, the test items are given as input to the mt systems and it is tested on whether the respective mt output consists a correct translation. by observing the amount of the test items that are translated correctly, one can calculate the performance of the mt systems regarding the respective phenomenon. the evaluation presented in this paper is based on the dfki test suite for mt on german to english, which has been presented in @xcite and applied extensively in the wmt shared task of 2018 @xcite and 2019 @xcite . the current version includes 5,560 test items in order to control 107 phenomena organised in 14 categories. some sample test items can be seen in ## results the accuracy of each system per linguistic category is briefly shown in very high scores do not necessarily mean that the mt of the respective grammatical phenomenon has been solved, but rather that the current test items of the test suite (which was engineered with the emergence of the first neural mt systems in 2017) are unable to expose difficulties of the systems. the artificial nature of the test suite and the variable number of test items per category and phenomenon should also be taken in consideration when doing comparisons between categories and phenomena. ## conclusions and further work in this paper we present the results of the application of the dfki test suite in the output of the stateof-the-art mt systems participating in the shared task of the fifth conference of machine translation (wmt20). based on about 5,500 test items, we present detailed accuracies regarding 107 phenomena organized in 14 categories. additionally, the evolution of systems submitted also in previous years is observed. the best system of this year is not significantly better than the one from 2019 in a macro-average, but one can see significant improvement from two years ago. the systems that seem to have the best accuracies are tohoku and volcantrans. the phenomena that most systems face difficulties are again this year the idioms, the resultative predicates and some moods of the pluperfect, whereas some systems still have issues with quotation marks and lexical ambiguity. as discussed previously, the high accuracies achieved for particular phenomena or categories raise questions on whether these phenomena are getting solved, or whether the test suite (which was originally built to challenge the systems from 2017) should raise the difficulty by including more test items. in further work, we would like to be able to associate the performance on specific phenomena with decisions related to decisions during the development of the systems, once there is enough information about this process for all systems. +14.6 +5.3 +10.7 +10.0 +1.2 named entitiy & terminology +3.1 -3.2 -1.5 +7.8 +15.6 +7.8 +3.1 -2.3 +9.0 negation -5.0 +5.0 +4 +10.0 -10.0 +10.0 non-verbal agreement +6.9 +22.4 +10.4 +10.4 +10.4 +6.9 +9.8 punctuation -21.9 +25.5 -3.6 -7.3 +3.7 +16.4 +21.8 +9.1 +30.0 +3.4 subordination -11.3 +10.3 +1.0 +7.2 +6.2 +5.1 +4.1 -2.1 +7.2 +3.8 +1.2 verb tense/aspect/mood +11.9 -6.3 +0.2 +2.0 +19.8 +13.9 -0.2 +5.6 -1.1 0.8 5.8 verb valency +1.5 +9.0 +13.5 +11.9 +6.0 +11.9 +3.0 +13.4 9.2 -1.2 micro-avg +9.9 -4.4 +0.2 +2.4 +18.5 +13.1 +1.5 +5.6 +0.6 +2.0 +4.8 macro-avg +0.1 +4.0 +0.4 +3.5 +12.7 +8.9 +9.3 +6.0 +7.6 +7.4 +0.6