The MUCOW word sense disambiguation test suite at WMT 2020 (2020)

Abstract: This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT systems to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. MUCOW is created automatically using existing resources, and the evaluation process is also entirely automated. We evaluate all participating systems of the language pairs English -> Czech, English -> German, and English -> Russian and compare the results with those obtained at WMT 2019. While current NMT systems are fairly good at handling ambiguous source words, we could not identify any substantial progress - at least to the extent that it is measurable by the MUCOW method - in that area over the last year.

## introduction at wmt 2019, we introduced the mucow (multilingual contrastive word sense disambiguation) test suite @xcite and evaluated the news task submissions of nine translation directions with it. 1 we observed that systems generally performed quite well on word sense disambiguation, but found a big gap between indomain and out-of-domain disambiguation performance for some translation directions, in particular with constrained systems. for wmt 2020, we reuse the same test suite for the same language pairs. this gives us the opportunity to measure the advancement of machine translation within a year. we expect the larger training data sets and the model improvements to have a small but positive impact on translation quality in general, and word sense disambiguation performance in particular. 1 the mucow test suite is available at http:// github.com/helsinki-nlp/mucow . we evaluated the systems participating in the wmt 2019 news translation task with mucow for the language pairs @xmath0, english ↔ german, english ↔ finnish, english ↔ russian, and english ↔ lithuanian. a substantial amount of mucow sentences and senses come from the opensubtitles2018 corpus, but most systems participating at wmt are tuned towards the news domain and therefore are not expected to handle lexical choices of colloquial speech reliably. therefore, we distinguished between in-domain and out-of-domain synsets: a synset is considered out-of-domain if more than half of its example sentences come from movie subtitles. ## evaluation and results the source language sentences were sent to the wmt participants as part of the test set, and we received the translations in the target language for evaluation. we then checked if any of the correct or incorrect target words listed in the metadata file could be identified in the translation output. although the sentences were selected to contain the uninflected base forms both in the source and target languages, we could not assume that all translation systems would output base forms. hence, if neither correct nor incorrect target words could be identified in the tokenized translations, we lemmatized them and searched the target words again in the lemmatized version. 2 depending on the morphological properties of the target language, lemmatization substantially increased the coverage (see the results are shown in tables 4 to 7 , with wmt 2019 and 2020 submissions side-by-side. for all four examined translation directions, the best 2019 results were beaten in 2020. however, one of the best-performing systems in 2019, facebook fair, did not participate in 2020. the facebook fair system is characterized by high precision rates, whereas the winning 2020 systems (such as tohoku-aip-ntt or online-g) benefit from higher recall. this shift suggests that the denominator of the precision computation comes closer to the one of the recall computation, or in other words that the translations themselves become more accurate. further analysis will be required to substantiate this claim. interesting year-over-year comparisons can be observed for the online-g system: it produces almost identical results in both years for english-german and english-russian, but shows substantial improvements for the german-english direction. 3 examples that contained both correct and incorrect target words were counted as incorrect. the overall result distributions show a slight upward trend in wsd performance for english-german and german-english, but less so for english-czech and english-russian. since the participating systems differed over the years, it is of course difficult to draw any reliable conclusions. for most language pairs, the in-domain and out-of-domain synsets produce similar rankings. just like in 2019, english-czech is an exception, where -contrarily to all expectations -an online system shows the best in-domain performance and a research system the best out-of-domain performance. ## conclusion in this paper, we report our participation with the mucow test suite at the wmt 2020 news translation task. mucow is an automatically built wsd test suite for machine translation that relies on large parallel corpora, the multilingual lexical resource babelnet and language-independent synset embeddings. we find that state-of-the-art nmt systems are fairly good at handling ambiguous source words, but that no substantial progress -at least to the extent that it is measurable by the mucow method -has been made in that area over the last year. among the top-performing systems, we observe a shift from high precision to high recall, hinting at general improvements in translation quality. it will therefore be particularly instructive to see how well the wsd test suite results correlate with human evaluation scores and with recently proposed evaluation metrics that are based on semantic representations of the translations @xcite .