Findings of the 2020 Conference on Machine Translation ( WMT 20) (2020)

Abstract: This paper presents the results of the news translation task and the similar language translation task, both organised alongside the Conference on Machine Translation (WMT) 2020. In the news task, participants were asked to build machine translation systems for any of 11 language pairs, to be evaluated on test sets consisting mainly of news stories. The task was also opened up to additional test suites to probe specific aspects of translation. In the similar language translation task, participants built machine translation systems for translating between closely related pairs of languages.

## introduction the fifth conference on machine translation (wmt20) 1 was held online with emnlp 2020 and hosted a number of shared tasks on various aspects of machine translation. this conference built on 14 previous editions of wmt as workshops and conferences @xcite @xcite @xcite @xcite @xcite @xcite @xcite . this year we conducted several official tasks. we report in this paper on the news and similar translation tasks. additional shared tasks are described in separate papers in these proceedings: the human evaluation (section 3) involves asking human judges to score sentences output by anonymized systems. we obtained large numbers of assessments from researchers who contributed evaluations proportional to the number of tasks they entered. in addition, we used mechanical turk to collect further evaluations, as well as a pool of linguists. this year, the official manual evaluation metric is again based on judgments of adequacy on a 100-point scale, a method (known as "direct assessment") that we explored in the previous years with convincing results in terms of the trade-off between annotation effort and reliable distinctions between systems. the primary objectives of wmt are to evaluate the state of the art in machine translation, to disseminate common test sets and public training data with published performance numbers, and to refine evaluation and estimation methodologies for machine translation. as before, all of the data, translations, and collected human judgments are publicly available. 2 we hope these datasets serve as a valuable resource for research into datadriven machine translation, automatic evaluation, or prediction of translation quality. news translations are also available for interactive visualization and comparison of differences between systems at http://wmt.ufal.cz/ using mt-compareval @xcite . in order to gain further insight into the performance of individual mt systems, we organized a call for dedicated "test suites", each focusing on some particular aspect of translation quality. a brief overview of the test suites is provided in section 4. following the success of the first similar language translation (slt) task at wmt 2019 and the interest of the community in this topic @xcite , we organize a second iteration of the slt task at wmt 2020. the goal of the shared task is to evaluate the performance of state-of-the-art mt systems on translating between pairs of closely-related languages from the same language family. slt 2020 features five pairs of similar languages from three language families: indo-aryan (hindi and marathi), romance (catalan, spanish, and portuguese), and south-slavic (croatian, serbian, and slovene). translations were evaluated in both directions using three automatic metrics: bleu, ribes, and ter. results and main findings of the slt shared task are discussed in section 5. ## news translation task this recurring wmt task assesses the quality of mt on news domain text. as in the previous year, we included chinese, czech, german and russian (into and out of english) as well as french-german. new language pairs for this year were inuktitut, japanese, polish and tamil (to and from 2 http://statmt.org/wmt20/results.html english). we also included the two language pairs from the corpus filtering task (@xmath0, to give participants the opportunity to build and test mt systems using the large noisy corpora released for that task. ## human evaluation a human evaluation campaign is run each year to assess translation quality and to determine the official ranking of systems taking part in the news translation task. this section describes how data for the human evaluation is prepared, the process of collecting human assessments, and computation of the official results of the shared task. ## test suites anyone can propose their own test suite and take part, and we also try to solicit evaluation from past successful test suite teams to support some cross-year insight. each team in the test suites track provides source texts (and optionally references) for any language pair that is being evaluated by wmt news task. we shuffle these additional texts into the inputs of news task and ship them as inputs to mt system developers jointly with the regular news texts. the shuffling happens at the document or sentence level as agreed with the test suite authors. (shuffling at the level of sentences can lead to a very high number of documents in the final test set because each sentence is treated as a separate document.) mt system developers may decide to skip these documents based on their id but most of them process test suites along with the main news texts. after collecting the output translations from all wmt news task participants, test suites translations are made available back to the test suite authors for evaluation. test suite sentences do not go through the manual evaluation as described in section 3. as in the previous years, test suites are not limited to the news domain, so news task system may actually underperform on them. ## similar language translation most shared tasks at wmt (e.g. news, biomedical) have historically dealt with translating texts from and to english. in recent years, we observed a growing interest in training systems to translate between languages other than english. this includes a number of papers applying mt to translate between pairs of closely-related languages, national language varieties, and dialects of the same language @xcite @xcite @xcite . to address this topic, the first similar language translation (slt) shared task at wmt 2019 has been organized. it featured data from three pairs of closely-related languages from different language families: spanish -portuguese (romance languages), czech -polish (slavic languages), and hindi -nepali (indo-aryan languages). following the success of the first slt shared task at wmt 2019 and the interest of the community in this topic, we organize, for the second time at wmt, this shared task to evaluate the performance of state-of-the-art translation systems on translating between pairs of languages from the same language family. slt 2020 features five pairs of similar languages from three different language families: indo-aryan, romance, and south-slavic. translations were evaluated in both directions using automatic evaluation metrics presented in this section. ## conclusion this paper presented the results of wmt20 news translation and similar language translation shared tasks, as well as the extra test suites added to the news translation task. our main findings rank participating systems in their sentence-level and document-level translation quality, as assessed in a large-scale manual evaluation using the method of direct assessment (da). for out-of-english language pairs, da was modified so that the context of the whole document is available while judging individual sentences and assessors are allowed to return to any sentence judgement within the document. as in previous years, the effect of translationese (translating from a source which itself was produced in translation) was avoided except lowerresourced inuktitut↔english, pashto↔english, khmer↔english, and german↔french by creating reference translations always in the same direction as the mt systems are run. furthermore, 8 out-of-english language pairs would not need human reference for our evaluation at all because the assessors are evaluating translation candidates bilingually, comparing them to the source the need for cautious interpretation of the results is also strengthened by the fact that even in @xmath1, the result is not confirmed this year. furthermore and similarly to previous year, a test suite this year again suggests that some aspects of translation are not handled by current systems at all. this year all mt systems fall into the gender bias trap @xcite and they tend to make more severe errors than humans @xcite . the results of the task on similar language translation indicate that the performance when translating between pairs of closely-related languages is extremely varied across different language pairs. the best performing systems trained to translate between catalan and spanish, for example, obtained significantly higher results in both directions than those trained to translate between other language pairs in terms of bleu, ribes, and ter.