Findings of the 2021 Conference on Machine Translation ( WMT 21) (2021)

Abstract: This paper presents the results of the newstranslation task, the multilingual low-resourcetranslation for Indo-European languages, thetriangular translation task, and the automaticpost-editing task organised as part of the Con-ference on Machine Translation (WMT) 2021.In the news task, participants were asked tobuild machine translation systems for any of10 language pairs, to be evaluated on test setsconsisting mainly of news stories. The taskwas also opened up to additional test suites toprobe specific aspects of translation.

## introduction the sixth conference on machine translation (wmt21) foot_0 was held online with emnlp 2021 and hosted a number of shared tasks on various aspects of machine translation. this conference built on 15 previous editions of wmt as workshops and conferences @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . this year we conducted several official tasks. in this paper we report on the news task, the multilingual low-resource translation for indo-european languages task, the triangular translation task, and the automatic post-editing task. additional shared tasks are described in separate papers in these proceedings: the human evaluation (section 3) involves asking human judges to score sentences output by anonymized systems. we obtained large numbers of assessments from researchers who contributed evaluations proportional to the number of tasks they entered. we collected additional assessments from a pool of linguists, as well as crowd-workers. this year, the official manual evaluation metric is again based on judgments of adequacy on a 100point scale, a method (known as "direct assessment", da) that we explored in the previous years with convincing results in terms of the trade-off between annotation effort and reliable distinctions between systems. in addition, other golden standards with this year's systems were collected. the human-in-the-loop genie leaderboard @xcite conducted @xmath0.5). we refer the reader to @xcite for mqm scoring of @xmath1, @xmath2, and @xmath3. the primary objectives of wmt are to evaluate the state of the art in machine translation, to disseminate common test sets and public training data with published performance numbers, and to refine evaluation and estimation methodologies for machine translation. as before, all of the data, translations, and collected human judgments are publicly available. 2 we hope these datasets serve as a valuable resource for research into datadriven machine translation, automatic evaluation, or prediction of translation quality. news translations are also available for interactive visualization and comparison of differences between systems at http://wmt.ufal.cz/ using mt-compareval @xcite , and also on explain-aboard 3 @xcite . in order to gain further insight into the performance of individual mt systems, we again organized a call for dedicated "test suites". test suites are custom additions to the inputs. anyone can provide a test suite for any subset of news translation task languages and we ensure that the test suite is requested from all participating mt systems. the mt outputs are delivered back to test suite authors for evaluation, which can be manual, automatic or both, focusing on any possible aspect of the mt systems. this year, five test suites were acquired and translated by participating mt systems but only two were then analyzed in time for these proceedings: the goal of the similar language translation (slt) task (section 4) is to evaluate the perfor-mance of mt systems taking into account the similarity between pairs of closely-related languages from the same language family. following the interest of the community in this topic @xcite and the success of the past two editions of the slt task task at wmt 2019 and wmt 2020, we organize a third iteration of the task at wmt 2021. slt 2021 features a pair of similar dravidian languages, namely tamil -telugu, and multiple pairs of romance languages involving catalan, spanish, portuguese, and romanian in all possible combinations. a new track with french and two similar low-resource manding languages: bambara and maninka was also included to encourage participants to take advantage of the similarity between bambara and maninka and explore data augmentation techniques, a typical scenario of low-resource languages. finally, translations were evaluated in both directions using three automatic metrics: bleu, ribes, and ter. the primary goals of the triangular mt task (section 5) are to promote translation between non-english languages, to optimally mix direct and indirect parallel resources and exploit noisy web data sources to build an mt system. specifically, the task was russian to chinese machine translation, given parallel data comprising of direct (russian-chinese) and indirect (russian-english and english-chinese) sources. the submitted systems were evaluated on a (secret) mixed-genre test set, drawn from the web and curated manually for high-quality segment pairs. the multilingual low-resource translation for indo-european languages task (mllr, section 6) aims to investigate the best approaches to deal with multilingual translation. usually, multilingual translation is done with the help of a high-resourced language, e.g. english. in mllr, we evaluate translation quality for icelandic-norwegian bokmål-swedish (north-germanic) and catalan-italian-occitan-romanian (romance). higher resourced languages (danish, german, english, spanish, french and portuguese) are allowed for training but not evaluated. we focus on a specific domain: cultural heritage documents are extracted from europeana and wikipedia, a domain where named entities may also play a role in translation quality. the evaluation is done at language family level with a combination of automatic metrics (bleu, ter, chrf, bertscore and comet) and complemented by a manual evaluation on a subset of language pairs. the automatic post-editing (ape) task (section 7) focuses on another mt-related problem: the correction of machine-translated text generated by an unknown system. in continuity with last year, in this seventh iteration of the task at wmt we focused on two language pairs (english-german and english-chinese), using data drawn from english wikipedia articles and translated with neural mt systems. the evaluation was carried out both automatically -with ter and bleu respectively used as primary and secondary metric -and manually -with the same direct assessment method used for the news translation task. ## human evaluation a human evaluation campaign is run each year to assess translation quality and to determine the official ranking of systems taking part in the news translation task. this section describes how data for the human evaluation is prepared, the process of collecting human assessments, and computation of the official results of the shared task. ## similar language translation in this section we present the findings of the third slt shared task organized at wmt 2021. the task follows the success of the two past slt shared tasks organized at wmt 2019 and wmt 2020. slt 2021 is motivated by the growing interest of the community in translating between similar lan-guages, low-resource languages, dialects, and language varieties, and the challenges faced by stateof-the-art systems in these settings evidenced in recent studies @xcite @xcite . the main goal of the task is to evaluate the performance of state-of-the-art mt systems on translating between closely-related language pairs of languages from the same language family. past editions of the task @xcite featured language pairs such as spanish -portuguese, czech -polish, and hindi -nepali to name a few. this year's slt features multiple pairs of similar languages from the indo-aryan and romance family. finally, slt 2021 also features a track including french and two similar low-resource manding languages spoken in west africa, namely bambara and maninka, where participants were profor 95% confidence intervals are calculated by bootstrapping @xcite . grayed entries indicate unconstrained settings. ## triangular mt this section presents an overview of the triangular mt shared task. given a low-resource language pair (x/y), the bulk of previous mt work has pursued one of two strategies. ## automatic post editing this section presents the results of the 7 th round of the wmt task on mt automatic post-editing. the task consists in automatically correcting the output of a "black-box" machine translation system by learning from human-revised machinetranslated output. in continuity with last year, the challenge consisted of fixing the errors present in english wikipedia pages translated -into german and chinese -by state-of-the-art, not domainadapted neural mt (nmt) systems unknown to participants. despite a number of data downloads in line with the previous rounds, this year we observed an unexpected drop in participation: two teams participated in the english-german task, submitting two runs each, while the english-chinese task had no participants. most likely, this setback can be ascribed to the difficulty to handle the released test data, which are characterized by nmt output of very high quality. this is reflected by much higher baseline results compared to last year (18.05 ter / 71.07 bleu for en-de, 22.73 ter / 69.2 bleu for en-zh), which only one run was able to improve according to both the automatic metrics used (-0.77 for the primary ter metric and +0.48 for the secondary bleu metric). nevertheless, the outcomes of human evaluation still reveal the ability of ape systems to improve mt output quality: significant gains over the baseline are indeed observed for all the participating systems. ## conclusion the news translation task in 2021 covered 20 translation pairs, 14 of which had english on the source or target side and 6 were without english. direct assessment (da) was the main golden truth again, although the style varied across language pairs. into-english translation was evaluated against human reference translation, preserving the order of sentences in a document but not presenting the whole document at once (sr+dc). out-of-english and some of non-english pairs offered the full document context to the annotators and allowed them to revisit the scores assigned to individual segments (sr+fd), evaluating against the source. four non-english pairs used a simpler evaluation without any document context (sr-dc). for @xmath4, @xmath5, a contrastive da scoring was also tested, presenting individual sentences in pairs of candidate translations (contr:sr-dc), aimed at a more discerning pairwise comparisons. and finally, an alternative scoring style called genie was additionally applied to @xmath6. document context was found to be extremely important for evaluation of high-quality mt systems. the ranking of participating systems differs considerably between sr+fd and contr:sr-dc. in particular, human reference is scored well if full document context is available throughout the annotation but tends to be surpassed by top systems when sentences are evaluated in isolation. surprising effects were also observed when using these evaluation methods on different human translations. the triangular machine translation task encouraged participants to use all the parallel data provided (involving direct and indirect sources) to build a better machine translation system for the particular language pair and direction (russian-to-chinese). the participants explored several modeling choices and data augmentation strategies that would help practitioners when building ma-chine translation systems involving non-english language pairs. the multilingual low-resource translation task dealt with two indo-european language families: north germanic and romance. the best performing systems used multilingual supervised machine translation models enriched with backtranslated data and additional sentences from higherresourced languages in the same family. pivot translation via these high-resourced counter-parts and in-domain data selection was not beneficial for the final performance. the results of the task on automatic post-editing were highly influenced by the difficulty of this year's data, which can also explain a drop in participation (two teams, only in the english-german sub-task). in light of the very high quality of the translation to be automatically corrected, the very skewed ter distribution towards nearperfect translations and the very low repetition rate in the data, it comes as no surprise that only one run was able to outperform the strong donothing baseline with statistically significant improvements. nevertheless, human evaluation results reveal significant gains by all runs, attesting the difficulty to apply automatic evaluation procedures to ape and, on a positive note, the effectiveness of the proposed methods.