Findings of the WMT 2021 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT (2021)

Abstract: We present the findings of the WMT2021 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT. Within the task, the community studied very low resource translation between German and Upper Sorbian, unsupervised translation between German and Lower Sorbian and low resource translation between Russian and Chuvash, all minority languages with active language communities working on preserving the languages, who are partners in the evaluation. Thanks to this, we were able to obtain most digital data available for these languages and offer them to the task participants. In total, six teams participated in the shared task. The paper discusses the background, presents the tasks and results, and discusses best practices for the future.

## introduction for some languages, machine translation (mt) reached such a high quality that allows a discussion of whether and under what circumstance human parity might have been reached @xcite . this is the case, however, for only a small minority of the world's language. for most of the 7k languages spoken in the world only very limited resources exist. the goal of the wmt shared task on unsupervised and very low resource mt is to promote research on methods for mt that alleviate such data sparsity in a real-world setup. a task on unsupervised mt was already held at wmt in 2018 @xcite and 2019 @xcite , where the lack of parallel data was simulated on high-resource language pairs: english-german in 2018 and german-czech in 2019. starting from last year, we cooperate with local communities working on preserving their languages. in cooperation with the sorbian insti-tute foot_0 and the witaj sprachzentrum foot_1 , we offered a shared task in translation between german and upper sorbian in low-resource and unsupervised tracks @xcite . for this year, we kept the low-resource track for upper sorbian and added unsupervised translation between german and lower sorbian. upper and lower sorbian are minority languages spoken in the east part of germany in the federal states of saxony and brandenburg. having only 30k and 7k native speakers, processing of the languages is an inherently low-resource problem, without any chance that the size of available resources would ever get close to the size of resources available for languages with millions of speakers. on the other hand, being western slavic languages, the sorbian languages can take advantage of existing resources for czech and polish. additionally, in cooperation with the chuvash language laboratory foot_2 , we added another lowresource task, translation between russian and chuvash. chuvash is a minority turkic language spoken by approximately one million people in the volga region in the southwest of russia. there is a larger amount of training data available for chuvash, but the language is rather isolated in the turkic language family, so unlike sorbian, it cannot benefit that much from the existence of closely related languages. five teams participated in the german-upper sorbian task, six teams in the german-lower sorbian task, and two teams in the russian-chuvash task. ## tasks and evaluation this year, there were three tasks for very low resource and unsupervised translation were: to make the submissions better comparable with each other, we only allowed using resources released for the task (see section 3) and resources for related languages commonly used in other wmt tasks. the use of large models pre-trained on large datasets was not allowed. by this decision, we wanted to motivate the participants to find better use of limited language resources. ## submitted systems six teams participated in the shared task, five teams in upper sorbian-german, slightly different five in lower sorbian-german, and two in the russian-chuvash direction. an overview of the systems is in noahnmt @xcite . noahnmt submitted their systems into the supervised tasks. the noahnmt submission is a standard transformer model equipped with our recent technique of dual transfer @xcite . compared to other systems, these submissions used a significantly larger amount of monolingual data. nrc-cnrc @xcite . the upper sorbian-german system is an ensemble of eight systems with 25k bpe vocabulary, incorporating transfer learning (from cs-de) with continued training, monolingual data filtering, backtranslation @xcite , bpe-dropout @xcite , and multilingual models. in the opposite direction, the submission is an ensemble of 7 systems. the lower sorbian-german and german-lower sorbian systems are ensembles of 2 and 4 systems, respectively, with 20k bpe vocabulary, incorporating transfer learning from hsb-de and de-hsb systems along with iterative backtranslation. iict-yverdon @xcite . the system used the transformer architecture with backtranslation of large german corpora and parentlanguage initialization using czech-german data. the final submission is an ensemble of different models with some changes in their training setups to maximize the diversity among the models. cfilt. the submitted systems cover four language pairs: german↔upper sorbian german↔lower sorbian. for de↔hsb, the system pre-trained using the mass objective @xcite and finetuned using iterative back-translation. final finetuning is performed using the provided parallel data for translation objective. for de↔dsb, no parallel data is provided in the task. the final de↔hsb model is used for initialization of the de↔dsb model, which is further trained using iterative back-translation, using the same vocabulary as used in the de↔hsb model. cl_rug @xcite . cl_rug's submission uses the mass model, focusing pretraining on 2 languages at a time, from least to most related to lower sorbian. the largest improvement comes from a novel method for initializing the lower sorbian word embeddings from upper sorbian, using a bilingual dictionary created in an unsupervised fashion. lmu munich @xcite . the lmu submissions for all tasks are transformer models first pre-trained on related languages and then finetuned on the low-resource languages. for the sorbian languages, the systems are pre-trained on german-czech translation. the system is finetuned using the authentic german-upper sorbian data, which is the starting point for four iterations of tagged back-translation. the unsupervised german-lower sorbian translation is trained by iterative backtranslation using the monolingual data only. the upper sorbian-german system is used to generate the first translation of lower sorbian. the russian-chuvash systems were pretrained on russian-kazakh translation and finetuned using the provided parallel data. ## results the results are presented in in german-upper sorbian translation, the best two system, nrc-cnrc and noahnmt reach very similar results although they use significantly different sizes of monolingual data for backtranslation. nrc-cnrc manage to compensate for the smaller data size by accumulating minor tricks including monolingual data selection @xcite , tagged backtranslation @xcite , bpe dropout @xcite , and language tags in multilingual training. lmu, which used data of a similar size to nrc-cnrc but did not use most of the further tricks, ranked below these two. in upper sorbian-german translation, all teams used german-czech parallel data for pre-training, except for cfilt who only used monolingual data for pre-training and scored 0 points in both directions. in the unsupervised german-lower sorbian task, cl_rug ranked on par with nrc-cnrc in translation into german (despite not using ensembling), but at third place in the opposite direction. this suggests that cl_rug's innovative vocabulary transfer method works better on the encoder side than on the decoder side. in the russian-chuvash translation, noah-nmt outperformed lmu munich by using larger datasets and a more advanced transfer learning technique. ## conclusions in wmt 2021 shard task on unsupervised and very low resource mt, we created realistic benchmarks for low-resource minority language which reflect the needs of the language communities trying to preserve their languages. in the task, we provided the participants with comprehensive resource for translation between german and upper and lower sorbian and for translation between russian and chuvash. we hope that this will increase the interest of the community in these languages. the six teams that participated in the task used state-of-the-art mt techniques to develop high quality systems. the main technical takeaway from the results are that pre-training on parallel data in related language is important and that carefully applying known tricks can to a large extent com-pensate for using smaller datasets.