The LMU M unich System for the WMT 2021 Large-Scale Multilingual Machine Translation Shared Task (2021)

Abstract: This paper describes the submission of LMU Munich to the WMT 2021 multilingual machine translation task for small track #1, which studies translation between 6 languages (Croatian, Hungarian, Estonian, Serbian, Macedonian, English) in 30 directions. We investigate the extent to which bilingual translation systems can influence multilingual translation systems. More specifically, we trained 30 bilingual translation systems, covering all language pairs, and used data augmentation technologies such as back-translation and knowledge distillation to improve the multilingual translation systems. Our best translation system scores 5 to 6 BLEU higher than a strong baseline system provided by the organizers. As seen in the dynalab leaderboard, our submission is the only fully constrained submission that uses only the corpus provided by the organizers and does not use any pre-trained models.

## introduction neural machine translation (nmt) @xcite has been shown to be effective with rich and in-domain bilingual parallel corpora. although the nmt model obtained promising performances for high resource language pairs, it is hardly feasible to train translation models for all directions of the language pairs since the training progress is time-and resource-consuming. recent work has shown the effectiveness of multilingual neural machine translation (mnmt), which aims to handle the translation from multiple source languages into multiple target languages with a single unified model @xcite @xcite @xcite . the mnmt model dramatically reduces training and serving costs. it is faster to train a mnmt model than to train bilingual models for all language pairs in both directions, and mnmt signif-icantly simplifies deployment in production systems @xcite . further, parameter sharing across different languages encourages knowledge transfer, which improves low-resource translation directions and potentially enables zero-shot translation (i.e., direct translation of a language pair not seen during training) @xcite @xcite . we participate in the wmt 2021 multilingual machine translation task for small track #1. the task aims to train a multilingual model to translate 5 central/east european languages (croatian, hungarian, estonian, serbian, macedonian) and english in 30 directions. the multilingual systems presented in this paper are based on the standard paradigm of mnmt proposed by @xcite , which prefixes the source sentence with a special token to indicate the desired target language and does not change the target sentence at all. language tags are typically used in mnmt to identify the language to translate to. a language code, in the form of a two-or three-character identification such as en for english, is the main constituent of a language tag and is provided by the iso 639 standard foot_0 (international organization for standardization, nd). following iso 639 standard, en indicates english, mk indicates macedonian, sr indicates serbian, et indicates estonian, hr indicates croatian and hu indicates hungarian in this paper. compared with the other three submissions to the task, our submissions have the following advantages: whole select no filter 387m 71m + punctuation filter 384m 71m + deduplicated filter 304m 44m + langid filter 302m 43m + length filter 274m 42m the training data provided by the organizers come from the public available opus repository (tiedemann, 2012), which contains data of mixed quality from a variety of domains (wmt-news, ted, qed, opensubtitles, etc.). in addition to the bilingual parallel corpora, in-domain wikipedia monolingual data for each language is provided. the validation and test sets are obtained from the flores 101 evaluation benchmark @xcite , which consists of 3001 sentences extracted from english wikipedia covering a variety of different topics and domains. see ## method description we first trained bilingual translation models with 30 directions for all language pairs. next, we trained a single multilingual model that can translate all language pairs. finally, we use back-translation and knowledge distillation technologies to further improve the performance of the multilingual translation system. the details of these components are outlined next. ## conclusions in this paper, we presented the lmu munich system for the wmt 2021 large-scale multilingual translation shared task for small track #1. the task evaluates translation between five central/eastern european languages and english, in total 30 translation directions. the system we submitted was fully constrained, using only the data provided by the organizers and not using any pre-trained model. the experiments show that back-translation and knowledge distillation techniques are effective for training multilingual machine translation systems.