Multilingual Machine Translation Systems from M icrosoft for WMT 21 Shared Task (2021)

Abstract: This report describes Microsoft’s machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. We participated in all three evaluation tracks including Large Track and two Small Tracks where the former one is unconstrained and the latter two are fully constrained. Our model submissions to the shared task were initialized with DeltaLM, a generic pre-trained multilingual encoder-decoder model, and fine-tuned correspondingly with the vast collected parallel data and allowed data sources according to track settings, together with applying progressive learning and iterative back-translation approaches to further improve the performance. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric.

## introduction recently, multilingual neural machine translation has attracted lots of attention because it enables one model to translate between multiple languages @xcite @xcite @xcite . to improve the performance of the multilingual translation models, there are various approaches on the training methods @xcite , the model structures @xcite @xcite , and the data augmentation @xcite . m2m @xcite leverages the large-scale data mined from the web data and explore the strategies to scale the model size and train the model effectively. meanwhile, the multilingual pre-trained language models have proven beneficial for the multilingual machine translation models. mbart @xcite pre-trains a multilingual model with the multilingual denoising objective to improve the multilingual machine translation. in this work, we explore the effects of different advanced approaches for multilingual machine translation models, especially on the large-scale dataset. we first explore the way to leverage the pre-trained language models that have been trained with large-scale monolingual data. we use the public available deltalm-large checkpoint to initialize the model. deltalm @xcite is a multilingual pre-trained encoder-decoder model, which has been proven useful for multilingual machine translation. we further explore the training methods and the data augmentation to improve the model. for efficient training, we apply progressive learning @xcite @xcite to our model that continue-trains a shallow model into a deep model. specifically, we first train a model with 24 encoder layers, and then continuetrain it by adding 12 layers on the top of the encoder. as for the data augmentation, we implement iterative back-translation @xcite that back-translates the data for multiple rounds. due to the limits of time and gpu memories of the shared task, we do not explore other approaches like mixture-of-experts (moe) and model ensemble. we participated in all three tracks including large track, small track #1, and small track #2. our final submissions are fine-tuned from deltalm with the allowed data sources according to the track settings, followed by progressive learning and iterative back-translation. the submissions on three tracks all rank first in terms of the automatic evaluation metric. small track #1 we use the constrained monolingual and bilingual data of 6 languages (croatian, hungarian, estonian, serbian, macedonian, and english) provided by the shared task. according to the statistics, the bitext data contains 273m sentence pairs of all translation directions. inspired by the previous work, we leverage the multilingual iterative back-translation method with one single multilingual model to generate parallel pseudo data. for @xmath0, we generate the back-translation data of 390m sentence pairs. as for @xmath1, we generate the dual-pseudo data of 1.18b sentence pairs, where x and y stand for any two non-english languages. ## large-scale data augmentation in this section, we introduce details about how to perform the iterative back-translation method @xcite to augment data. we use different models for data augmentation according to different tracks. for the small tracks, the multilingual models were trained over the constrained data sets to generate data. for the large track, we leverage the m2m model @xcite , the intermediate multilingual mt models, and in-house mt models to generate different language pairs' data respectively, so as to play their respective advantages to enhance the data generation quality. in practice, both the monolingual and bilingual corpora are effectively utilized in three ways: 1) for the back-translation data of @xmath2, we used the best model to generated x data accordingly by back-translating monolingual english wikipedia data; 2) for the dual-pseudo data of @xmath3, they are generated by back-translating the same english text to x and y respectively. alternatively, when the monolingual data of either x or y is enough, we also directly perform back-translation between x and y to obtain pseudo parallel data; 3) we try to augment existing bilingual corpora with the third language. given the bilingual corpus (x 1 , y 1 ), we generate pseudo parallel corpus of (x 1 , y 2 ) and (x 2 , y 1 ) by back-translating x 1 to x 2 and y 1 to y 2 , where x 2 and y 2 are non-english languages. ## preprocessing filtering to enhance the model performance, we remove the noisy sentence pairs with the incorrect language identification or character encoding. more specifically, we remove the sentences longer than 1024 words and truncate the sentence to 512 tokens. we also construct three corpora after tokenization with different length ratio limitations, i.e. {1.5, 2.0, 2.5, 3.0}, between the source and the target sentence. our multilingual model is first trained on the entire noisy data set and then continually tuned on cleaner data with descending length ratio, where the number of training directions is also gradually reduced by removing noisy language pairs. therefore, we can progressively fine-tune the multilingual model in an efficient way (noisy @xmath4. besides, to clean the backtranslation corpora, we remove the sentences containing unknown tokens ([unk]). regarding the language sr (serbian), those sentences comprised of latin characters in training data were also discarded since we found that the validation sets use cyrillic script for this language instead. tokenization after data filtering, we use the sen-tencepiece (kudo and richardson, 2018) to tokenize all raw training, validation, and test data sets, where the sentencepiece model is consistent with the one used for deltalm @xcite . we shuffled the whole training dataset before launching the training of multilingual models. the input sentence is prefixed with the language tag to indicate the translation direction. ## evaluation results following the previous work @xcite , we use the dev and the devtest of the flores-101 benchmark as our validation set and test set respectively. during the inference, the beam search strategy is performed with a beam size of 4 for the target sentence generation. we set the length penalty as 1.0 by default. the last n checkpoints (@xmath5, 5, 10, 15, 20}) are averaged for evaluation and we select the best checkpoint based on the performance on the validation set. we report the sentencepiece-based bleu using spbleu foot_1 . ## submissions considering the trade-off between the decoding time and the performance, we submit the model (24 encoder layers and 12 decoder layers) with the hybrid strategy to both the large track and small track #2, while the deep model (36 encoder layers and 12 decoder layers) with the direct translation is submitted to small track #1. ## conclusion this paper describes microsoft's submission to the large-scale multilingual machine translation of the wmt21 shared task. our multilingual translation model achieves substantial improvement over the baseline systems by fine-tuning the pre-trained language model deltalm. we further enhance the model performance with the progressive learning and the iterative back-translation methods. as a result, our submitted systems get the top evaluation results on three tracks, including large track, small track #1, and small track #2. @xcite goswami, and chao chen. 2021b. learning with feature-dependent label noise: a progressive approach. in iclr 2021. baohang zhou, xiangrui cai, ying zhang, and xiaojie yuan. 2021. an end-to-end progressive multi-task learning framework for medical named entity recognition and normalization. in acl 2021, pages 6214-6224. the language x in the i-th row and language y in the j-th column denotes the translation direction from the language x to language y. for example, the cell of the 1-th row (af) and the 2-th column (am) represents the result of the translation direction @xmath6. the table shows the results of all translation directions of 102 languages. the language x in the i-th row and language y in the j-th column denotes the translation direction from the language x to language y. for example, the cell of the 1-th row (af) and the 2-th column (am) represents the result of the translation direction @xmath7. the table shows the results of all translation directions of 102 languages.