T en T rans Large-Scale Multilingual Machine Translation System for WMT 21 (2021)

Abstract: This paper describes TenTrans large-scale multilingual machine translation system for WMT 2021. We participate in the Small Track 2 in five South East Asian languages, thirty directions: Javanese, Indonesian, Malay, Tagalog, Tamil, English. We mainly utilized forward/back-translation, in-domain data selection, knowledge distillation, and gradual fine-tuning from the pre-trained model FLORES-101. We find that forward/back-translation significantly improves the translation results, data selection and gradual fine-tuning are particularly effective during adapting domain, while knowledge distillation brings slight performance improvement. Also, model averaging is used to further improve the translation performance based on these systems. Our final system achieves an average BLEU score of 28.89 across thirty directions on the test set.

## introduction we participate in the wmt 2021 large-scale multilingual machine translation task small track 2 in 6 languages: english, indonesian, javanese, malay, tamil, tagalog (briefly, en, id, jv, ms, ta, tl). any two of these languages translated into each other produces a total of 30 directions, including english↔indonesian, english↔javanese, english↔malay, english↔tamil, english↔ tagalog, indonesian↔javanese, indonesian↔malay, indonesian↔tamil, indonesian↔tagalog, javanese ↔malay, javanese↔tamil, javanese↔ tagalog, malay↔tamil, malay↔tagalog and tamil↔tagalog. to meet the requirements for data restrictions, our systems are all built with constrained data sets. for all systems, we adopt a universal encoder-decoder architecture that shares parameters across all languages @xcite . our systems are based on several techniques and approaches. we experiment with base and deeper transformer @xcite architectures to get reliable baselines, fine-tune the pre-training model flores-101 @xcite to further improve the baseline system. moreover, we generate pseudo bilingual sentences from the large-scale monolingual data, apply sequence level knowledge distillation @xcite on partial language pairs, and try a more effectively fine-tuning strategy to domain adaptation @xcite . particularly in the language pairs with inferior translations, we specifically improve their performance. all of these technologies have improved our systems, particularly data selection and gradual finetuning. we carefully rethought this strategy and found the main gain may come from in-domain knowledge adaptation. this paper was structured as follows: section 2 describes the data set. then, we present a detailed overview of our systems in section 3. the experiment settings and main results are shown in section 4. finally, we conclude our work in section 5. ## data prepration we use flores-101 sentencepiece (spm) foot_0 tokenizer model with 256k tokens to tokenize bitext and monolingual sentences foot_1 . since it is important to clean data strictly @xcite , we follow m2m-100 data preprocessing procedures foot_2 to filter bitext data. the rules are as follows: for monolingual data, we still employ those rules except the length ratio filter. see ## conclusion this paper introduced our tentrans submissions on wmt21 large-scale multilingual machine translation small task 2. our main exploration is using more diversified architectures and fine-tuning strategy, utilizing forward-translation and back translation and approaches including in-domain data selection, knowledge distillation, and gradual finetuning. we experimented with these methods and continuously improve our system performance. on the whole, all of our systems performed competitively and ranked 3rd on the leaderboard.