Back-translation for Large-Scale Multilingual Machine Translation (2021)

Abstract: This paper illustrates our approach to the shared task on large-scale multilingual machine translation in the sixth conference on machine translation (WMT-21). In this work, we aim to build a single multilingual translation system with a hypothesis that a universal cross-language representation leads to better multilingual translation performance. We extend the exploration of different back-translation methods from bilingual translation to multilingual translation. Better performance is obtained by the constrained sampling method, which is different from the finding of the bilingual translation. Besides, we also explore the effect of vocabularies and the amount of synthetic data. Surprisingly, the smaller size of vocabularies perform better, and the extensive monolingual English data offers a modest improvement. We submitted to both the small tasks and achieve the second place.

## introduction bilingual neural machine translation (nmt) systems have achieved decent performance with the help of transformer @xcite . one of the most exciting recent trends in nmt is training a single system on multiple languages at once @xcite @xcite . this is a powerful paradigm for two reasons: simplifying system development and deployment, and improving the translation quality on low-resource language pairs by transferring similar knowledge from highresource languages. this paper describes our experiments on the task of large-scale multilingual machine translation in wmt-21. we primarily focus on the small tasks, especially on small task 2 which has a small amount of training data. small task 1 contains five cen-tral/east european languages and english, having 30 translation directions. similarly, small task 2 contains five south east asian languages and english, also having 30 translation directions. in this work, we mainly concentrate on different back-translation methods @xcite @xcite for multilingual machine translation, including beam search and other sampling methods. along with it, we also explore the effect of different sizes of vocabularies and the effect of various amounts of synthetic data. on this large-scale multilingual machine translation task, we achieved the second place for both small tasks, obtaining 34.96 and 33.34 average sp-bleu scores @xcite on the hidden test set for the small task 1 and 2, respectively. ## related work multilingual neural machine translation has received increasing attention recently. since @xcite extended the traditional bilingual nmt to one-to-many translation, there has been a massive increase in work on mt systems that involve more than two languages @xcite @xcite . the recent research on multilingual nmt can be split into two directions: developing language specific components @xcite and training a single model with extensive training data, including parallel and monolingual data @xcite . here, we continue to explore the second research direction, trying to built a single multilingual nmt model for simple industrial deployment. back-translation @xcite ) has been proven as a powerful technique to leverage monolingual data for improving low-resource language pairs. @xcite and @xcite explore different sampling methods for bilingual back-translation, including beam search, constrained and unconstrained sampling. constrained sampling randomly predicts the next word within some candidates that have higher prediction probability. and unconstrained sampling randomly predicts the next words from the whole vocabulary without caring for the output distribution. in this paper, we extend their exploration to the realm of multilingualism, where similar languages affect the results. ## conclusion we demonstrate that a pretrained model with the smaller size of vocabularies is a better choice. because of the memory and time limit of the evaluation system, we can only apply a 1.2b model with the smaller vocabularies to generate high-quality synthetic data. besides, we have a different obser-vation than previous research for bilingual backtranslation: the constrained sampling method performs the best among all three back-translation methods, including the beam search and the unconstrained sampling. finally, we also show that extensive monolingual english data offers a modest improvement. combining these three findings, we iteratively train our models on partial high-quality synthetic data, achieving the second place for both small tasks.