F acebook AI ’s WMT 21 News Translation Task Submission (2021)

Abstract: We describe Facebook’s multilingual model submission to the WMT2021 shared task on news translation. We participate in 14 language directions: English to and from Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese. To develop systems covering all these directions, we focus on multilingual models. We utilize data from all available sources — WMT, large-scale data mining, and in-domain backtranslation — to create high quality bilingual and multilingual baselines. Subsequently, we investigate strategies for scaling multilingual model size, such that one system has sufficient capacity for high quality representations of all eight languages. Our final submission is an ensemble of dense and sparse Mixture-of-Expert multilingual translation models, followed by finetuning on in-domain news data and noisy channel reranking. Compared to previous year’s winning submissions, our multilingual system improved the translation quality on all language directions, with an average improvement of 2.0 BLEU. In the WMT2021 task, our system ranks first in 10 directions based on automatic evaluation.

## introduction we participate in the wmt2021 shared task on news translation and submit a multilingual translation system. in recent years, multilingual translation has gained significant interest as an alternative to developing separate, specialized systems for different language directions @xcite @xcite @xcite . multilingual systems have great potential for simplicity and consolidation, making them attractive options for the development and maintenance of commercial translation technologies. from a research standpoint, studies of transfer learning between related languages and developing methods that incorporate low-resource languages are strong motivators for grouping languages together in one system @xcite . despite such motivations, existing multilingual translation systems have been unable to show that the translation quality of multilingual systems surpasses that of bilingual. several works compare to bilingual baselines, but these baselines do not incorporate standard techniques used across the field -such as backtranslation or dense model scaling. further, multilingual translation systems are often developed on non-standard training datasets and use different evaluation datasets. these factors make it difficult to assess the performance of multilingual translation, particularly when compared to the most competitive bilingual models. in this work, our aim is to demonstrate against the winning wmt2020 models and our bilingual wmt2021 systems that multilingual translation models have stronger performance than bilingual ones. we focus on 14 language directions: english to and from czech, german, hausa, icelandic, japanese, russian, and chinese. we create an unconstrained system that utilizes both wmt distributed and publicly available training data, apply large-scale backtranslation, and explore dense and mixture-of-expert architectures. we compare the impact of various techniques on bilingual and multilingual systems, demonstrating where multilingual systems have an advantage. our final multilingual submission improves the translation quality on average +2.0 compared to the wmt2020 winning models, and ranks first in 10 directions based on automatic evaluation on the wmt2021 leaderboard. ## data we participate in translation of english to and from czech (cs), german (de), hausa (ha), icelandic (is), japanese (ja), russian (ru), and chinese (zh). we describe our bitext and monolingual data sources, including additional mined data created for hausa, and our preprocessing pipeline. ## system overview we describe step-by-step how we created our final multilingual submission for wmt2021. we detail our bilingual and multilingual model architectures, as well as how we incorporate strategies such as backtranslation, news-domain finetuning, ensembling, and noisy channel reranking. ## experiments and results we conduct experiments to quantify the impact of each of the component in our system. all experiments are evaluated on newstest20 @xcite using sacrebleu @xcite . ## conclusion in this paper, we describe facebook's multilingual model submission to the wmt2021 shared task on news translation. we employed techniques such as large scale backtranslation, bitext mining, large scale dense and sparse multilingual models, indomain finetuning, ensembling, and noisy channel reranking. we provide extensive experiment results to quantify the impact of each technique, as well as how well they cumulatively stack to produce the final system. our results demonstrate that multilingual translation can achieve state-of-the-art performance on both low resource and high resource languages, beating our strong bilingual baselines and previous years' winning submissions.