F acebook AI ’s WMT 20 News Translation Task Submission (2020)

Abstract: This paper describes Facebook AI’s submission to WMT20 shared news translation task. We focus on the low resource setting and participate in two language pairs, Tamil <-> English and Inuktitut <-> English, where there are limited out-of-domain bitext and monolingual data. We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain. We explore techniques that leverage bitext and monolingual data from all languages, such as self-supervised model pretraining, multilingual models, data augmentation, and reranking. To better adapt the translation system to the test domain, we explore dataset tagging and fine-tuning on in-domain data. We observe that different techniques provide varied improvements based on the available data of the language pair. Based on the finding, we integrate these techniques into one training pipeline. For En->Ta, we explore an unconstrained setup with additional Tamil bitext and monolingual data and show that further improvement can be obtained. On the test set, our best submitted systems achieve 21.5 and 13.7 BLEU for Ta->En and En->Ta respectively, and 27.9 and 13.0 for Iu->En and En->Iu respectively.

## introduction we participate in the wmt20 news translation task in two low resource language pairs (four directions), tamil ↔ english (@xmath0. these language pairs are challenging due to the lack of in-domain bitext training data and limited monolingual data. for tamil, the available bitext corpora are from various sources; however, none of the sources is in the news domain, and each corpus is in limited size or noisy. inuktitut encompasses the challenges present for tamil, but is even more challenging because the quantity of available monolingual data is even less than the bitext data. we explore techniques that leverage available data from all languages. first, we investigate supervised learning together with pre-training using mbart @xcite . second, inspired by the recent success of improving low resource languages through multilingual models @xcite , we explore the utility of multilingual models, in the form of multilingual pretraining and subsequent fine-tuning. third, we leverage the monolingual data of the source and target languages using data augmentation techniques, such as back-translation @xcite and self-training @xcite @xcite . following @xcite , we apply these techniques iteratively. fourth, we use noisy-channel model reranking @xcite to further boost performance. the reranking uses language modeling to select a more fluent hypothesis, which requires monolingual data in the target language. additionally, we investigate how adding substantially more unconstrained data can further improve the performance of @xmath1. we incorporate data from bitext mining efforts such as ccma-trix @xcite and ccaligned @xcite , as well as additional monolingual data from ccnet @xcite curated from commoncrawl. the additional data is used for iterative back-translation and to train stronger language models for noisy-channel reranking. in a complementary direction, we investigate ways to adapt the translation system to the target domain. we explore controlled generation by adding dataset tags to indicate domain. furthermore, we fine-tune our system on the in-domain data. for all language directions, we obtain our final systems by fusing a combination of the tech-niques mentioned above. we observe that the bulk of the improvements in our systems are from iterative back-translation and self-training, except the @xmath2. noisy-channel reranking provides further improvement on top of strong systems, especially for to-english directions where we have high-quality news-domain monolingual data to train a good language model. each of the other techniques, including dataset tagging, fine-tuning on in-domain data, and ensembling also provides nice improvements. ## data for the constrained track, we use monolingual data from all languages provided in wmt20 for mbart pre-training @xcite , and we use bitext data between english and other languages for training the system from scratch or fine-tuning the pretrained mbart models. we also require english, tamil, and inuktitut monolingual data for techniques such as back-translation, self-training, and creating language models for noisy-channel reranking. for low resource languages, tamil and inuktitut, we use all the available monolingual data, e.g. newscrawl + commoncrawl + wikipedia dumps for tamil, and commoncrawl for inuktitut. for english, we only use newscrawl as the monolingual data because it is sufficiently large, high-quality, and in the news domain. for the unconstrained track, we use tamil monolingual data and tamil-english mined bitext data from external sources based on commoncrawl. the details are described in section 2.2. ## system overview we use the transformer @xcite as our model architecture for all of our systems. to better train models with datasets in different sizes, we use random search to select the hyperparameters that achieve the best bleu score on the validation set. we use sentencepiece @xcite to learn the subword units to tokenize the sentences. the details of selected hyper-parameters are listed in appendix d. all our systems are trained with fairseq foot_0 @xcite . ## results in this section, we describe the details of our systems, and we report sacrebleu @xcite on the validation set for intermediate iterations and ablations. for our validation set fine-tuned systems, we report the bleu score on our validation holdout set split. our general strategy for all language directions was to identify the best performing baseline setting, then iteratively improve upon the baseline using back-translation and self-training. finally, we apply noisy-channel reranking and fine-tuning on validation set to create our final submission. ## conclusion this paper describes facebook ai's transformer based translation systems for the wmt20 news translation shared task. we focused on two lowresource languages pairs, tamil ↔ english and inuktitut ↔ english, and we explored the same set of techniques, including dataset tagging, mbart pretraining and fine-tuning, back-translation and self-training, fine-tuning on domain-specific data, ensembling, and noisy-channel reranking. we demonstrated strong improvements by stacking these techniques properly on three language directions, @xmath3, @xmath4, and @xmath5. the @xmath6 to lack of target side monolingual data. surprisingly, self-training does not work on @xmath7. we are interested in continued exploration on how to better leverage source side monolingual data to improve @xmath8. 6 acknowledgements we thank marc'aurelio ranzato for providing discussion and guidance during the competition, vishrav chaudhary for sharing insightful data cleaning approaches, guillaume wenzek for previous work on ccnet for monolingual data used in unconstrained setting, yuqing tang for the work of mbart pretraining and multilingual fine tuning, ahmed el-kishky and holger schwenk for sharing their mined data for tamil, sergey edunov for sharing cleaned up dataset to speed up our early exploration, and michael auli for sharing experience about noisy-channel reranking technique.