E din S aar@ WMT 21: N orth- G ermanic Low-Resource Multilingual NMT (2021)

Abstract: We describe the EdinSaar submission to the shared task of Multilingual Low-Resource Translation for North Germanic Languages at the Sixth Conference on Machine Translation (WMT2021). We submit multilingual translation models for translations to/from Icelandic (is), Norwegian-Bokmal (nb), and Swedish (sv). We employ various experimental approaches, including multilingual pre-training, back-translation, fine-tuning, and ensembling. In most translation directions, our models outperform other submitted systems.

## introduction this paper presents the neural machine translation (nmt) systems jointly submitted by the university of edinburgh and saarland university to the wmt2021 multilingual low-resource translation for indo-european languages task, describing both primary and contrastive systems which translate to/from the three north germanic languages, icelandic (is), norwegian-bokmål (nb), and swedish (sv). our contrastive system, submitted as "edinsaarcontrastive" outperforms the other submissions across all evaluation metrics except for bleu, for which our "edinsaarprimary" system performs best. although low-resource mt has recently gained much attention, there is little prior work on north germanic languages. we contribute to this space by experimenting with both training a multilingual system from scratch and exploiting model adaptation from a large pre-trained language model. we fine-tune our initial translation models to the target languages, and then experiment with further indomain fine-tuning. data is sourced from openly available data sets in accordance with the corpora allowed in the shared task. we use parallel data sets pairing our target languages with each other and with the allowed high-resource languages, and monolingual data from wikipedia. the rest of the paper is structured as follows: we review related work in section 2, we introduce the methods and experimental settings including data and model architecture in section 3, we evaluate model performance in section 4, and, finally, we draw conclusions and suggest avenues for future work in section 5. ## related work recent work in nmt for north germanic languages is limited; however, opus-mt @xcite , which contains over 1,000 pre-trained, ready-to-use neural mt models including models for danish, norwegian, and swedish, is a notable exception. due to the scarcity of parallel data for lowresource languages, recent work leverages monolingual data, including pivoting from high-resource languages @xcite , and using back-translation @xcite to generate pseudoparallel data with synthetic sources from monolingual data. since the little parallel data that is available often comes from noisy web crawls, parallel corpus filtering is used to develop better translation models @xcite . additional methods for boosting the performance of low-resource pairs include transfer learning from models trained on higher-resource pairs @xcite , and developing multilingual systems to allow models to take advantage of linguistic relatedness. multilingual systems can employ either separate encoders or decoders for each language @xcite , or shared encoders/decoders, and can additionally make zeroshot mt possible @xcite , while scaling to hundreds of language pairs @xcite . sampling language pairs in proportion to their prevalence in the training data can ensure that all directions get enough coverage by the model @xcite . further fine-tuning multilingual systems on target language directions can improve performance of low-resource pairs @xcite . adapting a multilingual pre-trained language model to the translation task has led to improvements in translation quality @xcite . finally, combining multiple mt system checkpoints together by ensembling improves performance of the final system @xcite . ## method given a set of primary languages l p and secondary languages l s , we train a multilingual mt system on the parallel data between all the language combinations {l p , l s } ↔ {l p , l s }. this is our baseline. we extend this approach with a combination of the following methods: pre-training: we initialize a base model using a highly multilingual pre-trained model, in order to transfer the learned parameters to the translation task. this is our primary system. back-translation: we use the baseline model to back-translate monolingual corpora in l p into all other languages in l p to obtain a training data set of back-translations d bt . fine-tuning: we fine-tune the baseline model on the subset of languages {l p , l s } ↔ l p , on both parallel and back-translated data d bt . our contrastive system is an ensemble of the last four checkpoints of this model. ## evaluation comparing results between models, our primary mt5_base_ada system outperforms the marian model trained from scratch by an average of +10.7 and +8.6 bleu points on the internal and shared task validation sets, respectively. the further fine-tuned variant mt5_base_ada_ft leads to an additional average improvement of just under +1 bleu point on both sets, showing that the mt5 model already learned a good amount about our target task and languages from our initial adaptation step. the marian model is also outperformed by the fine-tuned variant marian_ft, resulting in an average improvement of +7.7 bleu points on the internal test set and +8.5 bleu points on the shared task validation set. both the mt5_base_ada_ft and marian_ft models are exposed to similar language data; however, the mt5 language model we adapted from (mt5-base) is much larger than our marian model (580 million vs 44 million parameters), and was trained on more language data (750 gb vs 46 gb), so it had a much stronger base to start from. ensembling the last 4 checkpoints of the fine-tuned marian model for marian_ft_esmb boosts performance by +0.3 and +0.6 average bleu on the internal and shared task validation sets over marian_ft; however, the mt5_base_ada_ft model still outperforms the marian_ft_esmb model by +3.7 and +0.5 average bleu on the internal test set and the shared task validation set, respectively. therefore, we submitted the mt5_base_ada_ft model as our primary system to the shared task; however, our contrastive system, the marian_ft_esmb model, won in the shared task rankings. in the global automated evaluations of the shared task, our contrastive system is the best-performing submitted system foot_3 , outperforming the official mt5 baseline by approximately +8.5 bleu. we hypothesize that the mt5 baseline, while being pretrained on massive amounts of partially noisy monolingual data, has learned the translation task via training on the development set only, so it has less informative parallel data available than our models. the m2m-100 @xcite baseline outperforms all submitted systems, despite having been trained on noisy parallel data only. we hypothesize that the highly-multilingual nature of the m2m-100 model allows the target languages to benefit from the supervisory signals between related language combinations. ## conclusion and future work we contribute to the growing space of nmt for north germanic languages. we explore multilingualism by training a transformer with a shared encoder and decoder for all language pairs from scratch, as well as adapting a pre-trained multilingual language model. fine-tuning these models to our low-resource language pairs was a key component in our success in the task, and we additionally confirm that employing popular techniques in machine translation, such as data filtering, back-translation, and model ensembling are beneficial for improving performance on low-resource directions. in future work, we would like to experiment with fine-tuning additional pre-trained models such as the m2m-100, incorporating iterative back-translation, and trying different sampling methods during training to boost lower performing low-resource language pairs.