The U niversity of E dinburgh’s E nglish- T amil and E nglish- I nuktitut Submissions to the WMT 20 News Translation Task (2020)

Abstract: We describe the University of Edinburgh’s submissions to the WMT20 news translation shared task for the low resource language pair English-Tamil and the mid-resource language pair English-Inuktitut. We use the neural machine translation transformer architecture for all submissions and explore a variety of techniques to improve translation quality to compensate for the lack of parallel training data. For the very low-resource English-Tamil, this involves exploring pretraining, using both language model objectives and translation using an unrelated high-resource language pair (German-English), and iterative backtranslation. For English-Inuktitut, we explore the use of multilingual systems, which, despite not being part of the primary submission, would have achieved the best results on the test set.

## introduction the university of edinburgh participated in the wmt20 news translation shared task for english-tamil and english-inuktitut in both translation directions. 1,2 neither language pair benefits from large quantities of parallel data, so we approach training using different techniques to compensate for this lack of data: pretraining and iterative backtranslation for english-tamil and multilingual systems for english-inuktitut. we use neural machine translation (mt) and specifically the transformer architecture @xcite : the base variant for the lower-resourced english-tamil and the big variant for the mid-resource english-inuktit. in both cases, significant improvements are seen when compared to the in-house baselines tested, particularly notable for pretraining for english-tamil. awaiting the results of the official human evaluation, we report the automatic evaluation scores using bleu @xcite as implemented in sacrebleu @xcite . a summary of these results on the dev and test sets can be found in ## english↔tamil as for our english↔gujarati systems last year at wmt19 @xcite , we use pretraining and data augmentation to tackle the low-resource language pair english-tamil. our experiments show that pre-training, training on backtranslated data and then fine-tuning is useful in both directions, although we introduce slight variations in the training and fine-tuning approaches used for each language direction. ## english↔inuktitut compared to english-tamil, the english-inuktitut language pair is relatively well-resourced at approximately 1.3m sentence pairs. as such we were able to train conventional bilingual transformer systems, which formed the basis of our submission. we also trained multilingual systems, but opted not to use these in our submission as results on the dev set did not appear to be promising (although evaluation proved challenging for this pair due to overlap between the training and dev data). postsubmission evaluation showed that our multilingual systems actually outperformed our submitted systems on the test sets. ## conclusion in this submission we focused on a low-resource language pair (english-tamil) and a mediumresource language pair (english-inuktitut). all our translation systems are based on the transformer architecture. we found it beneficial to use monolingual data in the form of backtranslations. in the case of en-ta, we saw notable gains by using pretraining using both the denoising autoencoding (mbart) objective and multilinguality in the form of german-english pretraining. however, we were not able to gain any quality from multilingual training on data for other indian languages. for english-inuktit, multilinguality did not appear to help on the dev set, but was found, post-submission, to help on the test set. in general, we found that english-tamil is a much more challenging task, where pretraining is absolutely necessary to reach acceptable quality, while for english-inuktit reasonable translation quality can be achieved using only parallel data.