Miðeind’s WMT 2021 Submission (2021)

Abstract: We present Miðeind’s submission for the English→Icelandic and Icelandic→English subsets of the 2021 WMT news translation task. Transformer-base models are trained for translation on parallel data to generate backtranslations teratively. A pretrained mBART-25 model is then adapted for translation using parallel data as well as the last backtranslation iteration. This adapted pretrained model is then used to re-generate backtranslations, and the training of the adapted model is continued.

## introduction our work on machine translation for icelandic has been going on for a couple of years as a part of the government sponsored icelandic language technology programme @xcite . by building on state-of-the-art solutions we have developed an open and effective translation system between icelandic and english. to achieve this, we collect parallel icelandic and english texts which are filtered for good quality alignments. we also collect monolingual text for backtranslations. we follow tried and tested methods in neural machine translation using iterative backtranslation @xcite and adapt the multilingual denoising autoencoder model mbart-25 @xcite for translation. ## datasets we used several parallel and monolingual datasets, both publicly available and created in-house. ## adapting mbart-25 for translation the mbart-25 @xcite ) (610m parameters) language model is far larger than the transformer-base model (110m parameters). it was pretrained on 25 languages, including english and swedish, but not icelandic. we adapt it for translation from icelandic to english and vice versa, using the same human-derived parallel translation data as for the transformer-base model along with the synthetic backtranslated corpus in a ratio of 1:2. we do not use any pre-tokenization and inherit the bpe sententencepiece vocabulary from mbart-25 (of size 250k) with the addition of an icelandic language marker that was randomly initialized. we use the same hyperparameters as in @xcite and the implementation from fairseq @xcite . the models are trained until their performance on the development sets plateaus. the initial learning rate was set to 3e-4. sixteen 32gb nvidia v100 gpus connected with infiniband were used for training. the effective batch size was around 10k sequences and the training took around 4 days of wall clock time per model. subsequently, these trained models were used to generate improved backtranslations. we then continued training the first iteration of our models with the new backtranslated data for another 30,000 steps for the icelandic-english direction, and 36,000 steps for the english-icelandic direction. the same training configurations were maintained as for the earlier runs. ## conclusion we have shown how a small team with modest resources can adapt state-of-the-art methods to a medium resource language and achieve competitive results on machine translation between english and icelandic. the trained models are available for translation at https://velthyding.is and will be made available at the open clarin-is foot_4 repository. while a formal human comparison of the current models to the popular google translate service has not been performed, hundreds of monthly active users choose our solutions for translation between icelandic and english.