Allegro.eu Submission to WMT 21 News Translation Task (2021)

Abstract: We submitted two uni-directional models, one for English→Icelandic direction and other for Icelandic→English direction. Our news translation system is based on the transformer-big architecture, it makes use of corpora filtering, back-translation and forward translation applied to parallel and monolingual data alike

## introduction we participated in the wmt21 news translation shared task for english↔icelandic language pair. it is a medium-resource regime with under 10m parallel sentences. in our experiments we focused on two approaches for improving translation system: data filtering methods inspired by work of @xcite and data augmentation methods like back-translation or self-training @xcite @xcite . we tried to use bi-directional translation models but single-directional proved to be better. we also tried to make use of pretraining on monolingual corpora, but it also was unsuccessful. @xcite showed in their ablation study that pretraining is the most successful for low-resource regimes under 1m parallel sentences. ## system overview all of our models are based on the transformer big architecture, as described in @xcite . for training we used opennmt-py framework @xcite together with sentencepiece tokenizer @xcite unigram model of size 32k with full character coverage. we trained models on a100 gpu for 210k steps with a batch of 8192 tokens which amounts to around 12h per model. we used half-precision and tied embeddings. for optimization we used adam (kingma and ba, 2014), with a linear warmup for learning rate for 15k steps up to 0.0005 and inverse square root decay afterwards. additionally, all of our models were randomly initialized. ## results results are presented in ## conclusion this paper describes allegro.eu submission for the wmt21 news translation shared task. we took part in constrained track for the english↔icelandic language pair only. participation in this task allowed us to deepen the understanding of filtering methods common in nmt. the experiments demonstrated the importance of data filtering in medium-resource regime machine translation. in this regime, less data but of higher quality can lead to superior results.