CUNI Submission for the I nuktitut Language in WMT News 2020 (2020)

Abstract: This paper describes CUNI submission to the WMT 2020 News Translation Shared Task for the low-resource scenario Inuktitut–English in both translation directions. Our system combines transfer learning from a Czech–English high-resource language pair and backtranslation. We notice surprising behaviour when using synthetic data, which can be possibly attributed to a narrow domain of training and test data. We are using the Transformer model in a constrained submission.

## introduction the rapid development of neural machine translations (nmt) systems helped nmt in approaching human translation quality for high-resource language pairs like chinese-english @xcite or english-czech @xcite . this is not true for a low-resource scenario, where the lack of a large quantity of parallel data limits the performance of an nmt system. thus, in recent years the research focused on low-resource nmt become important. in this paper, we describe our approach to lowresource nmt of inuktitut-english. we use the standard transformer-big model @xcite and apply two techniques to improve the performance on the low-resource language, namely transfer learning @xcite and backtranslation @xcite . we used a similar approach in wmt19 for gujarati and kazakh machine translation @xcite . training low-resource model solely on authentic parallel data results in poor performance, and that results in low quality of generated backtranslation of monolingual data as well. hence transfer learning is as an excellent method to first improve the performance of the nmt system that is later used for backtranslation of monolingual data. ## background in this section, we describe the technique of transfer learning, backtranslation, and models that we used for training inuktitut-english models. ## results all reported results are calculated on the testset of wmt 2020 and evaluated with case sensitive sacrebleu @xcite . 2 the evaluation is done on unmodified outputs of our system. for final wmt20 submission, we have automatically corrected quotes to match the source. this step is not used for results in the baseline models in in @xmath1. for a model, where we used a mix of "synthetic and authentic" data that is generated by our @xmath2, we can notice a performance dropped from 22.98 to 20.91. however, following with fine-tuning this model again with authentic data, we get an increase in performance to 25.38. this is unexpected behaviour. our understanding is that it could be attributed to a narrow domain of train and testset containing mainly speech transcripts. at the same time, the synthetic data are generated from english news articles, which is a more general domain. therefore, while the model is trained on the general domain, it loses score on a domain-specific testset. this could be tested if we could obtain a testset on a different domain than speech transcription; however, we do not have such testset available. during training on synthetic data, the model learns a general domain and loses performance on domain-specific testset to 20.91 bleu, however after fine-tuning again on authentic domain-specific data only it reaches the highest performance of 25.38 bleu. ## conclusion we participated in the low-resource inuktitut-english in the wmt 2020 news translation shared task. we combined transfer learning with the backtranslation and obtained significant improvements. surprisingly, we found out that although training the model on backtranslated data decreases the performance of the system in terms of bleu score; it is still helpful when continued with fine-tuning on authentic data. we believe this is mainly because the inuktitut-english training and test data are from a narrow domain of legal texts.