Language Model Pretraining and Transfer Learning for Very Low Resource Languages (2021)

Abstract: This paper describes our submission for the shared task on Unsupervised MT and Very Low Resource Supervised MT at WMT 2021. We submitted systems for two language pairs: German ↔ Upper Sorbian (de ↔ hsb) and German-Lower Sorbian (de ↔ dsb). For de ↔ hsb, we pretrain our system using MASS (Masked Sequence to Sequence) objective and then finetune using iterative back-translation. Final finetunng is performed using the parallel data provided for translation objective. For de ↔ dsb, no parallel data is provided in the task, we use final de ↔ hsb model as initialization of the de ↔ dsb model and train it further using iterative back-translation, using the same vocabulary as used in the de ↔ hsb model.

## introduction transformer based architecture @xcite has become the de-facto approach for training nmt models. these models have achieved good performance for resource rich languages. nmt models are usually data hungry and require lot of parallel data to get trained. however, many low-resource languages have very little or no parallel data to train a nmt model. for low resource language pairs, unsupervised mt @xcite @xcite , and transfer learning @xcite ) have proven to be helpful in improving the translation performance. unsupervised mt has gained a lot of attention in the past 3 years as it utilizes only monolingual data to train a nmt system. in this paper, we present our system for shared task on unsupervised mt and very low resource supervised mt at wmt2021. the task covers three languages pairs german (de) ↔ lower sorbian (dsb), german (de) ↔ upper sorbian (hsb), and russian (ru) ↔ chuvash (ch). we submitted systems for de ↔ hsb and de ↔ dsb. for de ↔ dsb there is no parallel data provided but for de ↔ hsb, there is small parallel data. summary of our submitted systems: ## system overview in this section, we describe the details of the submitted systems to shared task on unsupervised mt and very low resource supervised mt at wmt 2021. we report results for our 2 types of models: to train our models, we use shared encoderdecoder transformer architecture. we also use shared vocabulary of both source and target languages. for de ↔ dsb, we use the same vocabulary as used in de ↔ hsb model without considering the vocabulary mismatch. ## experiments in this section, we describe the experimental setup and the hyper-parameters used. ## conclusion in this paper, we study the impact of language model pretraining together with iterative backtranslation for very low resource language pair i.e. de ↔ hsb. we also study the impact of transfer learning from de ↔ hsb to de ↔ dsb. in future, we plan to filter bad back-translated data while training for de ↔ dsb using iterative back-translation and also different transfer learning techniques to improve the performance for de ↔ dsb.