Teaching Unseen Low-resource Languages to Large Translation Models (2022)

Abstract: In recent years, large multilingual pre-trained neural machine translation model research has grown and it is common for these models to be publicly available for usage and fine-tuning. Low-resource languages benefit from the pre-trained models, because of knowledge transfer from high- to medium-resource languages. The recently available M2M-100 model is our starting point for cross-lingual transfer learning to Finno-Ugric languages, like Livonian. We participate in the WMT22 General Machine Translation task, where we focus on the English-Livonian language pair. We leverage data from other Finno-Ugric languages and through that, we achieve high scores for English-Livonian translation directions. Overall, instead of training a model from scratch, we use transfer learning and back-translation as the main methods and fine-tune a publicly available pre-trained model. This in turn reduces the cost and duration of training high-quality multilingual neural machine translation models.

## introduction we participate in the wmt 2022 general machine translation shared task where we submit a system for english-livonian and livonian-english translation directions. our system is trained in the unconstrained setting utilizing data from other languages that are all in a way related to livonian. recently, the development of large multilingual models has been increasing @xcite @xcite and thus there are multiple pre-trained multilingual models available for further fine-tuning to a specific task. fine-tuning these models on indomain data saves time and computational costs by not having to train a multilingual model from scratch. we utilize the m2m-100 multilingual pretrained neural machine translation (nmt) model @xcite and do cross-lingual transfer learning to low-resource language pairs from the finno-ugric language family, including the livonian language. we further improve our model with two back-translation iterations and a final finetuning on languages that have available original parallel data paired with livonian. the languages we use to support the english (en)-livonian (liv) directions are from the finno-ugric language family or geographically close to that family of languages: finnish (fi), estonian (et), latvian (lv), norwegian (no), võro (vro), north sami (sme), south sami (sma), inari sami (smn), skolt sami (sms), lule sami (smj). the structure of the article consists of giving insight into the related work in the field of lowresource nmt and from the finno-ugric language family perspective in section 2, the description of data in section 3, the overview of our system architecture and training methods in section 4, description of experiments in section 5 and the results in section 6. ## conclusion large pre-trained multilingual neural machine translation models prove to be beneficial to lowresource finno-ugric languages, such as livonian. we placed in the top 2 for the english-livonian language pair in the wmt22 general machine translation shared task. training in an unconstrained setting gets reasonable and good-quality results, especially when using languages close to livonian to help achieve a better translation quality. in the future, we plan to test additional and more recent pretrained multilingual models as a starting point for cross-lingual transfer learning and add more lowresource finno-ugric languages into the dataset.