Machine Translation of Low-Resource I ndo- E uropean Languages (2021)

Abstract: In this work, we investigate methods for the challenging task of translating between low- resource language pairs that exhibit some level of similarity. In particular, we consider the utility of transfer learning for translating between several Indo-European low-resource languages from the Germanic and Romance language families. In particular, we build two main classes of transfer-based systems to study how relatedness can benefit the translation performance. The primary system fine-tunes a model pre-trained on a related language pair and the contrastive system fine-tunes one pre-trained on an unrelated language pair. Our experiments show that although relatedness is not necessary for transfer learning to work, it does benefit model performance.

## introduction machine translation (mt) is currently one of the hot application areas of deep learning, with neural machine translation (nmt) achieving outstanding performance where large amounts of parallel data are available @xcite @xcite . in low-resource settings, transfer learning methods have proven useful for improving system performance @xcite @xcite . in this work, we focus on studying nmt, including in the low-resource scenario. in particular, we focus our attention on investigating the effect of language relatedness on the transfer process. we define relatedness of a pair of languages based on belonging to the same language family. that is, by 'related' we mean 'within the same language family' whereas by 'unrelated' we mean 'belong to two different language families'. for example, we call english and swedish related since they belong to the germanic language family but english and french not related since the latter belongs to the romance language family. as an analogy to human learning, we would like to ask: if there are two translators (pre-trained models), for example one @xmath0, will they (after extra training, i.e., fine-tuning/transfer learning process) have different abilities to translate from catalan into occitan? if the catalan-spanish translator proves to perform better @xmath1, we may attribute this to spanish and occitan being members of the romance language family while english being a member of the, germanic, different family. of particular interest to us are two sets of languages belonging to two different language families, one set to romance and the other set to germanic. for the former set, we take catalan (ca), italian (it), occitan (oc), romanian (ro), and spanish (es); and we take english (en) for the latter set. we note that both romance and germanic are two branches of the larger indo-european language family, and hence there are some level of relatedness between all the languages we study in this work. nevertheless, languages in romance and germanic differ in some syntactic structures. for example, the position of attributive adjectives in germanic languages is before the noun while it is after the noun for romance languages (van de @xcite . despite differences, the writing system of all languages in this work is the latin script. this can be beneficial to transfer learning because these languages can potentially share common lexical items or morphemes, which may facilitate the transfer learning process. as mentioned, we adopt transfer learning since it has been shown to improve translation quality for low-resource language pairs. for example, @xcite sequentially build a parent model and a child model where each is trained, respectively, on high-resource and low-resource language pairs with the child model retaining parameters from lang. pair primary contrastive ca-it ca-it/ca-es ca-en ca-oc ca-es ca-en ca-ro ca-es ca-en the parent model. in addition, nguyen and chiang (2017) successfully transfer knowledge from a parent model to a child model where both models are trained on low-resource but related language pairs. kocmi and bojar (2018) also adopt a similar approach to these previous works, but based their work on the transformer architecture @xcite instead of using a recurrent encoderdecoder network with attention @xcite . our work builds on these studies. namely, we empirically apply transfer learning under different conditions to members of various language families. the outcomes of our work are similar to those of @xcite ; @xcite . that is, while we find relatedness to be beneficial, a positive transfer between an unrelated language pair can still be possible (although with a potentially diminished performance). the rest of this paper is organized as follows: in section 2, we overview related work. we introduce our datasets and experiments in section 3. in section 4, we present and analyze our results. we conclude in section 5. ## conclusion in this work, we confirm previous works showing that transfer learning benefits nmt. besides, an empirical comparison between transferring from related and unrelated languages shows that relatedness is not strictly required for knowledge transfer, but it does result in higher performance than transferring with unrelated languages.