Improving Neural Machine Translation of Indigenous Languages with Multilingual Transfer Learning (2023)

Abstract: Machine translation (MT) involving Indigenous languages, including endangered ones, is challenging primarily due to lack of sufficient parallel data. We describe an approach exploiting bilingual and multilingual pretrained MT models in a transfer learning setting to translate from Spanish into ten South American Indigenous languages. Our models set new SOTA on five out of the ten language pairs we consider, even doubling performance on one of these five pairs. Unlike previous SOTA that perform data augmentation to enlarge the train sets, we retain the low-resource setting to test the effectiveness of our models under such a constraint. In spite of the rarity of linguistic information available about the Indigenous languages, we offer a number of quantitative and qualitative analyses (e.g., as to morphology, tokenization, and orthography) to contextualize our results.

## introduction artificial intelligence (ai) is being widely integrated into many natural language processing (nlp) applications in our daily lives. however, these language technologies have focused almost exclusively on widely-spoken languages @xcite . under-represented languages such as endangered languages are thus left out. for example, the google machine translation (mt) system does not support any of the languages included in our current study. 1 our objective in this work is hence to build machine translation (mt) models for indigenous languages, which are by definition low-resource and possibly endangered. more specifically, we focus on south american indigenous languages. in a mt scenario, a language pair is considered 'low-resource' if the parallel corpora consists of less than 0.5 million of parallel sentences and 'extremely low-resource' if less than 0.1 million of parallel sentences @xcite . in this work, nine out of ten languages pairs we consider have under 0.1 million pairs of sentences (with only one language pair having roughly 0.1 million pairs of sentences). developing mt systems for endangered languages can help preserve these languages. neural machine translation (nmt) is a branch of mt that leverages neural networks to build translation systems. despite that nmt is able to produce powerful mt systems, it is data-hungry. that is, it requires large amounts of data to train a quality nmt model @xcite . contemporary machine translation systems are oftentimes trained on over a million of parallel sentences @xcite for high-resource language pairs. in contrast, the size of the dataset we have is limited. transfer learning has been shown to help mitigate this issue by porting knowledge e.g. from a parent model to a child model @xcite . we leverage two types of pretrained mt models: bilingual models and a multilingual model. the overall training approach is illustrated in figure 1 . our datasets are provided by @xcite shared task. we compare our performance to the winner of the shared task @xcite . the rest of this study is organized as follows: section 2 is a literature review on indigenous mt, transfer learning, the application of transfer learning to nmt, and the challenge of cross-lingual transfer. in section 3, we describe our experimental settings. we present our results in section 4, and provide discussions in section 5. we conclude in section 6. ## results we evaluate the translation performance with two automatic mt evaluation metrics: bleu @xcite and chrf @xcite . chrf is an automatic evaluation metric for mt task which can be seen as a f-score for text and has value between 0 and 1. bleu and chrf are the two metrics adopted by americasnlp 2021 shared task. we surpassed the winner of americas-nlp2021 @xcite , in either or both metrics, for 5 language pairs with the following languages as target: bribri (bzd), asháninka (cni), wixarika (hch), nahuatl (nah), and hñähñu (oto). notably, we double the performance in bleu score for es-bzd, increasing by about 7.7 bleu scores and 0.1 chrf. we increase ∼ 2 bleu points in es-oto and ∼ 1 bleu points in es-hch. for both es-cni and es-nah, we slightly surpass their performance in both metrics. the performance of experiments are shown in all surpassing results are achieved by mbart50 or mbart50 curr . surprisingly, mbart50 curr does not consistently improve the performance if compared to mbart50; some of the best performances are achieved by mbart50 (es-bzd, es-hch, es-oto). nevertheless, mbart50 curr performs slightly better than mbart50 on average by 0.076 bleu and 0.0034 chrf. averagely, mbart50 curr achieves 7.143 bleu score and 0.3134 chrf while mbart50 achieves 7.068 bleu score and 0.31 chrf. generally, multilingual models perform better than bilingual model despite that in some language pairs, es-en model performs nearly as good as multilingual models and outperform multilingual models in es-aym and es-tar. for 3 bilingual models, es-en model generally outperforms the other two es-ca and es-ro models. ## conclusion in this paper, we describe how we apply transfer learning to mt from spanish to ten low-resource south american indigenous languages. we finetune pretrained bilingual and multilingual mt models on downstream spanish to indigenous language pairs and show the utility of these models. we are able to surpass sota in five language pairs using multilingual pretrained mt models without leveraging any external data. empirically, our results show that this method performs robustly even with an oversegmentation issue on the target side. we also discuss multiple issues that interact with our task, including translating between languages of different morphological structures, effect of tokenization, and non-standard orthography.