Low-Resource Machine Translation Using Cross-Lingual Language Model Pretraining (2021)

Abstract: This paper describes UTokyo’s submission to the AmericasNLP 2021 Shared Task on machine translation systems for indigenous languages of the Americas. We present a low-resource machine translation system that improves translation accuracy using cross-lingual language model pretraining. Our system uses an mBART implementation of fairseq to pretrain on a large set of monolingual data from a diverse set of high-resource languages before finetuning on 10 low-resource indigenous American languages: Aymara, Bribri, Asháninka, Guaraní, Wixarika, Náhuatl, Hñähñu, Quechua, Shipibo-Konibo, and Rarámuri. On average, our system achieved BLEU scores that were 1.64 higher and chrF scores that were 0.0749 higher than the baseline.

## introduction neural machine translation (nmt) systems have produced translations of commendable accuracy under large-data training conditions but are datahungry @xcite and perform poorly in low-resource languages, where parallel data is lacking @xcite . many of the indigenous languages of the americas lack adequate amounts of parallel data, so existing nmt systems have difficulty producing accurate translations for these languages. additionally, many of these indigenous languages exhibit linguistic properties that are uncommon in high-resource languages, such as english or chinese, that are used to train nmt systems. one striking feature of many indigenous american languages is their polysynthesis @xcite . polysynthetic languages display high levels of inflection and are morphologically complex. however, nmt systems are weak in translating "low-frequency words belonging to highly-inflected categories (e.g. verbs)" @xcite . quechua, a low-resource, polysynthetic american language, has on average twice as many morphemes per word compared to english @xcite , which makes machine translation difficult. @xcite shows that information is often lost when translating polysynthetic languages into spanish due to a misalignment of morphemes. thus, existing nmt systems are not appropriate for indigenous american languages, which are low-resource, polysynthetic languages. despite the scarcity of parallel data for these indigenous languages, some are spoken widely and have a pressing need for improved machine translation. for example, quechua is spoken by more than 10 million people in south america, but some quechua speakers are not able to access health care due to a lack of spanish ability @xcite . other languages lack a large population of speakers and may appear to have relatively low demand for translation, but many of these languages are also crucial in many domains such as health care, the maintenance of cultural history, and international security @xcite . improved translation techniques for low-resource, polysynthetic languages are thus of great value. in light of this, we participated in the americas-nlp 2021 shared task to help further the development of new approaches to low-resource machine translation of polysynthetic languages, which are not commonly studied in natural language processing. the task consisted of producing translations from spanish to 10 different indigenous american languages. in this paper, we describe our system designed for the americasnlp 2021 shared task, which achieved bleu scores that were 1.64 higher and chrf scores that were 0.0749 higher than the baseline on average. our system improves translation accuracy by using monolingual data to improve understanding of natural language before finetuning for each of the 10 indigenous languages. ## results we describe our results in the baseline results and test2 results were both 7 https://github.com/mjpost/sacrebleu produced using the same test set and by systems where the development set was not used for training. thus, the baseline results and test2 results can be directly compared. on average, our system used to produce the test2 results achieved bleu scores that were 1.54 higher and chrf scores that were 0.0725 higher than the baseline. on the same test set, our test1 system produced higher bleu and chrf scores for nearly every language. this is expected, as the system used to produce test1 was trained on slightly more data; it used the development set of the indigenous american languages provided by americasnlp 2021 in addition to the training set. if we factor in our results from test1 to our test2 results, we achieved bleu scores that were 1.64 higher and chrf scores that were 0.0749 higher than the baseline on average. overall, we attribute this improvement in scores primarily to the cross-lingual language model pretraining (conneau and lample, 2019) we performed, allowing our model to learn about natural language from the monolingual data before finetuning on each of the 10 indigenous languages. ## conclusions and future work we described our system to improve low-resource machine translation for the americasnlp 2021 shared task. we constructed a system using the mbart implementation of fairseq to translate from spanish to 10 different low-resource indigenous languages from the americas. we demon-strated strong improvements over the baseline by pretraining on a large amount of monolingual data before finetuning our model for each of the lowresource languages. we are interested in using dictionary augmentation techniques and creating pseudo-monolingual data to use during the pretraining process, as we have seen improved results with these two techniques when translating several low-resource african languages. we can also incorporate these two techniques in an iterative pretraining procedure @xcite to produce more pseudomonolingual data and further train our pretrained model for potentially better results. future research should also explore using probabilistic finite-state morphological segmenters, which may improve translations by exploiting regular agglutinative patterns without the need for much linguistic knowledge @xcite and thus may work well with the low-resource, polysynthetic languages dealt with in this paper.