P eru is Multilingual, Its Machine Translation Should Be Too? (2021)

Abstract: Peru is a multilingual country with a long history of contact between the indigenous languages and Spanish. Taking advantage of this context for machine translation is possible with multilingual approaches for learning both unsupervised subword segmentation and neural machine translation models. The study proposes the first multilingual translation models for four languages spoken in Peru: Aymara, Ashaninka, Quechua and Shipibo-Konibo, providing both many-to-Spanish and Spanish-to-many models and outperforming pairwise baselines in most of them. The task exploited a large English-Spanish dataset for pre-training, monolingual texts with tagged back-translation, and parallel corpora aligned with English. Finally, by fine-tuning the best models, we also assessed the out-of-domain capabilities in two evaluation datasets for Quechua and a new one for Shipibo-Konibo.

## introduction neural machine translation (nmt) has opened several research directions to exploit as many and diverse data as possible. massive multilingual nmt models, for instance, take advantage of several language-pair datasets in a single system @xcite . this offers several advantages, such as a simple training process and enhanced performance of the language-pairs with little data (although sometimes detrimental to the high-resource language-pairs). however, massive models of dozens of languages are not necessarily the best outcome, as it is demonstrated that smaller clusters still offer the same benefits @xcite . peru offers a rich diversity context for machine translation research with 47 native languages @xcite . all of them are highly distinguishing from castilian spanish, the primary official language in the country and the one spoken by the majority of the population. however, from the computational perspective, all of these languages do not have enough resources, such as monolingual or parallel texts, and most of them are considered endangered @xcite . in this context, the main question then arises: shouldn't machine translation be multilingual for languages spoken in a multilingual country like peru? by taking advantage of few resources, and other strategies such as multilingual unsupervised subword segmentation models @xcite , pretraining with high resource language-pairs @xcite , back-translation @xcite , and fine-tuning @xcite , we deployed the first many-to-one and one-to-many multilingual nmt models (paired with spanish) for four indigenous languages: aymara, ashaninka, quechua and shipibo-konibo. ## related work in peru, before nmt, there were studies in rulebased mt, based on the apertium platform @xcite , for quechua eastern apurimac (qve) and quechua cuzco (quz) @xcite . furthermore, @xcite improved alignments for quz by using an agglutinative language as finnish as a pivot. apart from the quechua variants, only aymara @xcite and shipibo-konibo @xcite have been addressed with rule-based and statistical mt, respectively. ortega et al. (2020b) for southern quechua, and gómez @xcite for shipibo-konibo, are the only studies that employed sequence-tosequence nmt models. they also performed transfer learning experiments with potentially related language pairs (e.g. finnish or turkish, which are agglutinative languages). however, as far as we know, this is the first study that trains a multilingual model for some language spoken in peru. for related work on multilingual nmt, we refer the readers to the survey of @xcite . ## languages and datasets to enhance replicability, we only used the datasets provided in the americasnlp shared task foot_1 . the four languages are highly agglutinative or polysynthetic, meaning that they usually express a large amount of information in just one word with several joint morphemes. this is a real challenge for mt and subword segmentation methods, given the high probability of addressing a "rare word" for the system. we also note that each language belongs to a different language family, but that is not a problem for multilingual models, as usually the family-based clusters are not the most effective ones @xcite . pre-processing the datasets were noisy and not cleaned. lines are reduced according to several heuristics: arabic numbers or punctuation do not match in the parallel sentences, there are more symbols or numbers than words in a sentence, the ratio of words from one side is five times larger or shorter than the other, among others. ## analysis and discussion one of the most exciting outcomes is the deteriorated performance of the multilingual models using bt data, as we usually expect that added backtranslated texts would benefit performance. using tags @xcite ) to differentiate which data is synthetic or not is only a simple step to address this issue; however, there could be evaluated more informed strategies for denoising or performing online data selection @xcite . besides, in the translation into spanish, the multilingual model without bt data outperforms the rest models in all languages but quechua, where the pairwise system achieved the best translation accuracy. quechua is the "highest"-resource languagepair in the experiment, and its performance is deteriorated in the multilingual setting foot_4 . a similar scenario is shown in the other translation direction from spanish, where the best multilingual setting (+bt[t]) cannot overcome the @xmath0. nevertheless, the gains for aymara, ashaninka and shipibo-konibo are outstanding. moreover, we note that the models are not totally overfitted to any of the evaluation sets. exceptions are @xmath1, with a significant performance dropping from dev to devtest, meaning that it started to overfit to the training data. however, for @xmath2, we observe that the model achieved a better performance in the devtest set. this is due to oversampling of the same-domain dev partition for training ( §4.1) and the small original training set. concerning the results on the official test set, the performance is lower than the results with the custom evaluation sets. the main potential reason is that the official test is four times bigger than the custom devtest, and therefore, offers more diversity and challenge for the evaluation. another point to highlight is that the best result in the spanish-quechua language-pair is obtained by a multilingual model (the scores between the model ( e ) and (g) are not significantly different) instead of the pairwise baseline. decoding an indigenous language is still a challenging task, and the relatively low bleu scores cannot suggest a translation with proper adequacy or fluency. however, bleu works at the wordlevel, and other character-level metrics should be considered to better assess the highly agglutinative nature of the languages. for reference, we also report the chrf scores in ## out-of-domain evaluation it is relevant to assess out-of-domain capabilities, but more important to evaluate whether the models are still capable to fine-tune without overfitting. we use a small evaluation set for quechua (kallpa, with 100 sentences), which contains sentences extracted from a magazine @xcite . likewise, we introduce a new evaluation set for shipibo-konibo (kirika, 200 sentences), which contains short traditional stories. we tested our best model for each language-pair, fine-tune it (+ft) with half of the out-of-domain dataset, and evaluate it in the other half. to avoid overfitting, we controlled cross-entropy loss and considered very few updates for validation steps. results are shown in the quechua translations rapidly improved with the fine-tuning step, and there is a small gain in the original test for @xmath4, although the scores are relatively low in general. nevertheless, our model could outperform others (by extrapolation, we can assume that the scores for the rule-based apertium system (cavero and madariaga, 2007) and ortega et al. (2020b) for shipibo-konibo, we also observe some small gains in both directions without hurting the previous performance, but the scores are far from being robust. kirika is challenging given its old style: the translations are extracted from an old book written by missionaries, and even when the spelling has been modernised, there are differences in the use of some auxiliary verbs for instance (extra words that affect the evaluation metric) foot_6 . ## conclusion and future work peru is multilingual, ergo, its machine translation should be too! we conclude that multilingual machine translation models can enhance the performance in truly low-resource languages like aymara, ashaninka and shipibo-konibo, in translation from and into spanish. for quechua, even when the pairwise system performed better in this study, there is a simple step to give a multilingual setting another opportunity: to include a higher-resource languagepair that may support the multilingual learning process. this could be related in some aspect like morphology (another agglutinative language) or the discourse (domain). other approaches focused on more advanced sampling or adding specific layers to restore the performance of the higher-resource languages might be considered as well. besides, tagged back-translation allowed to take some advantage of the monolingual data; however, one of the most critical following steps is to obtain a more robust many-to-spanish model to generate backtranslated data with more quality. furthermore, to address the multi-domain nature of these datasets, we could use domain tags to send more signals to the model and support further fine-tuning steps. finally, after addressing the presented issues in this study, and to enable zero-shot translation, we plan to train the first many-to-many multilingual model for indigenous languages spoken in peru.