Four Approaches to Low-Resource Multilingual NMT : The H elsinki Submission to the A mericas NLP 2023 Shared Task (2023)

Abstract: The Helsinki-NLP team participated in the AmericasNLP 2023 Shared Task with 6 submissions for all 11 language pairs arising from 4 different multilingual systems. We provide a detailed look at the work that went into collecting and preprocessing the data that led to our submissions. We explore various setups for multilingual Neural Machine Translation (NMT), namely knowledge distillation and transfer learning, multilingual NMT including a high-resource language (English), language-specific fine-tuning, and multilingual NMT exclusively using low-resource data. Our multilingual Model B ranks first in 4 out of the 11 language pairs.

## introduction this paper presents the submission of the helsinki-nlp team to the americasnlp 2023 shared task. the task consisted in developing machine translation (mt) systems for 11 indigenous languages of the americas: aymara (aym), bribri (bzd), asháninka (cni), chatino (czn), guarani (gn), wixarika (hch), nahuatl (nah), hñähñu (oto), quechua (quy), shipibo-konibo (shp), and rarámuri (tar). the americasnlp task has been running for two years: in 2021 @xcite it was first introduced, and in 2022 it consisted of speech-to-text translation (stt). foot_0 this year's task is similar to the one held in 2021, but it includes an additional language (chatino) and the use of the development set in training is not allowed. our 2021 submission @xcite reached the first rank in nine out of ten languages and serves as the baseline for this year's task. the 11 target languages involved in the task vary a lot in terms of "resourcedness". on one side of the spectrum, there are languages like quechua and guarani with millions of native speakers, whereas on the other end, the variety of hñähñu used in the development and test sets only has about 100 elder speakers. 2 many of the target languages show dialectal variation, and some have different spelling norms and conventions. furthermore, some datasets contain instances of codeswitching with spanish, and some of the languages are polysynthetic. all these factors make the task at hand particularly challenging. a large part of our effort focuses on increasing the amount of parallel data for training. building on our work for the 2021 shared task, we employ several strategies: mining, extraction and alignment of publicly available parallel resources, backtranslation of monolingual data @xcite , and data augmentation by pivoting through english @xcite . on the modelling side, our winning 2021 submission was based on a multilingual (one-to-many) model that was pretrained mostly on the spanishto-english task and later fine-tuned on the lowresource indigenous languages. we keep this general approach in most of this year's submissions, but provide some variations to this theme: model a uses knowledge distillation and transfer learning instead of training from scratch. in this context, we also experiment with different data labeling schemes. model b reproduces our 2021 setup with updated data. ## data collection and preparation similar to our 2021 submission, we worked on finding relevant corpora from additional sources and cleaning and filtering them. we utilised the opusfilter toolbox foot_4 @xcite , which provides both ready-made and extensible methods for combining, cleaning, and filtering parallel and monolingual corpora. opusfilter uses a configuration file that lists all the steps for processing the data; in order to make quick changes and extensions programmatically, we generated the configuration file with a python script. ## models we tested four major model configurations, which we refer to as a, b, c and d. all models are multilingual neural mt (nmt) models and include the spanish-english translation task in some form. models b and c also include language-specific finetuning steps. all models are based on the transformer architecture @xcite . models a and c are trained using the mariannmt toolkit @xcite , while b and d are implemented with opennmt-py 2.0 @xcite . all models were trained on a single gpu, except model d, which was trained on 4 gpus. we use subword sentencepiece segmentation @xcite for the training data. we train a shared vocabulary for all languages with size 32k that is used in all the models. further details of the configurations are listed in appendix b. ## results our results are shown in the baseline turned out to be quite hard to beat: for five languages (hch, nah, oto, shp, tar), the best submission was less than 2 chrf points above the baseline. the competition among participants was also very tight this year: for the same five languages, there is less than 1 chrf point difference between the first and second participant. differences of less than 2 chrf points can be observed for two additional languages (cni, gn). we believe that conducting significance testing to compare the participants' results would be beneficial in this scenario. regarding our models, model b is our clear bestperforming system. it reached first rank on 4 out of the 11 language pairs and third rank on two other occasions. model b consistently outperformed all our other models. its good performance can be attributed to its pre-training phase on spanish-english data including a small percentage of the indigenous data. for this model, we also focused our efforts in checkpoints' selection. further analysis will be required to investigate the performance differences between our models b and c, which used the same overall setup but show various minor differences in terms of toolkits, hyperparameters and curriculum definition. the variants of model a perform very similarly to each other, although removing the quality tags (a.2) leads to a significant increase for es-shp. comparing models a and model b, our results indicate that training a multilingual model jointly from scratch is more beneficial than transfer learning approaches. model c seems to be on par with models a, although it works particularly well for es-czn. with model c, we expected that language-specific finetuning would boost results. if we compare models b and c, our results match previous research, where it is stated that low-resource translation benefits from jointly-trained multilingual models @xcite . finally, while model d works well for es-shp, outperforming models c and a.2, we observe that in general it yields poor results. nonetheless, we decided to use it anyway to test it in a real use case. specifically for model d, we were interested in testing the knowledge transfer capabilities of modular systems in low-resource multilingual scenarios. indeed, these systems have demonstrated efficient transfer learning properties (escolano peinado, 2022). however, in this set of experiments, model d lags behind our other nonmodular systems for all other languages, indicating that perhaps the data available to train the languagespecific modules was insufficient or that the parameter sharing strategies we chose were not optimal. in our experiments we also noticed that the modular systems ignore the variant and quality tags, which hampers their performance due to the imbalance of training resources. this can be seen in the case of es-czn, where the model is unable to learn the variant of the test set due to the unbalanced amount of that variant in the training data (only 5%). ## conclusions in this paper, we have presented our contribution to the americasnlp 2023 shared task. we have described our efforts in terms of data collection and processing. we presented our 6 submissions to the task for all language pairs. we explore various setups for multilingual nmt, including knowledge distillation, transfer learning, multilingual nmt with english, language-specific fine-tuning, and a multilingual modular system. our strongest system follows the same architecture as our winning submission in 2021, which was used as the baseline for this year. there are two main differences between our current submission and the baseline: these two additions have allowed us to beat our own baseline. mengzhou xia, xiang kong, antonios anastasopoulos, and graham neubig. 2019. generalized data augmentation for low-resource translation. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5786-5796, florence, italy. association for computational linguistics. barret zoph, deniz yuret, jonathan may, and kevin knight. 2016. transfer learning for low-resource neural machine translation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 1568-1575, austin, texas. association for computational linguistics.