S heffield’s Submission to the A mericas NLP Shared Task on Machine Translation into Indigenous Languages (2023)

Abstract: The University of Sheffield took part in the shared task 2023 AmericasNLP for all eleven language pairs. Our models consist of training different variations of NLLB-200 model on data provided by the organizers and available data from various sources such as constitutions, handbooks and news articles. Our models outperform the baseline model on the development set on chrF with substantial improvements particularly for Aymara, Guarani and Quechua. On the test set, our best submission achieves the highest average chrF of all the submissions, we rank first in four of the eleven languages, and at least one of our models ranks in the top 3 for all languages.

## introduction the 2023 americasnlp shared task @xcite involves developing machine translation systems for translating from spanish to eleven low resource indigenous languages: aymara (aym), bribri (bzd), asháninka (cni), chatino (czn), guarani (gn), wixarika (hch), nahuatl (nah), hñähñu (oto), quechua (quy), shipibo-konibo (shp), and rarámuri (tar). developing machine translation systems for these languages is challenging since many of them are polysynthetic (i.e., words are composed of several morphemes) and word boundaries are not standardized; they present different orthographic variations (e.g., classical vs. modern nahuatl variations); presence of codeswitching is common, among other difficulties of low resource settings. previous work has explored the effectiveness of pretrained machine translation models in low resource settings @xcite showing their impact on improving translation quality and addressing data scarcity challenges. following this approach, our submissions to the 2023 americas-nlp shared task consist of extending and finetuning various versions of @xcite , a state-of-the-art machine translation model specifically designed for low resource settings. nllb-200 is trained on 202 languages across 1 220 language pairs, including three of the languages present in the americasnlp shared task: aym, gn, and quy. 2 we further train our models on data from various sources such as constitutions and news articles, and we leverage multilingual training and ensembling to improve their performance. models are evaluated using chrf @xcite , the official metric of the task. on the test set, we achieve the highest average chrf across all languages, and the best chrf for four of the languages. the rest of the paper is organised as follows: section 2 describes the data sources for training our models, section 3 explains our three submissions in detail, section 4 presents the results on the dev and test sets, section 5 analyses the impact of different factors to the model's performance, section 6 looks at zero-shot capabilities, and we draw conclusions in section 7. ## models to tackle the 2023 americasnlp task on automatic translation of eleven low resource indigenous languages, we use @xcite , a state-of-the-art machine translation model specifically designed for low resource settings. we experiment with different distilled versions of nllb-200 with 600m and 1.3b parameters, and the version with 3.3b parameters. although inference results on three languages 3 show that the largest version, nllb-3.3b, performs better than smaller versions (see submission 3 we train nllb-200 1.3b distilled on the training data foot_2 and we choose the best checkpoint based on average chrf across all languages. we submit translations for all languages using this model (nllb-1.3b (single best)). submission 2 we take the best-performing single model per language, excluding ensembles. we find that for the majority of languages, the best single model (by dev chrf) is the same as submission 3, so we only submit additional translations for five languages: ## additional experiments we provide the results of additional experiments to better understand the impact of various factors to our model's performance. the results of these experiments are shown in ## zero-shot performance we investigate whether our models have any zeroshot capabilities, i.e. translating a language pair for which the model has not seen any training data. for this, we take the best-performing model for es-shp (nllb-1.3b + backtrans 2), and evaluate it on translating quy-shp, aym-shp, and gn-shp. 5 the results of these experiments are shown in ## conclusions in this paper we describe our submissions to the americasnlp 2023 shared task. we participated with three submissions which consist of training different versions of the nllb-200 model on publicly available data from different sources. models are trained in a multilingual fashion and we experiment with different ensembles of models to further improve performance. we improve upon the inference scores for nllb-200 3.3b for its three supported languages, and our best submission achieved the highest average chrf across all languages of any submission to the task.