Findings of the A mericas NLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the A mericas (2021)

Abstract: This paper presents the results of the 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas. The shared task featured two independent tracks, and participants submitted machine translation systems for up to 10 indigenous languages. Overall, 8 teams participated with a total of 214 submissions. We provided training sets consisting of data collected from various sources, as well as manually translated sentences for the development and test sets. An official baseline trained on this data was also provided. Team submissions featured a variety of architectures, including both statistical and neural models, and for the majority of languages, many teams were able to considerably improve over the baseline. The best performing systems achieved 12.97 ChrF higher than baseline, when averaged across languages.

## introduction many of the world's languages, including languages native to the americas, receive worryingly little attention from nlp researchers. according to glottolog @xcite , 86 language families and 95 language isolates can be found in the americas, and many of them are labeled as endangered. from an nlp perspective, the development of language technologies has the potential to help language communities and activists in the documentation, promotion and revitalization of their languages @xcite . there have been recent initiatives to promote research on languages of the americas @xcite @xcite @xcite @xcite . * *the first three authors contributed equally. the americasnlp 2021 shared task on open machine translation (omt) aimed at moving research on indigenous and endangered languages more into the focus of the nlp community. as the official shared task training sets, we provided a collection of publicly available parallel corpora ( §3). additionally, all participants were allowed to use other existing datasets or create their own resources for training in order to improve their systems. each language pair used in the shared task consisted of an indigenous language and a high-resource language (spanish). the languages belong to a diverse set of language families: aymaran, arawak, chibchan, tupi-guarani, uto-aztecan, oto-manguean, quechuan, and panoan. the ten language pairs included in the shared task are: quechua-spanish, wixarika-spanish, shipibo-konibo-spanish, asháninka-spanish, raramuri-spanish, nahuatl-spanish, otomí-spanish, aymara-spanish, guarani-spanish, and bribri-spanish. for development and testing, we used parallel sentences belonging to a new natural language inference dataset for the 10 indigenous languages featured in our shared task, which is a manual translation of the spanish version of the multilingual xnli dataset @xcite . for a complete description of this dataset we refer the reader to @xcite . together with the data, we also provided: a simple baseline based on the small transformer architecture @xcite proposed together with the flores dataset @xcite ; and a description of challenges and particular characteristics for all provided resources 1 . we established two tracks: one where training models on the development set after hyperparameter tuning is allowed (track 1), and one where models cannot be trained directly on the development set (track 2). machine translation for indigenous languages often presents unique challenges. as many indigenous languages do not have a strong written tradition, orthographic rules are not well defined or standardized, and even if they are regulated, often times native speakers do not follow them or create their own adapted versions. simply normalizing the data is generally not a viable option, as even the definition of what constitutes a morpheme or a orthographic word is frequently ill defined. furthermore, the huge dialectal variability among those languages, even from one village to the other, adds additional complexity to the task. we describe the particular challenges for each language in section §3. eight teams participated in the americasnlp 2021 shared task on omt. most teams submitted systems in both tracks and for all 10 language pairs, yielding a total of 214 submissions. ## languages and datasets in this section, we will present the languages and datasets featured in our shared task. figure 1 additionally provides an overview of the languages, their linguistic families, and the number of parallel sentences with spanish. ## error analysis to extend the analysis in the previous sections, tables 6 and 7 show output samples using the best ranked system (helsinki-5) for shipibo-konibo and otomí, respectively. in each table, we present the top-3 outputs ranked by chrf and the top-3 ranked by adequacy and fluency. for shipibo-konibo, in for otomí, in in summary, there are some elements to explore further in the rest of the outputs: how many loanwords or how much code-switched text from spanish is presented in the reference translation? is there consistency in the punctuation, e.g., period at the end of a segment, between all the source and reference sentences? ## conclusion this paper presents the results of the americasnlp 2021 shared task on omt. we received 214 submissions of machine translation systems by 8 teams. all systems suffered from the minimal amount of data and the challenging orthographic, dialectal and domain mismatches of the training and test set. however, most teams achieved huge improvements over the official baseline. we found that text cleaning and normalization, as well as domain adaptation played large roles in the best performing systems. the best nmt systems were multilingual approaches with a limited size (over massive multilingual). additionally, smt models also performed well, outperforming larger pretrained submissions.