Findings of the A mericas NLP 2024 Shared Task on Machine Translation into Indigenous Languages (2024)

Abstract: This paper presents the findings of the third iteration of the AmericasNLP Shared Task on Machine Translation. This year’s competition features eleven Indigenous languages found across North, Central, and South America. A total of six teams participate with a total of 157 submissions across all languages and models. Two baselines – the Sheffield and Helsinki systems from 2023 – are provided and represent hard-to-beat starting points for the competition. In addition to the baselines, teams are given access to a new repository of training data which consists of data collected by teams in prior shared tasks. Using ChrF++ as the main competition metric, we see improvements over the baseline for 4 languages: Chatino, Guarani, Quechua, and Rarámuri, with performance increases over the best baseline of 4.2 ChrF++. In this work, we present a summary of the submitted systems, results, and a human evaluation of system outputs for Bribri, which consists of both (1) a rating of meaning and fluency and (2) a qualitative error analysis of outputs from the best submitted system.

## introduction though the field of natural language processing (nlp) has seen a steep increase in interest and impressive performance improvements over the past decade, a large performance gap still remains between a handful of so-called "high-resource," mostly colonial, languages and the remaining majority of the world's languages @xcite . the indigenous languages of the americas exemplify this reality, representing nearly 15% of the world's linguistic diversity @xcite ) and yet, until recently, receiving little attention in nlp research. the americasnlp shared task on machine translation (mt), now in its third iteration @xcite , is focused on pushing the performance of mt on this group of languages through two main avenues: by applying modeling and architectural advancements, and through the creation of new linguistic resources which support the training and evaluation of these systems. this year's shared task continues to focus on the eleven indigenous languages from the last competition. while this year's competition does not feature new data for evaluation, competitors are given access to a new repository of training data which extends the original set of parallel examples with additional data collected by teams in prior years. this repository represents the first step in creating a new living source of data which can grow through contributions from teams participating in future iterations of the shared task. this year's competition also features two baselines: the university of sheffield @xcite and university of helsinki @xcite systems which each achieved the best performance for a subset of languages in 2023 @xcite . these baselines are strong and hardto-beat; across 157 submissions from 6 different teams, we see improvements for only 4 of the 11 languages: chatino, guarani, quechua, and rarámuri. as two of these four languages are the relatively highest-resourced, this finding may indicate that we are approaching a plateau in performance gains achievable purely through modeling and architectural approaches; therefore, a focus on collecting additional training data may yield the most future improvements. the paper is structured as follows. in section 2, we provide a brief overview of the data and languages provided by the organizers at the beginning of the competition. section 3 contains summary descriptions of the approaches used by each team. section 4 discusses the results of the competition. in section 5, we conduct a human evaluation of system outputs for bribri. in the first part of this evaluation, we follow the prior shared tasks in quantitatively rating a sample of outputs on two axes: meaning and fluency. for the second part, we conduct a qualitative error analysis, comparing baseline systems to the best submitted system. in section 6, we conclude with a brief discussion of future directions in improving mt quality for indigenous languages of the americas. ## data and languages the shared task features 11 indigenous languages of the americas. the language direction we are interested in is from spanish into the low-resource language. we use the americasnlp 2021 data for development and evaluation. it consists of a multi-way parallel dataset of the spanish xnli test set into 10 languages of the americas (asháninka, aymara, bribri, guarani, nahuatl, otomí, quechua, rarámuri, shipibo-konibo, and wixarika). the task also includes chatino, for which the data comes from mexican court proceedings. chatino was introduced as a surprise language in last year's edition @xcite . for an in-depth review of development and evaluation data, please refer to @xcite and @xcite . for training data, besides the data used in previous editions, this year we include the data collected by de @xcite as part of their helsinki-nlp submission. this consists of extra data, made up of different sources listed in their system description paper, as well as syn, which refers to synthetic data obtained through backtranslation. we publicly release the training and development data in our github repostitory. foot_0 ## metrics for evaluation, we use the automatic metric chrf++ @xcite as implemented in sacrebleu @xcite . it is an overlap-based metric at the character-level, which is adequate for our task since most languages are morphologically rich. while teams are not required to submit a system for all languages, the final score for each submission (chrf++ column in ## baselines and submitted systems in this section, we describe the 2024 baseline systems and each team's approach. we present a summary of all approaches in ## results the overall ranking for the shared task can be found in the first place in the shared task, across all eleven language pairs, is awarded to the nordi-calps team (submission 1). their overall score significantly surpasses those of the second and third place teams, dc_dmv and uedin, respectively. notably, only three of the six teams submit entries for all eleven languages. nordicalps secures the top performance on five language pairs (spanish to asháninka, chatino, wizarika, nahuatl, and otomí), although they only exceed the baseline for chatino. similarly, the second-ranked team, dc_dmv, leads for four language pairs (spanish to aymara, bribri, shipibo-konibo, and rarámuri) but surpasses the baselines solely for rarámuri. these results highlight the importance of meticulous pipeline design for data preprocessing and segmentation, as implemented by nordicalps and the use of large multilingual models (nllb) for finetuning, as employed by dc_dmv, for achieving robust results across most language pairs. finally, the bsc team, which participates for only two language pairs, spanish to guarani and quechua, achieves the highest performance on both, surpassing the established baselines. their strategic focus on finetuning a large multilingual model (nllb) and gathering new data for these languages is key to their success. ## human evaluation following prior americasnlp shared tasks @xcite , we also conduct a human evaluation of system outputs, focusing on bribri. ## future directions in this section, we briefly discuss several possible future directions for the americasnlp shared task, given the results from the current as well as prior competitions. evaluation data one bottleneck in the advancement of language technologies for low-resource, and particularly indigenous, languages is the availability of evaluation data. high quality, gold standard data in target low-resource languages supports many important roles in the nlp research pipeline. first, and most importantly, it is the single resource which is necessary for experimentation; without held out data for evaluation, there cannot be any idea of how well a system performs for a given language. second, the domain and source of data is important, as, over time, models are created to perform best on the data they are evaluated on. particularly for low-resource languages, where there may not be great diversity in available data, it becomes vital to consider what data is used for evalu- ms bribri english translation 4 t ö, be' én a iàna tö ie' dör bua'ë. yes, you know she was great. t ö, be' wa i ujchen tö ie' bák bua'. yes, you know[sic] that she was good. 3 hum, uk öki sa' mìneyal ù páli a. hm, afterwards we moved to a new house. um, e' uk öki sa' e' mìne ù pâali a. hm, after that we went us to a new[sic] house. 2 ramona ta ye' ujté skàne i spoke to ramona again. ye' ujté ramona tamalé. i spoke, ramona, [cuajiniquil] fruit [sic]. 1 ye' tso' kanè maúk èwewa semana i ët wa. i am finishing with my project for next week. ye' tso' kanèbalök ènuk móköl i ëk. i am working[sic]. finish. other[sic] weapon. additional training data this iteration of the shared task marks the first where performance did not increase for the majority of languages in the shared task. of the four languages which did see improvements, two are relatively high-resource and have recently been included in large pretrained models @xcite . as such, additional data for training likely plays a large role in improving the performance for these languages. while teams continue to find new digital data for training, other non-digital sources may need to be considered for future systems. language identification one of the main bottlenecks for gathering additional data is that every process of collecting resources from online sources starts with a good language identifier. investing efforts into developing a language identification system for the shared task languages could boost the collection of additional training data. new language pairs the performance of lowresource language pairs in multilingual mt models can benefit from incorporating additional data from other language pairs. furthermore, our goal is to expand the scope of our shared task in future editions to include more underserved languages of the americas. to achieve this, we plan to engage more researchers who have developed and published resources for the indigenous languages of the americas, both at our workshop and in other venues. ## conclusion in this work, we present the results of the americ-asnlp 2024 shared task on machine translation. overall, 6 teams participated in the shared task, and submitted a combined 157 submissions across all eleven supported languages. prior to the start of the competition, the organizers provided two strong baselines and a training data set which includes data collected from prior submissions. while there were improvements for four languages in this year's shared task, the majority of languages did not see any performance gains over the baselines, which were the strongest systems from 2023.