Findings of the A mericas NLP 2023 Shared Task on Machine Translation into Indigenous Languages (2023)

Abstract: In this work, we present the results of the AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages of the Americas. This edition of the shared task featured eleven language pairs, one of which – Chatino-Spanish – uses a newly collected evaluation dataset, consisting of professionally translated text from the legal domain. Seven teams participated in the shared task, with a total of 181 submissions. Additionally, we conduct a human evaluation of the best system outputs, and compare them to the best submissions from the prior shared task. We find that this analysis agrees with the quantitative measures used to rank submissions, which shows further improvements of 9.64 ChrF on average across all languages, when compared to the prior winning system.

## introduction the majority of indigenous languages, including those native to the americas, are under-represented in modern natural language processing (nlp), as technological advances are often concentrated on the small set of languages that have large amounts of easily available data @xcite . beyond the lack of data, linguistic factors like morphological complexity, non-standard orthographies, and language isolates make it even more challenging to adapt existing nlp methods to indigenous languages @xcite . however, there are multiple benefits of developing technologies that support indigenous languages -building nlp models for under-represented languages can bring equitable access to information and technology to speakers of these languages @xcite . additionally, several indigenous languages in the americas are endangered, and language technologies have proven to be beneficial to indigenous communities and linguistic researchers in the documentation, preservation, and revitalization of endangered languages (galla, 2016; @xcite @xcite . the americasnlp workshop seeks to highlight nlp and linguistic research on indigenous languages spoken across the americas, and promote the development of computational approaches which work well for these languages. the americasnlp shared task on machine translation into indigenous languages is hosted as part of the workshop to specifically focus on improvements in machine translation (mt) systems for these languages. in this work, we describe the third edition of the shared task. for this year, a new goldstandard parallel dataset for translation evaluation, between spanish and chatino, was developed. this dataset uses text from the legal domain, with source sentences taken from press releases of the supreme court of mexico. this allows for evaluation on technical and challenging text, which are likely to be relevant to speakers of the language. this work is structured as follows: in section 2, we present a brief overview of related work on mt and indigenous languages; in section 3 and 4, we provide details on the shared task rules, and newly collected data; in section 5, we summarize the submitted systems; and, in sections 6 and 7, we provide an analysis of the main results and further team andes cic-nlp helsinki-nlp* lct-ehu ltlamsterdam playground sheffield* langs 1 11 11 1 11 10 11 subs 1 33 66 5 33 10 33 data crawl ✓ ✓ ext. bilingual ✓ ✓ ✓ opus ✓ religous ✓ ✓ ✓ wikipedia ✓ prior year ✓ ✓ ✓ ✓ no addtl. ✓ monolingual trans ✓ ✓ ✓ ✓ pivot trans. ## task and evaluation the shared task focuses on open machine translation: outside of the development set and any prohibited datasets, teams are allowed to collect and train on an unlimited amount of external data. as translation performance for low-resource indigenous languages is generally low, we choose this setting to allow models to achieve the best possible performance, in hopes that usable translation models become more quickly developed. metrics translation evaluation is done with chrf @xcite , as implemented in scarebleu @xcite , as the target languages are morphologically rich. while teams are not required to submit a system for all languages, the final score for each submission is calculated by taking an average over all eleven languages; if there is no model output for a given language, the score is taken as 0. ## languages and data for development and evaluation, the americasnlp 2021 shared task used multi-way parallel translations of the spanish xnli test set across 10 languages: asháninka, aymara, bribri, guarani, nahuatl, otomí, quechua, rarámuri, shipibo-konibo and wixarika @xcite . for this edition of the shared task, we use the same evaluation set and additionally introduce a new evaluation dataset, created from mexican court proceedings, for spanish-chatino. this set was released as a surprise language near the end of the competition, along with a small amount of spanish-chatino and english-chatino data for training. in this section, we describe the chatino language, spanish source data, and translation process. for a detailed overview of the ten other evaluation languages, we refer the reader to @xcite and @xcite . ## baseline and submitted systems in this section, we describe the 2023 baseline system and each team's approach. we present a summary of all approaches in ## results we present the overall ranking of submissions to the shared task in the overall winner of the shared task, the sheffield submission 1, achieves the best performance for 7 languages: aymara, bribri, asháninka, chatino, nahuatl, quechua, and shipibo-konibo. the helsinki submission 6 (i.e., model b) has the highest performance for 4 languages: guarani, wixarika, otomí, and rarámuri. systems are much more competitive than prior competitions, achieving extremely close chrf scores for many languages, such as asháninka, guarani, wixarika, and shipibo-konibo. the sheffield and helsinki teams both collect additional data, and train models in a multilingual and multi-stage fashion. both also mention data cleaning and preprocessing in their pipeline, and we hypothesize that this step is likely vital for good performance, due to noise, domain mismatch, and differences in variants between the training and evaluation sets. for all languages except for aymara, all teams have at least one submission which improves (often by a large margin) over the original 2021 baseline. comparison with prior years as the evaluation set for 10 of the languages is the same as for 2021, we can analyze the performance of submitted mt systems over time. in this year's shared task, we see improvements over the best 2021 system, the 2021 helsinki submission @xcite , for all languages, but to varying degree. the largest improvements are for bribri, aymara, guarani and quechua. we also see small improvements for asháninka and wixarika. however, improvements for nahuatl, otomí, and shipibo-konibo are marginal. overall, the improvements over @xcite are smaller in magnitude, compared to the improvements in 2021. this can be expected, however, as the baseline for this year's shared task represents a much stronger lower bound. of the four languages with largest improvement, three are achieved by a sheffield submission: aymara, bribri, and quechua. this may be attributed, in part, to the use of the nllb model by the team, which supports aymara and quechua in its original set of pretraining languages. on average across the 10 shared languages, we see a further 9.63 improvement in chrf over 2021 results by the best submitted systems. ## conclusion in this paper we present the results of the amer-icasnlp 2023 shared task. for this iteration, we collect a new dataset for translation evaluation between spanish and chatino, consisting of legal text from court press releases. additionally, we keep the prior 10 evaluation languages used in 2021. overall, 7 teams participated in the shared task. for all languages, multiple submissions improve over the previous best chrf, but the magnitude varies per language. the best results were achieved by either finetuned versions of nllb or a from-scratch transformer encoder-decoder model. to confirm the improvement in chrf from the previous shared task, we conduct a human evaluation of system outputs, which, although it supports the quantitative improvement, highlights the fact that systems are still not able to produce translations of the highest quality. furthermore, there is still variability in the absolute performance across languages. as such, while the results of the shared task mark a promising trend in increasing translation quality for indigenous languages, there are still improvements which can be made in order to create usable translation systems for indigenous languages.