Findings of the A mericas NLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages (2024)

Abstract: This paper presents the results of the first shared task about the creation of educational materials for three indigenous languages of the Americas.The task proposes to automatically generate variations of sentences according to linguistic features that could be used for grammar exercises.The languages involved in this task are Bribri, Maya, and Guarani.Seven teams took part in the challenge, submitting a total of 22 systems, obtaining very promising results.

## introduction the americasnlp 2024 shared task on the creation of educational materials for indigenous languages is a competition aimed at encouraging the development of natural language processing systems (nlp) to help with the teaching and diffusion of indigenous languages of the americas. many of the indigenous languages of the americas are vulnerable or endangered. this means that, depending on the language, no or only a few children are learning them and, generally, they are only spoken by a few small groups of people. because of this, some of these languages are at a high risk of becoming extinct in the near future. many communities are carrying out revitalization efforts, including teaching their languages to their community members. creating materials to teach these languages is an urgent priority, but this process is expensive and time consuming. nlp presents an opportunity to help with these efforts. in addition to being endangered, the indigenous languages of the americas are so-called low-resource languages @xcite : the data needed to train any nlp systems, let alone deep learning-based systems, is severely limited. this means that many approaches used for highresource languages, such as english and chinese, are not directly applicable or perform poorly. on top of this, many indigenous languages exhibit linguistic properties uncommon among languages frequently studied in nlp. this constitutes an additional difficulty. in this task, participants built systems for transforming sentences in an indigenous language according to some linguistic feature (such as negation or tense), in a way that could enable to automatically create grammar exercises. this often implies inflecting the main verb of the sentence, but other types of changes could be necessary as well, such as including different adverbs or particles, or making adaptations according to agreement rules. we hope that this challenge helps to motivate researchers to develop systems for these indigenous languages, as well as spark the interest in nlp research for the huge diversity of languages across the american continent, as is the goal of the ameri-casnlp workshop since its inception @xcite . ## task description the idea of this shared task is to automatically convert sentences in indigenous languages into small exercises for language learners. in particular, we aim to create grammar exercises in which students must tweak a sentence changing its tense, aspect, or other morphosyntactic features. in order to do this, participants have to create systems that can automatically modify sentences with regard to a given property (e.g., they must create a negated version of a sentence). those sentences could then be used as exercises by either asking learners to do the same transformation or by masking out all changed words in the sentence and asking learners to fill in the blank. for instance, if a model can correctly reproduce the linguistic labels, it will also be capable of transforming simple sentences from first-person singular to first-person plural, as in the following example in maya: original sentence: j-jaan en tin najil (1s) tr. i ate at my home. transformed sentence: j-jaano'ob tu najil (1p) tr. we ate at his/her/their house. using that pair of sentences, we could come up with the grammar exercise below. exercise 1. transform the following sentence to first-person plural: task format the participants were provided with one data file for each language, containing the following columns: systems were expected to take the original sentence and a morphosyntactic feature marker, and generate the target sentence as output. internally, the examples were organized in clusters in which, starting from one original sentence, one or more morphosyntactic variations (deltas) were created. the task was evaluated in terms of exact accuracy (fraction of times the system output matched the expected output), and also two classic metrics for generative tasks: bleu @xcite and chrf @xcite . the main metric for the task was exact accuracy. ## dataset ## approaches and results this section describes the different approaches that the participants used to solve the task, as well as the baseline approach we implemented, and then presents the results obtained by these approaches. seven teams took part in the shared task, submitting a total of 22 systems. all seven teams submitted results for the bribri and maya languages, while for guarani only four teams presented results. the methods of both systems by the anonymous submission are not known. ## conclusions we presented the results of the first task on the creation of educational materials for indigenous languages of the americas. in this task, the participants had to create systems that could transform a source sentence into a target sentence by changing some linguistic feature, usually associated to the main verb (e.g., negation, aspect or tense). these pairs of sentences can be used to create grammar exercises for students of the indigenous languages. the languages targeted in this task were bribri, maya and guarani, three indigenous languages of the americas with different characteristics. seven teams took part in the task, submitting 22 systems. different teams obtained the best results for each language: jaj for bribri, giving it a shot for maya, and meenzer team for guarani. the results in general were very promising, obtaining high scores in terms of the generative task metrics bleu and chrf, but still with a lot of room for improvement in terms of the main accuracy metric. notably, most of the teams used neural methods, in particular llms like gpt-4 or mixtral, often with some strategies for data augmentation. this is interesting because such models have often shown worse performance on lower-resource languages than those with higher resources, but in this case where the systems did not need to generate a full sentence but make some localized changes, they seem to work quite well.