JGU Mainz’s Submission to the A mericas NLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages (2024)

Abstract: In this paper, we present the four systems developed by the Meenzer team from JGU for the AmericasNLP 2024 shared task on the creation of educational materials for Indigenous languages. The task involves accurately applying specific grammatical modifications to given source sentences across three low-resource Indigenous languages: Bribri, Guarani, and Maya. We train two types of model architectures: finetuning a sequence-to-sequence pointer-generator LSTM and finetuning the Mixtral 8x7B model by incorporating in-context examples into the training phase. System 1, an ensemble combining finetuned LSTMs, finetuned Mixtral models, and GPT-4, achieves the best performance on Guarani. Meanwhile, system 4, another ensemble consisting solely of fine-tuned Mixtral models, outperforms all other teams on Maya and secures the second place overall. Additionally, we conduct an ablation study to understand the performance of our system 4.

## introduction natural language processing (nlp) serves as a valuable educational tool for facilitating the learning of (endangered) languages. one effective method for generating learning material involves a system automatically transforming sentences based on specific properties. subsequently, language learners are tasked with replicating the transformation, thus reinforcing their understanding of the language structure. the americasnlp 2024 shared task on the creation of educational materials for indigenous languages (st 2) @xcite focuses on creating such material for three lowresource indigenous languages: bribri, guarani, and maya. participants are tasked with applying a specific grammatical property to a given source sentence and producing the accurate modification. our systems (which we submitted under the name "meenzer team") are ensembles composed of a range of models: finetuned character-level pointer-generator lstms @xcite , finetuned mixtral 8x7b large language models (llms) @xcite via training on in-context examples, and @xcite . the main metric of the shared task is accuracy. we outperform all teams on guarani by employing an ensemble across all models. additionally, our ensemble of finetuned mixtral models achieves the highest performance on maya and reaches the second place overall. the remainder of this paper is organized as follows: section 2 details the task at hand and introduces the provided data. following that, section 3 dives into the details of our four system submissions. section 4 presents the outcomes observed on both the development and test sets of the shared task. lastly, an ablation study on our best performing system is provided in section 5. ## meenzer team's system our systems consist of ensembles comprising various models, including finetuned character-level pointer-generator lstms, finetuned mixtral 8x7b llms utilizing in-context finetuning, and gpt-4. ## results the primary metric for evaluating the shared task performance is accuracy (acc.), supplemented by bleu @xcite and chrf @xcite as additional metrics. we present the results for the development set and test set in ## ablation study in this section, we conduct a brief ablation study on our best-performing system, (4) mixtrals. the results on the development set are presented in ## conclusion we presented the systems of the meenzer team by jgu for the americasnlp 2024 shared task on the creation of educational resources. we trained character-level pointer-generator lstms as well as mixtral 8x7b models finetuned through sicl. in addition, we used gpt-4 models via in-context learning. we secured second place with an ensemble of the finetuned mixtral 8x7b models and reached the highest accuracy on maya. additionally, we achieved the highest performance on guarani using an ensemble of lstm, mixtral, and gpt-4 models.