Applying Linguistic Expertise to LLM s for Educational Material Development in Indigenous Languages (2024)

Abstract: This paper presents our approach to the AmericasNLP 2024 Shared Task 2 as the JAJ (/dʒæz/) team. The task aimed at creating educational materials for indigenous languages, and we focused on Maya and Bribri. Given the unique linguistic features and challenges of these languages, and the limited size of the training datasets, we developed a hybrid methodology combining rule-based NLP methods with prompt-based techniques. This approach leverages the meta-linguistic capabilities of large language models, enabling us to blend broad, language-agnostic processing with customized solutions. Our approach lays a foundational framework that can be expanded to other indigenous languages languages in future work.

## introduction in recent years, there has been a surge of interest in developing natural language processing (nlp) technologies for low-resource languages. this is motivated by the desire to preserve cultural heritage and foster linguistic diversity. among 7,000 living languages on earth, 42% of languages are in danger of disappearing, according to ethnologue foot_0 . in the americas, there are approximately 1,200 languages @xcite , and about 70% of them are classified as endangered or already extinct by the united nations educational, scientific and cultural organization (unesco) @xcite . the endangered languages are losing their vitality, and intergenerational transmission is barely occurring or not occurring at all, which suggests these languages are at a high risk of becoming extinct in the near future, along with the the cultural legacy residing within. while many communities are engaged in revitalization endeavors, such as developing educational resources for language preservation, these initiatives are confronted with significant challenges in terms of cost and time investment. nlp can be one way of ameliorating the situation. the americasnlp 2024 shared task 2 is specifically aimed at creating educational materials for indigenous languages in the americas, such as maya, guarani, and bribri @xcite . these languages each have their own unique linguistic features and challenges, offering a rich opportunity for the application of innovative nlp techniques. in this shared task, we are given a dataset with base sentences and the grammatical changes expected to be applied to each base sentence. based on this information, we will train a system that can transform the base sentences according to the indicated changes. base sentence: ye' shka' (bribri for "i walked") expected change: polarity: negative target sentence: ye' kë shkane (bribri for "i didn't walk") we participated in this task with the goal of developing a method that effectively handles the linguistic complexities of these languages. our hybrid methodology combines rule-based nlp methods with prompt-based techniques, making use of the meta-linguistic capabilities of large language models (llms). this approach enabled us to blend broad, language-agnostic processing with customized solutions that address each language's unique needs. maya largely does not conjugate verbs based on grammatical tense; instead, it uses aspectual marking to express time-related information. the abundance of apparently simple examples with only a single change, presented an interesting challenge. rather than using all the available examples, we experimented with selection methods to find the most syntactically similar example sentences. bribri, on the other hand, had a much more com-plicated system of verb conjugation. using more examples was not as effective as a separate system to handle the intricate verb morphology. this aspect of bribri posed a substantial challenge, further complicated by variations in notation and spelling observed across different educational resources. while our system addresses specific linguistic features of maya and bribri, we encountered time constraints that limited our ability to apply similar dedicated modifications to guarani. consequently, guarani was processed using the base system, without the benefit of language-specific adaptations. recognizing the unique linguistic structure and cultural significance of guarani, we are committed to expanding our system in the future to include tailored strategies that cater specifically to its needs, enhancing the model's overall effectiveness. ## prior work our approach is inspired by previous work on rosetta stone puzzles @xcite . these puzzles are carefully designed to encourage solvers to deduce implicit linguistic features from a low-resource language, relying solely on a limited set of bilingual sentence pairs, one of which is a low-resource language with uncommon linguistic features. leveraging the inferred linguistic patterns, solvers are tasked with translating new sentences in both directions, which is in essence a few-shot translation task. şahin et al. (2020) explored the resolution of rosetta stone problems using a range of algorithms, from random word selection to transformerbased language models. @xcite conducted similar experiments, specifically focusing on assessing the metalinguistic awareness of pretrained language models. their findings not only underscored the limitations of various algorithms in tasks requiring linguistic inference skills but also outlined directions for advancing machines' comprehension of human language. recent advancements in the application of llms to rosetta stone puzzles have shown encouraging outcomes, demonstrating the potential of llms to extract and apply linguistic knowledge from limited data @xcite . this research supports the feasibility of our approach, suggesting that llms can adeptly adapt to the intricacies of low-resource languages by leveraging their inherent capabilities in pattern recognition and language comprehension. another relevant research area is the machine translation from one book (mtob) task @xcite . in this task, models are trained to translate between english and kalamang using only a single kalamang textbook. llms seem to be breaking into this space as well, as shown by gemini 1.5l's state-of-the-art performance on this task. the mtob task is particularly pertinent to our work because it demonstrates the potential of machine learning models to operate effectively even with severely limited datasets, mirroring the conditions we faced in the shared task @xcite . ## results our systems for maya and bribri improved on the baselines provided by the task organizers by considerable amounts. in particular, our system's bribri accuracy was over six times higher than the as indicated in tables 1, maya became more resource efficient with example selection and pos tagging. bribri performance saw the largest boost from verb hints and moderate improvements from mode hints, as shown in our bribri system was the best performing in the competition. our maya system came a close third. due to this, and despite regrettably not submitting guarani results, our contribution was ranked first overall. we submitted guarani after the deadline, and report the results of all three languages on the development and test sets compared to the baseline in ## discussion we submitted our results using gpt-4 for system prompting due to its superior performance on the development set.