A Comparison of Fine-Tuning and In-Context Learning for Clause-Level Morphosyntactic Alternation (2024)

Abstract: This paper presents our submission to the AmericasNLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages. We frame this task as one of morphological inflection generation, treating each sentence as a single word. We investigate and compare two distinct approaches: fine-tuning neural encoder-decoder models such as NLLB- 200, and in-context learning with proprietary large language models (LLMs). Our findings demonstrate that for this task, no one approach is perfect. Anthropic’s Claude 3 Opus, when supplied with grammatical description entries, achieves the highest performance on Bribri among the evaluated models. This outcome corroborates and extends previous research exploring the efficacy of in-context learning in low- resource settings. For Maya, fine-tuning NLLB- 200-3.3B using StemCorrupt augmented data yielded the best performance.

## introduction the americasnlp 2024 shared task on the creation of educational materials for indigenous languages @xcite focuses on the challenge of clause-level morphosyntactic alternation for low-resource indigenous languages of the americas. the objective of this task is to develop a system capable of applying a set of grammatical attributes to a given source sentence, thereby generating a target sentence with the desired changes. the motivation behind this task lies in the potential for such systems to aid in the preservation and revitalization of endangered languages @xcite . this task involves three indigenous languages of the americas: bribri, guaraní, and maya. two examples are provided below: example 1.1. bribri source sentence: ye'shka' ("i walked") attributes: type:neg (negative polarity) target sentence: ye'kë shkàne ("i didn't walk") example 1.2. maya source sentence: táan in xíimbal tu jáal já' ("i'm walking on the beach") attributes: type:neg (negative polarity) target sentence: ma' táan in xíimbal tu jáal ja'i' ("i'm not walking on the beach") we frame this task as one of morphological inflection generation, treating each sentence as a single word. our objective is thus twofold: to develop a system that performs sentence-level morphological inflection for low-resource indigenous languages of the americas, and to provide insight into what techniques are effective for future practitioners who attempt this task. in pursuit of this goal, we compare the performance of two distinct approaches: fine-tuning pre-trained transformer models and leveraging llms through in-context learning. by evaluating these two approaches, we aim to contribute to the understanding of effective strategies for addressing the unique challenges posed by low-resource languages in tasks such as morphosyntactic alternation. ## background this task is unique, as previous literature has explored morphological inflection generation on the word level rather than on the sentence level @xcite . further, this task is challenging for two reasons: prior results have demonstrated the effectiveness of transformer-based models @xcite on the word-level inflection task @xcite . building upon this success, we evaluate the performance of fine-tuned transformer models on the sentence-level task, exploring their ability to capture and generate morphological inflections in context. in an effort to extend the available data, we search for external sentence-level parallel corpora aligned with the task format. however, our search yields no suitable resources. while it may be possible to preprocess and adapt data from other formats in a separate pre-training stage, this approach is complex and may require a significant time investment for developing custom preprocessing pipelines, which is not possible in our study, given the short time-frame of this shared task. to address the challenge of limited data resources, we opt for data augmentation using stem-corrupt @xcite , generating synthetic instances based on the existing data. stemcorrupt is a data augmentation technique created for generating additional instances for the word-level inflection task. the use of stemcorrupt is motivated by the availability of pre-existing code and the relative simplicity of this technique, which allow us to quickly run data augmentation and focus our efforts on other aspects of the task. the limited supervised data challenge also prompts us to explore the use of proprietary large language models (llms). these models have the capability to process long context windows of arbitrary text as input and do not require finetuning, making them a promising alternative for low-resource settings where extensive task-specific data is unavailable. recent advancements have demonstrated that by scaling training compute and corpus size, llms may excel in tasks for they which they are not explicitly trained @xcite . studies exploring the use in-context learning with llms on lowresource machine translation have shown promising results @xcite . more recent work in the area suggests that when paired with appropriate language resources, llms can even surpass human baselines in translation quality @xcite . these findings highlight the potential of in-context learning in llms for addressing the challenges posed by low-resource languages and the importance of incorporating relevant linguistic knowledge to maximize their performance. ## data for fine-tuning, we use the provided training dataset foot_0 and an augmented dataset that we create by applying stemcorrupt to the provided training dataset. for in-context learning, we experiment with the inclusion of a grammatical description in the prompt. previous work investigating the use of proprietary llms on low resource languages has shown that, when combined with grammatical descriptions, these models obtain strong performance on tasks such as machine translation @xcite . we hypothesize that using grammatical descriptions in an in-context learning setting can improve performance on this task as well. ## experiments we perform four experiments and compare the results: ## future work much future work remains. our search for language resources reveals a wide variety of language resources of varying types and orthographies. a future area of research is an exploration of the effect of different orthographies on the llm performance in an in-context learning setting. additionally, a significant advantage of incontext learning is the reduction of restrictions on data types that can be utilized by the model. exploring the effect of different resource types, such as dictionaries and learning worksheets, would be valuable. a historic bottleneck for the translation or inflection of low-resource languages has been data, specifically gold-standard data that adheres to a specialized format. by leveraging in-context learning, the variety of usable data is greatly increased and can offer opportunities for further exploration. stemcorrupt has shown promise for sentencelevel inflection despite initially being developed for word-level inflection. exploring the feasibility of extending this technique to other languages is a worthwhile future endeavor. ## conclusion in this paper, we present the systems submitted by our team for the americasnlp 2024 shared task on the creation of educational materials for indigenous languages. we find that while llmsthrough in-context learning-exhibit impressive capabilities, fine-tuning still has a role to play in the modern nlp space. moreover, we reaffirm the results of prior literature regarding the promise of llms when applied to low-resource languages using in-context learning. additional work must be done to explore the abilities of such systems, but initial results point to promising potential for the task of morphosyntactic alternation. our work also extends prior literature on stemcorrupt and demonstrates potential applications for the technique on sentence-level inflection generation.