Can we teach language models to gloss endangered languages? (2024)

Abstract: Interlinear glossed text (IGT) is a popular format in language documentation projects, where each morpheme is labeled with a descriptive annotation. Automating the creation of interlinear glossed text would be desirable to reduce annotator effort and maintain consistency across annotated corpora. Prior research has explored a number of statistical and neural methods for automatically producing IGT. As large language models (LLMs) have showed promising results across multilingual tasks, even for rare, endangered languages, it is natural to wonder whether they can be utilized for the task of generating IGT. We explore whether LLMs can be effective at the task of interlinear glossing with in-context learning, without any traditional training. We propose new approaches for selecting examples to provide in-context, observing that targeted selection can significantly improve performance. We find that LLM-based methods beat standard transformer baselines, despite requiring no training at all. These approaches still underperform state-of-the-art supervised systems for the task, but are highly practical for researchers outside of the NLP community, requiring minimal effort to use.

## introduction with thousands of endangered languages at risk of extinction, language documentation has become a major area of linguistic research @xcite , aiming to produce permanent artifacts such as annotated corpora, reference grammars, and dictionaries. furthermore, research has explored the potential for computational methods to aid in language documentation and revitalization @xcite @xcite ; gessler, 2022; @xcite @xcite . in particular, we study the task of generating interlinear glossed text (igt), a line-by-line format for annotated text corpora that is commonly used in documentation projects. igt generation has been studied using statistical @xcite @xcite and neural @xcite @xcite methods. a key challenge when working with endangered languages is that, in nearly all cases, foot_0 there is very little labeled or unlabeled data available. this is particularly challenging for large neural models which depend on large, representative training data sets. research has explored methods to overcome this challenge for igt generation systems, such as crosslingual transfer @xcite @xcite and architectural modifications @xcite , but these approaches struggle in very low-resource scenarios. in addition, previous approaches generally require expertise in model training, implementation, and deployment, as well as the computational resources needed to serve large neural models. as large language models (llms) have demonstrated impressive performance on various natural language tasks, the question arises whether they can benefit language documentation. we seek to evaluate the ability of current llms to generate interlinear glossed text, compared with earlier state-of-the-art methods. this research can also shed light on the language-agnostic capabilities of llms, requiring the model to learn patterns in very rare languages which are unlikely to have significant presence in their training data. we study strategies for selecting in-context examples, finding significant impacts to performance. our best-performing systems outperform transformer model baselines, despite involving no training whatsoever. they still underperform sota systems that induce morphological segmentation, but at the same time hold promise for offering a new approach to interlinear glossing for language documentation practitioners. our code is available on github. 2 2 background ## methodology we study the igt generation task described in @xcite . given a transcription line and translation line, systems must predict the gloss line. we focus on the closed track setting, where the input words are not segmented into morphemes. this task is strictly more difficult than the setting where words are already segmented, as models must jointly learn segmentation and gloss prediction. as reported in @xcite , the sota on this task remains far weaker than the setting with segmented inputs, with up to a 40 point discrepency in sota performance. ## many-shot prompting few-shot prompting, where a model is provided with a small number of examples in the context, has proven very effective at a variety of tasks @xcite @xcite . furthermore, as model context lengths have continued to increase, it has become possible to provide hundreds or even thousands of examples, and performance typically continues to improve @xcite . on the other hand, increasingly long prompts bear a high cost, and strategies to retrieve relevant examples can often achieve similar performance at a fraction of the cost (see section 5). ## retrieval strategies while including a large number of in-context examples can certainly improve performance, long prompts carry a high cost that may be infeasible for real-world documentation projects. for example, running prompts with a thousand examples in uspanteko costs roughly 10 cents per inference call, which can quickly add up over thousands of examples. many llms still have limited context length, particularly among open-source models, and including many examples may not even be possible. finally, @xcite suggests that the effectiveness of many-shot prompting is mainly due to the model seeing relevant examples, and ignoring many irrelevant ones. with this in mind, we consider a method inspired by retrieval-augmented generation (rag, @xcite . rag was originally used for knowledge-intensive tasks, using document embeddings to search for relevant documents to a given query and include them in prompt context. we apply a similar strategy in order to search for relevant igt examples from our training corpus to include in our prompt. ## conclusion we find that sota large language models struggle to produce interlinear glosses for the endangered languages used in our research. however, by selecting relevant examples from a training corpus and providing them as part of the context for each example to be glossed, we can significantly improve performance. we find that the relationship between performance and the number of few-shot examples is roughly logarithmic. performance improves by a wide margin when we select examples with a high chrf++ score relative to the target sentence. our best systems outperform a standard transformer model, despite involving no explicit training and using a fraction of the training data. however, they still underperform the sota system for the glossing task on three out of four languages. thus, for documentary linguists hoping to use automated glossing solutions, the use of llms may not achieve ideal accuracy. at the same time, llms may still be a preferrable choice for languages with very limited data comparable to gitksan, and the use of an api is often far more accessible than training and hosting a neural model. our results encourage further exploration of this approach.