Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing (2024)

Abstract: In this paper, we address the data scarcity problem in automatic data-driven glossing for low-resource languages by coordinating multiple sources of linguistic expertise. We enhance models by incorporating both token-level and sentence-level translations, utilizing the extensive linguistic capabilities of modern LLMs, and incorporating available dictionary resources. Our enhancements lead to an average absolute improvement of 5%-points in word-level accuracy over the previous state of the art on a typologically diverse dataset spanning six low-resource languages. The improvements are particularly noticeable for the lowest-resourced language Gitksan, where we achieve a 10%-point improvement. Furthermore, in a simulated ultra-low resource setting for the same six languages, training on fewer than 100 glossed sentences, we establish an average 10%-point improvement in word-level accuracy over the previous state-of-the-art system.

## introduction the extinction rate of languages is alarmingly high, with an estimated 90% of the world's languages at risk of disappearing within the next century @xcite . as speech communities dwindle, linguists are urgently prioritizing the documentation of these languages. this is a multi-step process involving: 1. phonetic and orthographic transcription, 2. translation into a so-called matrix language like english or spanish, which provides a common frame of reference for all annotations, 3. morpheme segmentation, and 4. grammatical annotation @xcite . the end-result is represented as interlinear glossed text (igt) like the gitksan example below (see appendix a for additional details): orthography: ii hahla'lsdi'y goohl ibm segmentation: ii hahla'lst-'y goo-hl ibm gloss: ccnj work-1sg.ii loc-cn ibm translation: and i worked for ibm. the traditional manual approach to language documentation, while thorough, is notably laborintensive. this has spurred the development of automated tools leveraging machine learning for tasks such as word segmentation and glossing. for example, @xcite train neural models for automatic glossing of lezgi, a nakh-daghestanian language. their models deliver reasonable performance when trained on a small training set of 3,000 glossed tokens of lezgi text. however, neural models are data-hungry and the small training set prevents the models from reaching their full potential. the most straightforward way to improve model performance would be to manually gloss more training data. however, as stated above, manual glossing is a very time-consuming process. therefore, additional data sources should be considered. figure 1 : when glossing input such as the french sentence le chien aboie, our system utilizes multiple information sources: an english sentence-level translation, general linguistic knowledge provided by an llm and dictionary definitions for the input tokens. many recent glossing approaches @xcite exclusively train on glossed source language transcripts. however, we often have access to additional helpful knowledge sources. one option is to augment the data using translations of the training examples into the matrix language. 1 these provide an important source of lexical information because the gloss of nouns and verbs can often be found within the translation. 2 because translation is a part of the language documentation process, these are often readily available and, thus, represent a quick and cost-effective way to provide an additional source of supervision. our system incorporates translations as an added information source. unfortunately, the availability of translations for igt data is necessarily limited simply because the quantity of igt data itself is limited. as an additional source of lexical information, our system incorporates external dictionaries which provide word-level translations of target language lexemes into the matrix language. this helps the system generalize to words missing from the training data. recently, powerful pretrained models have emerged as a viable approach to strengthen and supplement the training signal for nlp tasks in low-resource settings @xcite @xcite . advancements in large language models (llms) also present new opportunities for enhancing the language documentation process. pretrained language models such as bert @xcite and llms like gpt-4 @xcite , trained on billions of tokens of text, encode extensive lexical and linguistic knowledge in the matrix language, and their incorporation has improved the benchmarks in many natural language tasks @xcite @xcite . we integrate llms into our glossing pipeline as a post-correction step through in-context learning. it is worth noting that our approach does not require fine-tuning and is, therefore, appropriate in low-resource settings where compute capacity is limited. by leveraging three external sources of information (see figure 1 ): utterance translations, external dictionaries and llms, our glossing pipeline achieves an average absolute improvement of 5%points over the previous state-of-the-art on datasets from the sigmorphon 2023 shared task on interlinear glossing @xcite . in particular, the incorporation of dictionaries leads to significant advancements for ultra-low resource languages such as gitksan, resulting in a 10%-points increase in word-level accuracy. our key contributions are: 1. we enhance the training of glossing systems-in addition to plain glossed training examples, we introduce additional supervision in the form of input translations which are encoded using a pre-trained language model. 2. we utilize external dictionaries which improve glossing performance, particularly for the lowestresourced languages. 3. we pioneer the use of llm prompting and incontext learning techniques as a post-correction step in the glossing pipeline. to our knowledge, this is the first time llms have been applied to the automatic glossing task. our findings show that in-context prompting results in substantial improvements, especially when very limited training data is available. ## related work interlinear glossing research into automatic glossing starts with rule-based analysis @xcite followed by datadriven neural models @xcite @xcite . more recently, the integration of pre-trained multilingual models @xcite has shown great potential to aid documentation projects. our work is inspired by the success of these powerful models and aims to build upon their strengths. ## data we conduct experiments on data from the 2023 sigmorphon shared task on interlinear glossing @xcite . the shared task provides two distinct tracks: an open track, where the input is morphologically segmented, and a closed track, where no segmentations are provided. our analysis focuses on data from the closed track. this setting is substantially more challenging because morphological segmentation now, effectively, becomes a part of the glossing task. the closed-track languages are arapaho (arp), gitksan (git), lezgi (lez), natügu (ntu), tsez (ddo), and uspanteko (usp). 5 data details are shown as in ## baseline model our glossing system is based upon a neural glossing model developed by @xcite . this is the winning system of the 2023 sigmorphon shared task on internlinear glossing. as shown in figure 2 , the model accomplishes glossing of morphological segments through a three-stage process: input encoding, unsupervised morpheme segmentation, and morpheme classification. ## our methods our glossing system enhances the baseline model by incorporating utterance translations (both at the sentence level and token level) and a characterbased decoder. 6 model and training details are provided in appendix b. additionally, we implement a gloss post-correction component using llmpowered in-context learning. figure 3 presents an overview of the system. ## experiments and results in all experiments, we evaluate based on word-level glossing accuracy. ## limitations the limitations of our study primarily pertain to the extent of our experimentation and the models we have chosen. firstly, our investigation relies solely on an lstm decoder. this decision was influenced by time constraints, which limited our ability to explore more complex decoders. additionally, our experimentation is confined to the t5-large model. while this model has shown promising results in our study, we acknowledge the existence of other large language models in the field of natural language processing. although we did explore other large language models such as llama-2 @xcite , our preliminary experiments yielded unsatisfactory results compared to t5. consequently, we made the decision not to include llama-2 in our paper due to its inferior performance. these limitations underscore the need for future research to explore a wider range of decoding architectures and incorporate various large language models to enhance our understanding of the subject matter. however, using large language models requires significant computational resources, which can have an environmental impact due to increased energy consumption.