G loss LM : A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text (2024)

Abstract: Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages.Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6%. Our pretrained model and dataset are available on Hugging Face: https://huggingface.co/collections/lecslab/glosslm-66da150854209e910113dd87

## introduction with nearly half of the world's 7,000 languages considered endangered, communities of minoritized language speakers are working to preserve and revitalize their languages @xcite . these efforts often involve collection, analysis, and annotation of linguistic data. annotated text can be used in the creation of reference materials (such as dictionaries and grammars) as well as to develop language technologies including searchable digital text @xcite and computer-assisted educational tools @xcite . @xcite . blue boxes show transcriptions that are unsegmented (top) or segmented (bottom). segmented text is split into morphemes which are aligned with the gloss labels shown in the green box. the task of automatic glossing uses some or all of the information in the gray box @xcite to generate the gloss line. interlinear glossed text (igt) is a widespread format in language documentation for linguistic annotation. igt is a multi-line data format (see figure 1 ) which includes (1) a transcription of speech in the language, (2) an aligned morpheme-by-morpheme description, and oftentimes (3) a free translation. igt can be used to illustrate morphosyntactic features of languages that other researchers may not be familiar with, and it is a popular format for examples in linguistics papers and textbooks. it also serves as a resource in the nlp context for the creation of morphological paradigms @xcite , machine translation @xcite , generating precision grammars @xcite , and other tools including pos taggers and dependency parsers @xcite . compiling igt data though igt often follows a common glossing format, gloss conventions vary wildly. furthermore, igt data is rarely compiled into large, standardized corpora, often existing as scattered examples in research papers. to address this, we compile the largest corpus of digitized igt from various existing sources, with over 450k examples in 1.8k languages ( §3). we explore meth-ods to normalize this data ( §4)., standardizing over 80% of the grammatical glosses in the corpus to follow the unimorph schema @xcite . we are releasing our corpora for future nlp, linguistics, and language documentation research. automating igt generation the creation of new igt corpora is often difficult and timeconsuming, requiring documenters to perform linguistic analysis and extensive documentation simultaneously. research has found that computational tools can help accelerate annotation and overcome this bottleneck @xcite by predicting the gloss line of igt given a transcription. the majority of prior work on automatic glossing focuses on training monolingual models that can predict igt for a single language @xcite @xcite , however, these models can struggle when data is limited and require dedicated effort to train and deploy. we aim to overcome the monolingual data bottleneck by creating a multilingual pretrained glossing model that can be adapted to specific languages and gloss formats. we continually pretrain a model on our corpus, and find that the pretrained multilingual model retains high performance across languages. we then finetune the pretrained model on monolingual data, including languages that do not appear in the pretraining corpus. our models achieve new sota performance on five out of seven languages, and demonstrate clear improvements for low-resource language settings over an equivalent finetuned model without our continual pretraining ( §7). ## glosslm corpus while various publicly available sources of digitized igt exist, the lack of unified data formatting and ease of access is a roadblock to using these resources effectively. to solve this problem, we compile and clean the largest igt dataset from a variety of sources and languages. in total, our dataset contains over 450k igt instances (from 250k unique sentences) in 1.8k languages, collected from six different igt corpora. all sources are publicly available under the cc by 4.0 license, allowing free use and redistribution, and we have confirmed with the creators of each source that our usage is within the intended use. we make our artifacts available under the same license. ## automatic igt generation next, we evaluate the applicability of our dataset to the nlp task of automatic gloss prediction. we select the igt data from the sigmorphon shared task on interlinear glossing @xcite to use for evaluation and testing, as this data consistently adheres to a set of glossing conventions and has been evaluated on prior models. ## glosslm model using our igt corpus described in section 3, we train a single multilingual pretrained model for the glossing task that can be easily adapted to documentation projects, for both seen languages and unseen ones. ## conclusion high-quality language documentation involves an incredible amount of effort. we compile, normalize, and release the largest corpus of multilingual igt data, enabling future research in linguistics, nlp, and documentation. furthermore, we demonstrate the applicability of our corpus by pretraining a multilingual neural model for automatic generation of igt. we finetune the model on monolingual corpora, showing benefits on low-resource languages due to multilingual pretraining. in five out of seven languages, we achieve a new sota on automatic igt generation.