Findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing (2023)

Abstract: This paper presents the findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing. This first iteration of the shared task explores glossing of a set of six typologically diverse languages: Arapaho, Gitksan, Lezgi, Natügu, Tsez and Uspanteko. The shared task encompasses two tracks: a resource-scarce closed track and an open track, where participants are allowed to utilize external data resources. Five teams participated in the shared task. The winning team Tü-CL achieved a 23.99%-point improvement over a baseline RoBERTa system in the closed track and a 17.42%-point improvement in the open track.

## introduction roughly half of the world's languages are currently endangered @xcite . as a result, language preservation and revitalization have become significant areas of focus in linguistic research. both of these endeavors require thorough documentation of the language, which is crucial for creating grammatical descriptions, dictionaries, and educational materials that aid in language revitalization. however, traditional manual language documentation is a time-consuming and resource-intensive process due to the costs associated with collecting, transcribing, and annotating linguistic data. therefore, there is a need to expedite the documentation process through the use of automated methods. while these methods can never fully replace the expertise of a dedicated documentary linguist, they have the potential to greatly facilitate and accelerate the annotation of linguistic data @xcite . linguistic annotation involves several interconnected subtasks, including: (1) transcription of speech recordings, (2) morphological segmentation of transcribed speech, (3) glossing of segmented morphemes, and (4) translation of the transcriptions into a matrix language, such as english. these processes result in a semi-structured output known as an interlinear gloss, as demonstrated in the natügu example below: this paper presents the findings of the sigmor-phon 2023 shared task on interlinear glossing foot_0 , which focuses on automating step (3) of the language documentation pipeline. notably, this shared task represents the first initiative specifically dedicated to interlinear glossing. despite the prevalence of interlinear glossed text as a data format in language documentation, the automatic generation of glossed text remains relatively underexplored in the field of natural language processing (nlp). we hope that this shared task can help stimulate further work in automated glossing. ## background existing work in data driven automated glossing has utilized both traditional feature-based approaches like maximum entropy classifiers (memm) @xcite and conditional random fields (crf) @xcite as well as more recent neural models like lstm encoderdecoders @xcite and transformers @xcite . @xcite investigate active learning for interlinear glossing using the memm architecture. mcmillan-major (2020) incorporated translations as auxiliary supervision in a crf glossing model. @xcite and barriga @xcite compare traditional feature-based models and lstm encoder-decoder models. @xcite present a modified multi-source transformer model which incorporates translations as auxiliary supervision. the current literature on automatic glossing exhibits notable gaps, as several techniques that have proven valuable for other morphology tasks have yet to be explored for glossing. there are several intriguing directions for future research, including: the submissions in this shared task explore several of these techniques, including the use of pretrained language models, data augmentation, utilization of external data, and the application of hard attention models. ## future directions the submissions in this shared task have explored several novel techniques that have not been previously applied to automatic interlinear glossing. surprisingly, pretrained language models like byt5 did not perform as well as one might expect based on their strong performance on other morphology tasks. this unexpected outcome raises the need for further investigation. one interesting observation is that the winning submission, tü-cl, completely disregards the provided translations. while this could suggest that translations may not be as useful for the glossing task, we believe there is still room for improvement in this area. incorporating large pretrained english models as a reliable source of translated text could potentially lead to additional enhancements. considering the availability of extensive morphological resources for many languages, such as those provided by unimorph and similar projects, multi-task learning holds promise for interlinear glossing. additionally, we encourage further exploration of crosslingual approaches, leveraging the odin database of interlinear glossed text, which despite being noisy, offers a highly multilingual resource for research purposes. ## conclusion the 2023 sigmorphon shared task on interlinear glossing received submissions from five teams which presented a wealth of interesting techniques greatly expanding the field of automated interlinear glossing. the submissions achieved substantial im-provements over a baseline roberta system. the winning team tü-cl achieved a 23.99%-point improvement over the baseline in the closed track and a 17.42%-point improvement in the open track using a hard attention model.