S ig M ore F un Submission to the SIGMORPHON Shared Task on Interlinear Glossing (2023)

Abstract: In our submission to the SIGMORPHON 2023 Shared Task on interlinear glossing (IGT), we explore approaches to data augmentation and modeling across seven low-resource languages. For data augmentation, we explore two approaches: creating artificial data from the provided training data and utilizing existing IGT resources in other languages. On the modeling side, we test an enhanced version of the provided token classification baseline as well as a pretrained multilingual seq2seq model. Additionally, we apply post-correction using a dictionary for Gitksan, the language with the smallest amount of data. We find that our token classification models are the best performing, with the highest word-level accuracy for Arapaho and highest morpheme-level accuracy for Gitksan out of all submissions. We also show that data augmentation is an effective strategy, though applying artificial data pretraining has very different effects across both models tested.

## introduction this paper describes the sigmorefun submission to the sigmorphon 2023 shared task on interlinear glossing. given input text in a target language, the task is to predict the corresponding interlinear gloss (using leipzig glossing conventions). igt is an important form of linguistic annotation for the morphological analysis of languages, and also serves as an extremely valuable resource for language documentation and education for speakers of low-resource languages. there were two tracks for this shared task, track 1 (closed) and track 2 (open). for track 1, systems could only be trained on input sentences and glosses; in track 2, systems could make use of the morphological segmentation of the input as well as any (non-igt) external resources. since the track 2 setting better matches the long-term re- * these authors contributed equally search goals of our team, we only participate in this open track. in our submission, we investigate two different approaches. first, we attempt data augmentation by either creating our own artificial gloss data by manipulating the existing training data, or by utilizing existing resources containing igt in other languages ( §2). second, we explore two different models for gloss generation ( §3). the first builds off the token classification baseline, while the second uses a pretrained multilingual seq2seq model. finally, we also attempt to post-correct model outputs with a dictionary. we apply this to gitksan and find that this, combined with our other approaches, results in the highest morpheme-level accuracy for gitksan in track 2. ## data augmentation one major challenge for this shared task is the scale of data provided. all of the languages have less than 40k lines of training data, and all but arapaho have less than 10k. the smallest dataset (gitksan) has only 31 lines of data. thus, one obvious method to try is data augmentation. more specifically, we try pretraining our models on different forms of augmented data before training them on the original target language data. we explored two forms of data augmentation. first, we generated artificial gloss data in the target language by swapping words in the existing training data. second, we utilized data from odin @xcite to see if transfer learning from data in other languages can help improve performance. ## models we explore two models for gloss generation. the first one is built upon the token classification baseline with some improvements, and we treat this model as our internal baseline. the second model we deploy tests whether we can achieve competitive performance by finetuning a pretrained character based multilingual and multitask model, byt5. for this model, we perform minimal preprocessing and use raw segmented morphemes and free translations if available. ## dictionary post-correction: gitksan one of the key challenges for extremely low resource languages is the integration of structured linguistic data in other forms, such as a dictionary, into machine learning pipelines. we test a simple post-correction method from a pre-existing dictionary on gitksan only, due to its unique combination of low resource and easily obtainable dictionary in machine readable form. we use the dictionary compiled by @xcite , without consulting the morphological analyzers that they also provided. at inference time, if a morpheme is unseen during training, we search for the exact form in the dictionary. we also expand the search to all sub sequences of morphemes within the enclosing word, plus the previous whole word in cases where a particle is included in the dictionary form. the first matched definition is used as the gloss and if none of the search yields an exact match, we fall back to the model prediction. we only apply this method to the token classification models because the alignment between morphemes and glosses is directly established, whereas the seq2seq models do not guarantee that the number of glosses matches the number of morphemes. ## results and discussion tables 1 and 2 show our systems' performance (as well as the original baseline) on the test data with respect to word-and morpheme-level micro-averaged accuracy, respectively. overall, the token classification model trained first on the artificially generated augmented data perform the best, with the model trained on the shared task data only not far behind. meanwhile, byt5 models perform worse, with the model finetuned first on odin trailing our best model by a few percentage points, while the model finetuned first on augmented data performs worse than the baseline. ## related work there have been a variety of approaches to the problem of (semi-) automatically generating interlinear gloss. @xcite investigate the efficacy of active learning for the task of interlinear glossing, using annotation time required by expert and non-expert annotators as their metric. the system they use to generate gloss label suggestions is a standard maximum entropy classifier. a rule-based approach by @xcite utilizes an fst to generate glosses for plains cree, focusing on nouns. @xcite view the task of glossing segmented text as a two-step process, first treating it as a standard pos tagging task and then adding lexical glosses from a dictionary. they demonstrate this method on a chintang corpus of about 1.2 million words. a number of other works focusing on interlinear glossing utilize conditional random field (crf) models. @xcite test three different models on a very small lezgi dataset (< 3000 words): a crf (that outputs bio labels with the corresponding gloss per character in the input), a segmentation and labelling pipeline that utilizes a crf (for bio labels) and svm (for gloss labels), and an lstm seq2seq model. they find that the crf that jointly produces the bio labels and tags produced the best results. mcmillan-major (2020) utilizes translations in their training data by creating two crf models, one that predicts gloss from the segmented input and another than pre- ## conclusion in our shared task submission, we explore data augmentation methods and modeling strategies for the task of interlinear glossing in seven low-resource languages. our best performing models are token classification models using xlmr. we demonstrate that pretraining on artificial data with xlmr is an effective technique for the mid-resource test languages. additionally, in our error analysis we find that we may have actually undertrained our token classification models, and thus our systems may have the potential to perform better with additional hyperparameter tuning. while our byt5 models did not perform as well as our other systems, we show that pretraining on odin data is effective, despite this data being very noisy. finally, we also demonstrate improvements by utilizing a dictionary to post-correct model outputs for gitksan.