Robust Generalization Strategies for Morpheme Glossing in an Endangered Language Documentation Context (2023)

Abstract: Generalization is of particular importance in resource-constrained settings, where the available training data may represent only a small fraction of the distribution of possible texts. We investigate the ability of morpheme labeling models to generalize by evaluating their performance on unseen genres of text, and we experiment with strategies for closing the gap between performance on in-distribution and out-of-distribution data. Specifically, we use weight decay optimization, output denoising, and iterative pseudo-labeling, and achieve a 2% improvement on a test set containing texts from unseen genres. All experiments are performed using texts written in the Mayan language Uspanteko.

## introduction with over half of the world's languages endangered @xcite , language documentation is one of several strategies for preservation. traditionally, many documentation projects have aimed to create grammatical descriptions, dictionaries, and annotated text corpora, in the form of interlinear glossed text (igt; see section 2.1). the annotated texts can be used in the creation of reference tools and pedagogical materials, as well as providing input data for downstream tasks such as machine translation @xcite , morphological paradigm induction @xcite , dependency parsing @xcite , and other tasks @xcite , making it particularly valuable for low-resource languages. annotation of large corpora can be timeconsuming and monotonous, so there is a desire for systems to automatically produce igt, annotating plain text with labels describing the part-of-speech, morphology, and syntax of each word in the corpus @xcite . these systems can be used in conjunction with human annotators to create annotated corpora rapidly, ensuring consistency and reducing the amount of human effort required. importantly, reducing annotation time also frees up language experts to work on other types of language preservation or revitalization activities. however, generalization for automated annotation systems remains a critical problem. preexisting corpora of annotated text are often small, contain transcriptions of spoken language from a small number of distinct speakers, and focus on specific types of language such as story-telling and oration. thus, systems trained on these corpora have difficulty generalizing to out-of-distribution (ood) language, limiting their utility and robustness. as acquiring additional annotated data is generally expensive and difficult, it is preferable to design models that generalize well to ood data. in this work, we design models for one type of text annotation: labeling each morpheme in a text with its grammatical function. we envision these models being used alongside human annotators to provide suggestions and annotate text more quickly and consistently than by human labeling alone. we examine three strategies to improve the robustness of these morpheme labeling models with limited data: our experiments evaluate model performance on texts of different genres than the texts in the training set, in order to investigate their ability to generalize to future, out-of-distribution texts. we find that these strategies achieve small performance improvements on in-and out-of-distribution texts, and may be valuable for building more robust morpheme labeling models. our code is available on github. 1 2 background ## experiments for a baseline model, we fine-tune uspmlm on the token classification task. fine-tuning uses the same hyperparameters listed in all strategies perform worse on the out-ofdistribution data. the goal of the following experiments is to improve generalization of the model and thereby close the gap in performance for the id and ood evaluation sets. though the neural model and the naïve model using the most frequent gloss perform similarly, we will conduct experiments with the neural model, which can be more readily manipulated to improve generalization. ## results in each step, we use the best trained model from the previous step. we do not iterate pseudolabeling on the test set, since the test set should have the same distribution as the ood eval set. through weight decay optimization, denoising, and iterative pseudo-labeling, we are able to accomplish an improvement of 2 percentage points in performance on ood data, with an 8.2% reduction in overall error. pseudo-labeled 77.5 these techniques also improve performance on the in-distribution eval set, although by a smaller margin than the out-of-distribution eval set. this is desirable, as it narrows the gap between performance on in-and out-of-distribution data, resulting in more predictable model performance. ## discussion although the techniques used in this work do yield performance improvements, generalization in language documentation remains a difficult task, largely due to hard-to-overcome challenges such as unseen morphemes, labels for morphemes that do not appear in the training set, and ambiguity in labeling. weight decay optimization, like all forms of hyperparameter tuning, is highly situation-dependent and requires good evaluation. generally, avoiding overfitting and minimizing variance is critical to generalization in documentation, where the training sets may represent only a small fraction of the distribution of possible texts. denoising is a promising strategy for making high-quality predictions on completely unknown morphemes, using the surrounding context. this approach may be particularly useful in a humanin-the-loop situation, where the denoiser provides several top guesses for an unknown morpheme, and a human annotator can select between the options, allowing for easier annotation and possibly active learning @xcite . denoising will likely show more robust performance for languages with highly structured and productive morphological systems and relationships such as agreement and regular word order. some aspects of uspanteko morphology are productive and structured. for example, verbs can take multiple affixes, both prefixes and suffixes, and these occur in a predictable order, according to a morphological pattern. at the same time, the language also has relatively flexible classes of morphemes, allowing non-verbal stems to act as predicates (coon, 2016), taking some of the same morphology as seen on verb stems. this flexibility may have decreased the utility of the denoising approach, as unseen stems appearing in verbal positions could be verbal or non-verbal morphemes, with no clear distinction. iterative pseudo-labeling similarly shows only a small improvement. in these experiments, the ood texts still share fairly similar contexts and labeling strategies with the training set, as evidenced by the perplexity values. however, in a case where the unseen texts are more dissimilar to the training set, this strategy could be more effective at tuning the model to the particular target distribution. ## future research this work presents a preliminary exploration into generalization for documentation models, and much work remains to be done. documentation data for even the most widely-spoken languages is limited, yet robust generalization from the training set is crucial for improving usability. one promising approach for creating more robust documentation models is through cross-lingual transfer that utilizes the morphological similarities between languages. @xcite demonstrates that this approach can effect performance improvements on in-distribution data, and it would likely benefit out-of-distribution data as well. another technique for avoiding overfitting and improving generalization is ensuring models focus on linguistic information, relying less on semantic patterns that may lead to spurious generalizations. this could involve morphologically inspired loss functions, data augmentation using rule-based systems, or pretraining on other linguistic tasks. ## conclusion in this work, we presented three strategies for improving generalization of interlinear glossed text generation models, which to our knowledge are novel approaches to the problem. we use weight decay optimization, denoising, and iterative pseudolabeling, finding that iterative pseudo-labeling provides the greatest improvement in performance. overall, our best model achieves a 2% improvement from the baseline on a test set representing texts of unseen genres. we also investigate the discrepancy in performance between in-and out-of-distribution data, finding that out-of-vocabulary morphemes and differences in context are key sources of error. we hope these approaches can inspire future work in improving generalization for documentation models, which is difficult but critical to the usability of automated documentation systems in real-world projects.