Getting The Most Out of Your Training Data: Exploring Unsupervised Tasks for Morphological Inflection (2024)

Abstract: Pre-trained transformers such as BERT have been shown to be effective in many natural language tasks. However, they are under-explored for character-level sequence to sequence tasks. In this work, we investigate pre-training transformers for the character-level task of morphological inflection in several languages. We compare various training setups and secondary tasks where unsupervised data taken directly from the target task is used. We show that training on secondary unsupervised tasks increases inflection performance even without any external data, suggesting that models learn from additional unsupervised tasks themselves—not just from additional data. We also find that this does not hold true for specific combinations of secondary task and training setup, which has interesting implications for denoising objectives in character-level tasks.

## introduction transformers have been shown to be an effective architecture for various natural language processing tasks @xcite , facilitating the ubiquitous method of pretraining on some unsupervised task with an abundance of data and then finetuning to a specific supervised task. transformers have also been shown to be an effective architecture for character-level tasks such as graphemeto-phoneme conversion (g2p) and morphological inflection @xcite . however, very little work has explored the application of pretrained models to character-level tasks, which likely require different inductive biases than the more semantically-oriented tasks where pretraining is typical. for instance, @xcite , a multilingual pretrained transformer using byte inputs, showed impressive performance on several semantically-oriented benchmarks, as well as on some character-level tasks including morphological inflection. however, it still under-performs the best two shared task submissions for the inflection benchmark @xcite . the computational morphology community is frequently interested in low-resource languageslanguages that do not have sufficient data available to apply standard nlp techniques. this is harder for morphologically complex languages, where the large set of inflectional patterns lead to an explosion in possible words, which become difficult to model with a small dataset. for these reasons, there is interest in building tools to aid in expanding morphological resources for language education tools, research, and documentation. using nlp methods to build systems for analyzing and applying morphology in generalizable way to unseen words is thus a useful goal. several shared tasks have been held to this end @xcite @xcite @xcite , where a machine learning model that performs well can be seen as competently representing the underlying system of morphology for a given language. in this work, we explore utilizing secondary unsupervised tasks -tasks similar to language modeling which can serve as auxiliary tasks in a multitasking setup or pretraining tasks in a pretraining setup -when training encoder-decoder transformers for the task of morphological inflection. we investigate the benefits of pretraining (pt) beyond expanding the vocabulary distribution during training and also compare it to multi-task learning (mtl). following @xcite , we use autoencoding (ae) as an unsupervised secondary task and additionally compare it to the denoising task of character-level masked language modeling (cmlm) @xcite . we explore these methods in data-scarce settings to investigate their potential impact in the low-resource setting. our data samples and code are available publicly. 1 we specifically investigate the following research questions: our results show that both unsupervised pt and mtl are effective for morphological inflection, even with samples prepared exclusively from the supervised data itself. we find that simply autoencoding the training words is more effective than cmlm in these data-scarce settings. though the best method on average seems to be mtl with ae in our experiments, this is not consistent across every language. we also find that, in the mtl setup, cmlm actually performs worse than the baseline-though this is quickly reversed if we use out-of-distribution data for the secondary task. ## architecture and training in this section we discuss our training methodology including architecture, training setups, and tasks. ## when does denoising hurt mtl? there is a remarkable gap in performance between mtl-cmlm and pt-cmlm (6.29 absolute accuracy) as well as mtl-ae (10.9 absolute accuracy). while denosing is a useful objective to pretrain on, it actually hurts performance in an mtl setup in our experiments. this also begs the question: why is ae a valid secondary task when multitasking (our best overall setup), but not denoising? we hypothesize that denoising negatively impacts model learning because it is a sufficiently different task optimized on the same words as inflection. in the pt setup, if denoising learns a representation that conflicts with the finetuning task, this can be resolved by optimizing strictly on the finetuning task in a second phase. however, perhaps when optimizing jointly, the denoising objective skews the model distribution for the training words. this would imply that if denoising is done on external data, it should not have such a negative impact. based on these initial highly negative results for mtl-cmlm, we perform an additional analysis to investigate the impact of data diversity on both secondary tasks in an mtl setup. here all data for unsupervised learning is sampled from a source external to the finetuning data. we use universal dependencies @xcite as the source of external data, which we discuss in more detail in subsection a.3. ## conclusion in this work, we explored multiple methods for transfer learning for morphological inflection, many of which showed remarkable performance for a large set of languages. we investigated two different training methods: pretraining-finetuning and multi-task learning, and two different secondary tasks: denoising and autoencoding. in a low-resource setting, we found that secondary unsupervised tasks are effective even without the addition of any new vocabulary items beyond the finetuning dataset. while pretraining is an effective setup for improving morphological inflection without any external data, multi-task learning with an autoencoding objective is the best setup in all experiments. on the other hand, multi-task learning with the cmlm denoising objective is the worst performing setup, performing below the baseline on average. in further analysis, we found that performing cmlm on external data that is separate from the finetuning data solves this issue, resulting in significantly better performance. the success of denoising objectives such as mlm cannot be denied for large-scale training and semantically oriented tasks. our experiments and results show that similar tasks are effective in data-scarce settings for character-level tasks like morphological inflection. in practice, it seems that low-resource character-level tasks should always consider training in a multi-task setup with an autoencoding secondary task even if the supervised training data is the only available data -and exploring denoising objectives if unsupervised data from an external source is available.