Generalizing Morphological Inflection Systems to Unseen Lemmas (2022)

Abstract: This paper presents experiments on morphological inflection using data from the SIGMORPHON-UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection. We present a transformer inflection system, which enriches the standard transformer architecture with reverse positional encoding and type embeddings. We further apply data hallucination and lemma copying to augment training data. We train models using a two-stage procedure: (1) We first train on the augmented training data using standard backpropagation and teacher forcing. (2) We then continue training with a variant of the scheduled sampling algorithm dubbed student forcing. Our system delivers competitive performance under the small and large data conditions on the shared task datasets.

## introduction this paper presents experiments on morphological inflection using data from the sigmorphon-unimorph shared task 0: generalization and typologically diverse morphological inflection @xcite ). 1 our system focuses on typologically diverse inflection generation, that is, the task of inflecting a lemma in a given form, which is specified by a morphosyntactic description (msd). as an example, consider inflecting the english verb lemma walk in the past tense according to the msd verb+past, thereby generating the inflected form walked. the shared task investigates two data conditions: under the small data condition, up to 700 training examples are provided. under the large data condition, up to 7000 training examples are provided. our system beats the official neural shared task baseline by more than 8%-points under both the small and large data conditions. we apply transformer models @xcite to the inflection task. the model is trained to translate an input sequence consisting of lemma characters and an msd, like: w, a, l, k, +verb, +past into the inflected output sequence: w, a, l, k, e, d general purpose transformers were originally developed for machine translation, but they also deliver strong performance on morphology tasks @xcite . nevertheless, we observe that the vanilla transformer architecture is not ideally suited for inflection: in contrast to machine translation, many inflectional phenomena are strongly positionally dependent, which is something that the vanilla transformer architecture does not adequately model. for example, phonological alternations often happen at affix boundaries and these typically occur either at the start or end of word forms. whereas the positional encoding in the transformer architecture allows for uniquely conditioning on relative positions with regard to the start of the string, the same is not true for positions at the end of the input string. we, therefore, augment our transformers with reverse positional encoding, presented in section 3.1, which allow the model to condition directly on the end of the input string. in previous iterations of the sigmorphon inflection shared task @xcite @xcite @xcite , so called lemma overlap, where identical lemmas occur both in the training and test set, has caused inflated performance, resulting in near perfect inflection accuracy for many languages. @xcite and @xcite show that more challenging data splits with low lemma overlap can cause significant reduction in inflection performance. the data in this year's inflection task demonstrate varying lemma overlap, ranging @xmath0. accordingly, we decided to investigate different mechanisms which we hypothesized would improve generalization to unseen lemmas in the test set. data augmentation is a commonly used technique, which improves generalization in many nlp tasks. here the gold standard training data is augmented with synthetic examples. back-translation introduced for machine translation is perhaps the best known method @xcite , but has not been very successful in morphology tasks @xcite . we instead use the data hallucination approach by @xcite , which synthesizes new training examples from existing gold standard training examples by identifying a (possibly discontinuous) word stem and replacing this with a random character sequence. in addition to data hallucination, we experiment with another data augmentation technique: lemma copying @xcite , where the model is trained to copy input lemmas from the test set in order to adapt the model more closely to the test data. in our experiments, this method ultimately delivers better performance than data hallucination. as a further attempt to improve generalization, we experiment with modifications of the standard teacher forced training procedure of inflection models. when applying teacher forcing during training, the model is allowed to rely on gold standard history for time steps 1 up to t, when predicting output at time step t + 1. this speeds up convergence considerably but can also result in sub-optimal performance due to so-called exposure bias @xcite , which is caused by a mismatch when conditioning on gold standard history during training and predicted history during test time. we take an alternative approach called student forcing @xcite , which is an application of scheduled sampling @xcite for morphology tasks. here model-predicted output history is substituted for the gold standard history for a subset of training examples in order to counteract exposure bias while simultaneously maintaining efficient training (see figure 1 ). according to @xcite , student forcing can improve inflection performance under low-resource conditions. our experiments show that student forcing can deliver small improvements for some languages but does not outperform data hallucination. however, the techniques seem to be complementary; their combination provides improvements over plain data augmentation. in summary, our main contributions are as follow: another modification to the transformer architecture, which can improve performance on morphology tasks, is to add a so-called monotonicity loss @xcite . this can bias the transformer toward near-monotonic alignment between the input and output sequence, which is often the case in inflection. we use data augmentation to improve generalization to unseen lemmas. this has become a standard technique in low-resource inflection in recent years. a common approach is to generate synthetic examples by first identifying word stems in gold standard examples and then replacing the stems with random character sequences @xcite . @xcite introduce a more refined method to hallucinate synthetic stems, which aims to honor the phonology of the target language by generating sequences of random syllables rather than random characters. @xcite show that a simpler data augmentation method, where random strings or unlabeled word forms are copied from the input to the output, can also be effective. @xcite apply this approach to copying lemmas in the development and test set and show that this can lead to substantial gains in inflection accuracy. we apply their technique in section 3.4. other approaches to data augmentation in morphological inflection include: reframing the task as reinflection and generating reinflection examples from the existing inflection training data @xcite , as well as generating new training examples using back-translation @xcite , and self-training @xcite . in addition to data augmentation, we also experiment with student forcing to improve generalization. as mentioned above, this is an application of scheduled sampling. @xcite explore scheduled sampling for various sequence generation tasks (image captioning, constituency parsing and speech recognition). this is a curriculum learning approach @xcite , where the model is gradually exposed to more of its own prediction errors during training, thereby counteracting exposure bias. the student forcing approach presented by @xcite is a slight simplification of this approach. essentially, student forcing uses a fixed amount of modelpredicted contexts throughout training instead of a curriculum approach. ## methods in this section, we describe our contributions to the inflection task, before moving on to our experiments in subsequent sections. ## experiments and results here, we describe our experiments on small and large training sets. under both data conditions, we train models using the following procedure: we first augment the training data using data hallucination or lemma copying. we then train the model on the augmented data for a maximum of 20,000 steps without teacher forcing. we then identify the best checkpoint model based on development set accuracy and continue training this model with student forcing for an additional 1000 steps. when applying lemma copying, we have to train separate models for the development and test set: one model which augments the training set with lemmas from the development set and another one which augments with test lemmas. we first tune hyperparameters on the development set and then use this hyperparameter configuration when training the final model for the test set. crucially, this allows us to avoid augmenting the training data both with development and test lemmas in order to not use extra data for tuning model parameters. ## discussion the most prominent trend in our experiments is that lemma copying delivers sizable improvements in accuracy, particularly under the small data condition. it is also noteworthy that models trained on small training data using data augmentation(either hallucinated data or copied data) outperform models trained on large training data without data augmentation. based on these results, it is clear that data augmentation is a crucial technique in lowresource inflection, delivering substantial improvements which parallel improvements from a significant additional annotation effort. this might allow researchers to kick-start development of morphology resources for low-resource languages using very little annotated data. performance also seems to improve even under higher data conditions when lemma overlap in the training data and test data is small. student forcing delivers small improvements at best and is often harmful when combined with data augmentation. we do not have a good explanation for this phenomenon at the current time. based on our experimental results, we can conclude that data augmentation is a far more influential method for countering data sparsity. it is interesting to see that our base inflector, trained without student forcing or data augmentation, outperforms the shared task baseline. given that the baseline system is a character-level transformer @xcite , this might be attributable to our architectural innovation, namely reverse positional encoding. however, another difference between our system and the shared task baseline is that the baseline is a multilingual system, whereas our system is monolingual. further investigation is required to tease apart these effects. ## conclusion in this work, we advance the generation performance of inflectional forms with a joint effort including reverse positional encoding, data hallucination, copying lemmas, and student forcing. we improve the prediction accuracy by 9.6% and 8.6% above the official neural shared task baseline on the small and large test set respectively. according to our results, the joint effect of reverse positional encoding, lemma copying, and student forcing results in the best performance. we investigate two data augmentation strategies: the effect of data augmentation is more evident when less annotated data is available for training.