One Wug, Two Wug+s Transformer Inflection Models Hallucinate Affixes (2022)

Abstract: Data augmentation strategies are increasingly important in NLP pipelines for low-resourced and endangered languages, and in neural morphological inflection, augmentation by so called data hallucination is a popular technique. This paper presents a detailed analysis of inflection models trained with and without data hallucination for the low-resourced Canadian Indigenous language Gitksan. Our analysis reveals evidence for a concatenative inductive bias in augmented models—in contrast to models trained without hallucination, they strongly prefer affixing inflection patterns over suppletive ones. We find that preference for affixation in general improves inflection performance in “wug test” like settings, where the model is asked to inflect lexemes missing from the training set. However, data hallucination dramatically reduces prediction accuracy for reduplicative forms due to a misanalysis of reduplication as affixation. While the overall impact of data hallucination for unseen lexemes remains positive, our findings call for greater qualitative analysis and more varied evaluation conditions in testing automatic inflection systems. Our results indicate that further innovations in data augmentation for computational morphology are desirable.

## introduction data augmentation strategies, for instance, backtranslation @xcite and mixed sample data augmentation @xcite , are increasingly important components of nlp pipelines @xcite . these strategies often form the cornerstone of modern nlp models for lower-resourced and endangered languages and dialects in particular (e.g., @xcite @xcite , where models can otherwise badly overfit due to the paucity of training data. consider the task of low-resource morphological inflection: high-capacity neural models trained without data augmentation are prone to collapsing at test time, achieving as little as 0% accuracy @xcite . conversely, those very same models trained on artificially augmented data can generalize respectably. unfortunately, there is little research on understanding why these augmentation strategies work. we know little about the changes they cause in the model -are they simply a form of weight regularization? do they alleviate class imbalance? or do they provide a task-specific inductive bias? in this paper, we investigate the data hallucination strategy, a relatively commonplace strategy @xcite for increasing the size of small morphological datasets. we conduct our study in the context of developing a paradigm cell-filling (pcfp; @xcite system for the gitksan language -a critically endangered language with an estimated 300-850 speakers @xcite -that can be used for applications such as developing pedagogical noun and verb conjugation exercises and further computer-assisted language learning applications. given a partial inflectional paradigm with n filled slots and a number of empty slots, the task is to complete the paradigm by predicting all the missing slots from the given ones. following previous work on pcfp @xcite , we leverage morphological reinflection models to complete pcfp. specifically, we employ the one-source model of @xcite : we use each of the n given forms in turn to predict the form in an empty target slot, giving n output forms (see fig. 1 , where @xmath0. we then select one of the output forms as our prediction for the empty slot: we pick the predictions that the model makes with the highest confidence, a decision strategy we denote max. 1 given the relatively small size of our paradigm dataset, further described in section 2, we investigate whether data hallucination is an effective strategy for mitigating overfitting. in accordance with recent results @xcite , we find that data hallucination improves performance in "wug test" @xcite like conditions: where no inflectional variant of a lexeme was witnessed during training. surprisingly, however, we also find that data hallucination significantly worsens performance for lexemes which were partially observed during training; that is at least one of the inflectional variants of the lexeme was present in the training data. these findings motivated a controlled error analysis of our pcfp system to discover why data hallucination generalizes to the unobserved test setting but seemingly slashes performance in the observed test setting. this analysis yields two major insights. first, we find that the model trained without hallucination is "often right for the wrong reason" @xcite : our error analysis reveals that a unaugmented transformer model exhibits undesirable memorization to a significant degree, even when incorporating recently prescribed parameter settings for inflection @xcite . this allows the model to memorize lexeme-specific inflection patterns, rather than msd form root we / wa root-1pl.ii wa'm root-3.ii wet / wat second, we find evidence that data hallucination introduces an inductive bias towards concatenative morphology: where inflection is accomplished by appending affixes to a word stem. we find that the max strategy combined with data hallucination selects a simpler transformation: in fig. 1 , the augmented model prefers the simple transformation of appending diit to hap to predict the target hapdiit over the unpredictable transformation from tk'esxw to hapdiit. conversely, the model trained without hallucination exhibits no strong preference over either transformation. since concatenative morphology is the dominant inflection process in gitksan, this inductive bias serves the hallucination model well in inflecting unfamiliar lexemes during testing. data hallucination, however, can be damaging depending on the morphophonological phenomena at hand. we find, for instance, that it dramatically reduces performance in inflections involving reduplication, a transformation that requires copying of phonological material rather than a simple concatenation @xcite . while the overall effect of data augmentation in inflection has been reported as overwhelmingly positive (e.g., @xcite @xcite , our detailed analysis reveals that it carries both benefits and drawbacks and should therefore be applied with caution. furthermore, our findings call for greater qualitative analysis and more varied evaluation conditions in testing automatic inflection systems. ## data our dataset comprises paradigms that were programatically extracted from an interlinear-glossed dataset of 18, 000 tokens @xcite . de-tails of the gloss to paradigm conversion procedure can be found in appendix b. the interlinear glosses were collected during still-active language documentation efforts with gitksan speakers. the gitksan-speaking community recognizes two dialects: eastern (upriver) and western (downriver), and our dataset comprises forms from both dialects. although the two dialects are largely mutually intelligible, some lexical and phonological differences manifest, with the most prominent being a vowel shift. consider the gitksan translation for the word "name" in instead of attempting to model one-to-many (msd to form) paradigms, we adhere to the simplifying constraint that each paradigm have a single realization per morphosyntactic description. in order to convert a one-to-many paradigm to a one-to-one paradigm, we aim to select a single form for each msd so that, taken together, the inflected forms are maximally similar to each other. in the partial paradigm for for to obtain maximally similar inflected forms, we apply the following algorithm to a one-to-many paradigm. first, we generate all possible one-toone realizations of the paradigm. for instance, for in order to train or evaluate a reinflection system for pcfp, we first need all the paradigms to have at least two entries. this is necessary since a reinflection datapoint is of the form src_form:src_msd;tgt_form:tgt_msd. thus, our first step is to drop all paradigms that only have a single entry, providing us with 459 paradigms. next, we randomly sample paradigms (without replacement) and add them to π wug until π wug contains 10% of the 1303 forms in our dataset. 3 this procedure guarantees that no forms in paradigms belonging to π wug are ever observed during training. for the remaining paradigms π, we split them into two disjoint sets: π train and π hold-out . the forms in π train are added to the training set π train . the forms in π hold-out are added either to the development set π dev or partially observed test set π obs . this way, the model is allowed to observe some of the forms belonging to the (partial) paradigms in π dev and π obs during training. however, it is guaranteed not to have observed the particular forms in π dev and π obs during training. 4 more concretely, for a paradigm of size n, between 2 and n -1 forms (inclusive) are placed into train and the remaining forms are all placed into test (or all placed into dev). we obtain the following number of inflectional variants in each disjoint subset: in the next section, we describe our procedure for employing these four sets of (partial) paradigms for training and evaluating a pcfp system. ## experiments and results having split our paradigm dataset into the desired disjoint subsets π train , π obs , π dev , π wug , we can train transformers in morphological reinflection that can, in turn, be used for the pcfp task. 5 training. we form reinflection training pairs by using the given forms in each paradigm in π train . concretely, for every π ∈ π train , we take the cross product of the entries in π and learn to reinflect each given form in the paradigm to another form in the same paradigm as demonstrated in fig. 2 . 6 counting reinflection datapoints over all paradigms, we obtain 1365 datapoints in the training set for the reinflection system. we train two transformer models. first, we train a "standard" transformer model on the aforementioned 1365 datapoints using the parameter settings described in @xcite and @xcite ; see appendix a. next, we train a second "augmented" transformer model, using the same hyperparameter settings, on the original 1365 datapoints in addition to 10, 000 datapoints hallucinated from the original training dataset. we obtain the hallucination method, number of hallucinated examples (10, 000), and implementation from @xcite . evaluation. we evaluate the models both on paradigms describing lexemes whose inflections were partially observed (π obs ) and lexemes that are entirely unfamiliar (π wug ). since most of our paradigms are very sparse, containing only contain a few forms, we do a leave-one-out style evaluation procedure where, for every target form in either π wug or π obs that belongs to paradigm π, we predict it using every other form that belongs to the same paradigm π. 7 this gives us |π| -1 predictions for a target form, where |π| is the total number filled slots in the paradigm. finally, we use the max strategy to select the form that was predicted with the highest likelihood averaged across output characters. we consider a paradigm π as correctly predicted if all forms for the paradigm that are present in π obs or π wug were correctly predicted. results and discussion. we make a number of observations regarding the results in fig. 3 . first, we observe that there is a significant reduction in performance for the unfamiliar lexemes (π wug ) relative to the familiar lexemes (π obs ) -replicating observations made in the context of the sigmor-phon shared tasks @xcite @xcite . we find that the augmented model reduces the deficit to 10%. that hallucination improves performance on unfamiliar lexemes has been previously observed @xcite . we also find, however, that hallucination worsens performance on familiar lexemes. in both cases, the aggregate accuracy scores glean little insights into these surprising results. why does accuracy drop by nearly 50% for the non-hallucination model across the two testing conditions? how does hallucination improve performance on unfamiliar lexemes? and why does hallucination reduce performance on familiar lexeme paradigms? to understand these differences in performance between the two models and testing conditions, we turn to an analysis of the errors. ## error analysis to reveal insights into the behaviour of the two transformer models, we look into the case of gitksan pluralization, which is instantiated as suppletion or reduplication depending on the lexeme, enabling us to investigate whether either transformer can learn two disparate inflectional strategies. this error analysis enables us to systematically characterize the effects hallucination has on the transformer model in inflection, demonstrating that the effects can be both beneficial and adverse. unaugmented transformers memorize inflection patterns. we begin by analyzing the models' behaviour on suppletive forms; gitksan uses suppletion as a productive strategy for pluralization. for instance, the stem for singular forms for "laugh" is tk'esxw, but the plural stem is hap. the transformation from a singular form to a suppletive plural form is unpredictable (ts'@xmath1; the model must instead rely on other plural source forms (e.g., @xmath2. even if the model is unable to produce the correct suppletive plural inflection, it should be able to perform the simpler task of placing higher confidence in the prediction from the plural source form (hap) over the singular source form (ts'ehlx). failing to exhibit this preference would indicate that the model is simply niye'et ## general discussion in this paper, we explore the effect of data hallucination on the gitksan language that is currently underserved in nlp. given the low amount of training data for the model, inflection models are likely to encounter many unfamiliar lexemes during test 9 it could also explain why we don't see a greater increase in performance on the πwug test set with the augmented model. under these disparate conditions, we find that a data-augmented model and a standard model exhibit drastically different behaviours. we found that the standard model, a transformer model trained under recommended parameter settings @xcite , memorizes inflection patterns to a significant degree (section 3 and section 4). at the same time, we find that data hallucination alleviates the need for memorization significantly, generalizing well to unfamiliar lexemes (section 3) with an inductive bias towards concatenative morphology (section 4). data hallucination, however, is not universally beneficial: we find it reduces the model's capacity to recognize common morphophonological phenomena (section 4), limiting the performance improvements it can bring. although our study was conducted on a single language, we note that our characterization of data hallucination could be informative for languages other than gitksan. as section 4 demonstrates, data hallucination can encourage the model to apply voicing in incorrect contexts. such effects are not limited to gitksan. in english, data hallucination could give rise to erroneously conditioned allomorphy: for instance, hallucination can generate a synthetic past tense inflection example mar -> mard from a gold standard training example such as like -> liked. the desired hallucinated past tense form is of course mared. overall, our work suggests common data augmentation strategies for nlp like data hallucination merit closer inspection and that further innovations in data augmentation for computational morphology are desirable. hongyi zhang, moustapha cisse, yann n. dauphin, and david lopez-paz. 2018. mixup: beyond empirical risk minimization. in international conference on learning representations. xingyuan zhao, satoru ozaki, antonios anastasopoulos, graham neubig, and lori levin. 2020. automatic interlinear glossing for under-resourced languages leveraging translations. in proceedings of the 28th international conference on computational linguistics, pages 5397-5408, barcelona, spain (online). international committee on computational linguistics.