Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models (2022)

Abstract: Deep learning sequence models have been successful with morphological inflection generation. The SIGMORPHON shared task results in the past several years indicate that such models can perform well, but only if the training data covers a good amount of different lemmata, or if the lemmata to be inflected at test time have also been seen in training, as has indeed been largely the case in these tasks. Surprisingly, we find that standard models such as the Transformer almost completely fail at generalizing inflection patterns when trained on a limited number of lemmata and asked to inflect previously unseen lemmata—i.e. under “wug test”-like circumstances. This is true even though the actual number of training examples is very large. While established data augmentation techniques can be employed to alleviate this shortcoming by introducing a copying bias through hallucinating synthetic new word forms using the alphabet in the language at hand, our experiment results show that, to be more effective, the hallucination process needs to pay attention to substrings of syllable-like length rather than individual characters.

## introduction the transformer model has delivered convincing results in many different tasks related to wordformation and analysis @xcite @xcite . especially on inflection tasks, where an input lemma such as dog, and input inflectional features such as {n,pl}, are expected to produce an output such as dogs, the model has shown to be particularly adept at generalizing patterns @xcite @xcite . however, we have discovered that this is only true if the training data covers a diversity of lemmata or some variant of the input lemma to be inflected has been figure 1 : transformer performance in the commonpractice setting (left), "wug test"-like setting (middle), and ''wug test"-like setting with our best data hallucination method (right) witnessed during training. in a "wug test" @xcite setting where the witnessed lemmata are usually limited and a previously unseen lemma-like wug-is to be inflected in some way, we find that the transformer almost completely fails to generalize inflection patterns, despite abundant inflected forms for training. it has been noted earlier that neural sequence-to-sequence models are apt to perform poorly for morphological inflection if they have been exposed to little training data and data augmentation can be leveraged to alleviate the problem @xcite @xcite . our starting point is our observation that the poor "wug test" performance is maintained even with abundant training inflected forms. in our study, we show three main results. (1) we demonstrate that, even if trained with relatively large amounts of inflected forms, a transformer model of the kind that has been very successful at recent shared tasks largely fails to generalize inflection patterns if it has not been exposed during training to a variety of lemmata or any lemmata in the test set. this is true even for datasets where all words inflect in the same way-i.e. there are no inflectional classes or allomorphs of morphemes, as is found in the low-resource niger-congo language datasets used in sigmorphon 2020 shared task @xcite . (2) we show that simply exposing the model to uninflected lemmata in the test set-without providing a single inflected formallows the model to dramatically improve its performance when inflecting such lemmata. (3) further, we investigate several strategies that avoid leveraging test set lemmata. we show that when inducing a copy bias in the model by hallucinating new lemmata, or by hallucinating new inflected forms, the method of hallucination is much more effective if it is sensitive to substrings of syllable-like length rather than individual characters or stems. our best models achieve substantial improvement upon earlier state-of-the-art data hallucination methods @xcite . to prepare training data for the "wug test"-like circumstance, we select the unimorph @xcite paradigms for the first 100 most frequent lexemes found in wikipedia text, foot_1 which are not included in the 2018 shared task 1 dev and test sets. the shared task dev and test sets are used for validation and evaluation without any change. the 100 full inflection tables give us over 1,000 (for czech, german and russian) or over 7,000 (for finnish, spanish and turkish) training triples. niger-congo languages in addition, we use six niger-congo languages from sigmorphon 2020 shared task 0 @xcite : akan, ga, lingala, nyanja, southern sotho and swahili. these languages are low-resource, but the dataset only contains very regular inflections. in the original shared task data split, the overlap between the lemmata in the dev and test sets and those in the training set is 100%. the number of paradigms which we can obtain by combining the training, dev and test sets of this dataset is around 100 for akan, ga and swahili, 227 for nyanja, 57 for lingala and only 26 for southern sotho. for the "wug test", we divide the inflection tables reconstructed from this dataset into a 7:1:2 traindev-test split, i.e. we use the same ratio as the shared task, but the division is by inflection tables rather than lemma-tag-form triples, to ensure that the lemmata used for validation and test are disjoint from those for training. we provide details on the data statistics in appendix a for reference. ## experiments inflection model the transformer @xcite is the seq2seq architecture which produces the current state-of-the-art result on the morphological inflection task @xcite @xcite . it takes the lemma and target tag(s) as input and predicts the target form character by character. our experiments use the transformer implemented in fairseq @xcite and adopt the same hyperparameters as @xcite . 3 evaluation metric the evaluation metric is accuracy. for the original shared task data and experiments on 2018 languages, we train five inflection models each with a different random initialization and report the average accuracy with standard deviation. due to data scarcity, for niger-congo languages at the "wug test"-like setting, we perform a 5-fold cross-validation and report the average accuracy and the standard deviation. common-practice test and "wug test" we first compare the performance of the transformer in the common-practice setting and the "wug test"-like setting. the "common practice" is represented by previous years' shared tasks and related work @xcite @xcite @xcite ; here the training data usually covers a fair number of lemmata and there is overlap between lemmata in the training and test sets. we use the shared task data to represent the common-practice setting. in the "wug test" setting, we control the number of lemmata for training but not inflected forms (as explained in section 2) and the lemmata to be inflected are always previously unseen. to our surprise, the performance of the transformer at the "wug test"-like setting is very poor despite the large amount of training triples for 2018-languages or the very regular and straightforward inflection for niger-congo languages. the performance is dramatically inferior to the common-practice setting, even when the number of training triples is seven times larger for finnish, spanish and turkish (see figure 1 ). we hypothesize four reasons for the poor performance of the model under the "wug test"-like circumstance: (1) missing copy bias regarding the entire stem, i.e. the model can't copy a stem abcde if that exact stem has never been seen during training, (2) missing copy bias on individual letters, i.e. the model can't copy letter a if the letter is underrepresented in training, (3) missing copy bias on subsequences of letters, i.e. the model can't copy sequence ab if the sequence is underrepresented in training, (4) some combination of all the factors above. to test these hypotheses, we conduct five experiments designed to help the model learn to copy with different biases by adding to the training set for each language 2,000 foot_3 dummy data points generated in five different ways, explained below. considering that a natural linguistic sub-unit of a word is a syllable, we propose to use sub-figure 3 : "wug test" results. +copy-2k-char adds random strings generated with the alphabet. +copy-2k-substr adds random strings generated with the n-gram set. +hall-2k-char adds data hallucinated with the method by @xcite . +hall-2k-substr adds data hallucinated with our method. strings of syllable-like length for the +copy-2ksubstr method. the input of this method is the set of bigrams, trigrams and four-grams from the language's training data. for both methods, we generate the dummy lemma by uniformly sampling from the input and concatenating the sampled items to a random length between the minimum and maximum word length we see in the training data. the output of the dummy lemma generation process is a triple of a dummy lemma, a special symbol copy and the dummy lemma, which is added to the initial "wug test" training set for data augmentation. in addition, we propose to generate the dummy stem by uniformly sampling from substrings of syllable-like length, i.e. the bigram, trigram and four-gram set. this experiment is referred to as +hall-2k-substr. specifically, both data hallucination methods (illustrated in figure 2 (b) ) take as input a triple from the training set, aligns the lemma and the target form with the alignment method from sigmorphon 2016 shared task baseline @xcite , finds the common substrings between the lemma and the target form as the stem, replaces the stem with a dummy stem, and outputs a dummy triple which is adopted for data augmentation. our proposed method is different from @xcite 's method at the dummy stem generation step in two main aspects: (1) instead of sampling from the alphabet, we sample from the set of bigrams, trigrams and four-grams. (2) instead of forcing the dummy stem to be of the same length as the stem to be replaced, we only constrain the minimum and maximum length of the stem based on the training data. in addition, for discontinuous stems, we only replace the first part of the stem. foot_4 4 results and discussion "wug test" with data augmentation figure 3 shows results for the "wug test"-like setting and results after augmenting the initial training set with different methods. every language sees a substantial improvement with data augmentation, indicating that the transformer model in the vanilla "wug test" circumstance will not learn a copy bias well. the substring-based data hallucination we propose, +hall-2k-substr, achieves accuracies which are substantially higher than other methods for most languages. for turkish and nyanja, +hall-2ksubstr is lower than the best performance, but the difference is not obvious. for lingala, +hall-2ksubstr has the same best performance as +hall-2kchar. the consistent advantage of +hall-2k-substr implies that substrings of syllable-like length is more helpful than individual characters for data hallucination. it also provides support to the fourth hypothesis we made in section 3 that the poor performance of the transformer in the vanilla "wug test"-like setting is due to a combination of factors including missing copying bias for letters, subsequences of letters and even entire stems. common practice vs "wug test" figure 1 plots the transformer accuracies with standard deviations in the common-practice setting, vanilla "wug test"-like setting, and "wug test"-like setting with data augmentation by the substring-based data hallucination methods @xcite . though data augmentation can improve the model's performance for a "wug test", results are still inferior to the common practice setting without any data augmentation for most languages, especially the morphologically challenging 2018 conll-sigmorphon languages. ## conclusion in this work, we examine limiting the number of training lemmata and keeping training lemmata disjoint from the evaluation sets in morphological inflection. by comparing the performance of a data information