Noise Isn’t Always Negative: Countering Exposure Bias in Sequence-to-Sequence Inflection Models (2020)

Abstract: Morphological inflection, like many sequence-to-sequence tasks, sees great performance from recurrent neural architectures when data is plentiful, but performance falls off sharply in lower-data settings. We investigate one aspect of neural seq2seq models that we hypothesize contributes to overfitting - teacher forcing. By creating different training and test conditions, exposure bias increases the likelihood that a system too closely models its training data. Experiments show that teacher-forced models struggle to recover when they enter unknown territory. However, a simple modification to the training algorithm to more closely mimic test conditions creates models that are better able to generalize to unseen environments.

## introduction morphological inflection has gained substantial interest in recent years due to a number of shared tasks focusing on morphology learning @xcite @xcite @xcite . systems for inflection are trained on a sequence of pairs: {lemma, morphosyntactic descriptor (msd)}, free of context, and must produce as output an inflected word form. for example, given the input pair {run, v.ptcp;prs}, a successful system should produce the present participial form: running. the shared tasks have illuminated several idiosyncrasies of inflection. although the state-of-the-art methods have been inspired by neural machine translation (nmt), inflection is, in many ways, a more straightforward task. often, a majority of characters can be copied directly from input to output, and reordering of tokens is not necessary to the extent that it is in translation. thus, systems with copymechanisms @xcite , and hard, monotonic attention @xcite tend to perform well, even when data is scarce. recurrent neural architectures require that the output from a previous time step be fed back into the model. during training, existing systems all use a variant of teacher forcing, which feeds gold-standard tokens into the model at time t + 1. this methodology differs significantly from how the model progresses at test-time, where silver tokens must be used. this problem has been dubbed exposure bias by @xcite . we hypothesize that exposure bias can lead models to overfit the training data, particularly in low-resource settings. in this paper, we compare teacher forcing to silver label propagation, or student forcing, where model predictions are fed into the decoder during training instead of gold standard labels. our experiments on a geographically and linguistically diverse set of 10 languages show that student forcing typically outperforms teacher forcing in a low-resource setting. our analysis of the results suggests that teacher forced models are, indeed, overfitting the training data by ignoring input characters at test time. we note substantial gains when student forcing is applied to soft-attentional systems, which bears promise for tasks other than inflection; however, less prominent gains are also observed in conjunction with hard attention. ## methods we investigate the role that teacher forcing plays on morphological inflection by comparing it against a simple alternative learning strategy, which we call student forcing. in our experiments, we apply student forcing to three established neural inflection models: the pointer-generator network (pg), a soft attention model originally introduced for document summarization @xcite and subsequently applied to inflection by @xcite , a neural transducer utilizing hard attention trained on aligned lemmata and inflected word forms @xcite (ha) and modified with an explicit copy mechanism, and a later development of the ha system which applies minimum risk training in order to avoid relying on an explicit aligner for the lemma and inflected word form during training (mrt) @xcite . ## related work to the best of our knowledge, there has been little investigation into the role of teacher forcing for inflection systems. however, @xcite apply scheduled sampling for rnn's in the context of image captioning, constituency parsing and speech recognition. this approach has been extended to transformers by @xcite . during training, @xcite randomly sample either the gold standard token y t-1 or the model suggestion ŷt-1 when predicting y t . the probability for selecting a model suggestion rather than the gold standard example is continuously increased during training which they call scheduling. our work can be seen as an application of this approach except that we make the choice between gold standard context and the model suggestion at the example-level instead of the level of individual characters. according to our preliminary experiments, this delivered superior results in the task of inflection. another difference is that we fix the probability for choosing the model suggestion throughout training and treat this as a hyperparameter. we do this because we did not observe consistent improvements from scheduling. @xcite present a modification of the approach by @xcite for named-entity recognition and machine translation. they replace the argmax model prediction with an average of all output embeddings, weighted by the prediction scores. also related to scheduled sampling is the approach by @xcite who experiment with beam search. the downside of this approach is that it can substantially increase training times. ## data and experiments in this section, we lay out our experiment schedule, and describe the system architectures, hyperparameters, and data specifications. we also motivate our choice of evaluation metrics. ## discussion we now turn our attention to the output of our systems, and devote some discussion to certain trends that we observed. ha vs. pg it is quickly apparent that the hard attentional model is superior to the pointer-generator, at least when training data is low -one hundred training examples is simply insufficient to learn an adequate soft attention mechanism. however, once training data is available in sufficient quantities, the pointer-generator approaches, and even, in some circumstances, surpasses the hard attentional model. in particular, the navajo model struggles with the constraints present with hard attention, but basque and persian also benefit from soft attention. we also note that while the pointer-generator sees the most benefit from student-forcing in the low and medium settings, the hard attentional model steadily improves as data increases. we find this to be a surprising result -by the time a model has observed 10,000 training examples, exposure bias is much less of a concern. it is possible that the soft attention mechanism is performing much of the same type of regularization that we would expect from student forcing -but only becomes strong enough to completely supplant it in the higher data setting. the ha model, conversely, requires pre-computed alignments, and the quality of these alignments may impact the results significantly. using these alignments as an approximation of when the model must learn insertion, deletion, or substitution operations, we note that navajo requires, on average, more than 50% more substitutions than the next closest languages -arabic, basque, and persian. basque and persian, in particular perform worse with hard attention in the medium and high settings, while arabic does so in the high setting. on the other extreme, english, sanskrit, and german perform few substitutions, and clearly prefer hard attention. substitutions are much more complex operations than either insertions or deletions, and much harder to generalize. it is possible that hard attention is too strict to fully capture the intricacies of a complex substitutional morphology. insertions / deletions when we look closer at insertions and deletions, particularly in the low setting, we observe an interesting pattern of errors. analyzing english -a language that benefits immensely from student forcing, we see that teacher-forced models regularly adopt a "clipping" strategy, eliminating in an attempt to further classify errors, we calculate the number of prediction errors that are insertions or deletions for each language, as well as the average number of characters that are inserted or deleted with each operation. furthermore, we calculate whether one or the other is more likely to be corrected by student forcing. we find that, on average, teacher forcing makes more insertion mistakes than deletions (although english is an exception, with 25% more deletions). the student model corrects nearly 37% of insertions, and 27% of deletions. what's more, the student model also significantly reduces the length of errors -reducing the average insertion mistake from 2.5 characters to 1.8, and the average deletion mistake from 2.0 characters to 1.6. by correcting deletion errors, student forcing manages the clipping problem. correcting spurious insertions, on the other hand, eliminates errors that occur when the model finds itself in an unfamiliar environment, and it generates false characters in an attempt to escape. for example, the pluralization of the german "nadelbaum" requires umlaut to produce "nadelbäume", but ä is a relatively rare character, so the model incorrectly produces "m", which drives the model into another unusual environment. although the model eventually recovers, the predicted output: "*nadelbmumummue" is quite distant from the gold target. navajo many of the languages we have investigated follow a regular pattern of results, particularly in the low setting -the pointer-generator performs poorly with teacher forcing, improves with student forcing, is outpaced by the ha model, which in turn then benefits from student forcing and mrt training. however, across all settings and experiments, there is one outlier -navajo. navajo sees little improvement from student forcing, regardless of the training algorithm. furthermore, its performance lags behind other languages when the data increases. when mrt is applied, ed spikes under teacher forcing -quadrupling in the low setting as exploration reveals a weak distribution. we hypothesize that navajo has several typological features that make it ill-suited both to student-forcing and hard-attentional low-resource models. as previously indicated, navajo has a significant number of stem changes, which may make learning with hard-attention difficult -nearly all of the improvement of the pg over the ha in the high setting can be attributed to navajo. secondly, navajo is a primarily prefixing language. our implementation of student forcing progresses from the beginning of the word to the end, and makes some strong assumptions in doing so. suffixing languages will learn to copy characters at the beginning of the word, an easy task that can be learned quickly. this establishes a strong language model before any true inflection occurs -by the time the first suffix character is produced, the model will have a strong prediction of the previous characters. this is not true of prefixing languages, which have no prior language model to inform prefixes; the information from the teacher serves as a strong regularizer. zulu is also a strongly prefixing language, but the majority of the inflection in our data set consists of insertion. if insertions are easier to learn than substitutions, then it is reasonable that zulu will eventually be able to learn a strong inflectional model, as long as it has some access to gold recurrence. ## conclusion we have presented student forcing, a simple method for countering exposure bias. the dissimilarity between how information is presented to the decoder during training and inference can lead models to overfit to the particular environments observed in the training data. by injecting some noise into training, through the modification of the recurrent step of the rnn, we create a training scenario that more closely models what systems will see at test time. we demonstrate that this simple change can lead to moderate improvements in morphological inflection, a simpler version of the general sequence-to-sequence task. whether measured in accuracy or edit distance, the evidence shows that student forcing injects enough variety into training that it becomes less prone to overfitting. while noise is typically seen as an obstacle to overcome in machine learning, a little bit injected in the right place can encourage learning over memorization.