Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (2022)

Abstract: Despite strong performance in many sequence-to-sequence tasks, autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between the ground-truth prefixes used during training and the model-generated prefixes used at inference time. Scheduled sampling is a simple and empirically successful approach which addresses this issue by incorporating model-generated prefixes into training. However, it has been argued that it is an inconsistent training objective leading to models ignoring the prefixes altogether. In this paper, we conduct systematic experiments and find that scheduled sampling, while it ameliorates exposure bias by increasing model reliance on the input sequence, worsens performance when the prefix at inference time is correct, a form of catastrophic forgetting. We propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance. Experiments on four IWSLT’14 and WMT’14 translation datasets demonstrate that our approach alleviates catastrophic forgetting and significantly outperforms maximum likelihood estimation and scheduled sampling baselines.

## introduction autoregressive models trained with maximum likelihood estimation (mle) constitute the dominant approach in several sequence-to-sequence tasks, such as machine translation @xcite ), text summarization @xcite , and conversational modeling @xcite . however, this paradigm suffers from exposure bias @xcite , i.e. during training the model generates tokens by conditioning on the ground-truth prefixes, while at inference time model-generated prefixes are used instead. since the model is never exposed to its own errors, if a token is mistakenly generated during inference the error will be propagated along the sequence @xcite . prior work has attributed to exposure bias various forms of text degeneration, such as repetitiveness, incoherence, and hallucinations @xcite , i.e. outputs which contain information irrelevant to the input sequence. @xcite introduced scheduled sampling to address exposure bias in mle-trained autoregressive models. scheduled sampling uses a stochastic mixture of ground-truth and modelgenerated prefixes during training, thereby allowing the model to learn how to recover from its own errors. while various alternatives to mle have been proposed to mitigate exposure bias @xcite @xcite , scheduled sampling remains one of the most popular due to its simplicity and performance improvements in many conditional sequence generation tasks @xcite @xcite . conversely, other studies have reported that using scheduled sampling may hurt performance @xcite . the dominant hypothesis for these negative results is that scheduled sampling creates models that are more likely to recover from their own mistakes by training them to ignore the prefixes entirely @xcite . however, no attempt has been made to empirically assess this hypothesis. thus it is still unclear how scheduled sampling affects training. in this paper, we provide insights into the working mechanisms of scheduled sampling. following @xcite , we apply layerwise relevance propagation (lrp) @xcite to quantify the contributions of the input sequence and the prefix during inference, and empirically evaluate the hypothesis of @xcite . we find that models trained with scheduled sampling increase their reliance on the input sequence, and therefore mitigate exposure bias by depending less on the potentially incorrect model-generated prefix. however, we also observe that this has the sideeffect of worsening the predictions when the modelgenerated prefix is correct. to address this form of catastrophic forgetting @xcite , we propose using elastic weight consolidation (ewc) @xcite to regularize the parameter updates in scheduled sampling so that the performance is not affected when the prefix is correct. experiments on the commonly-used iwslt'14 german-english, iwslt'14 vietnamese-english, wmt'14 english-german, and wmt'14 english-french translation datasets show that our proposed method mitigates catastrophic forgetting and significantly improves translation performance, in terms of bleu @xcite , over mle, standard scheduled sampling, and a recently proposed scheduled sampling variant @xcite . importantly, performance gains occur in both long short-term memory (lstm) @xcite and transformer @xcite models, and across different annealing schedules, demonstrating that our proposed ewcregularized scheduled sampling is more robust and easier to tune. finally, human evaluation showcases that our method can improve both translation adequacy and fluency. ## scheduled sampling autoregressive models estimate the conditional probability of the output y given the input x one token at a time in a monotonic fashion: here i indicates the i-th output sequence in the dataset and l (i) is the length of the i-th output algorithm 1 scheduled sampling an annealing schedule is used to gradually decrease the probability a of conditioning using the ground-truth prefix during training. typically, for each mini-batch b, a is decreased using the following annealing schedules: here k is a hyperparameter which controls the speed of decay of a in each schedule. algorithm 1 summarizes training with scheduled sampling. ## analysis of scheduled sampling in this section, we propose two ways to investigate how scheduled sampling works. first, we use layerwise relevance propagation (lrp) @xcite to examine the contributions of the input sequence and the prefix during inference, and thus assess the hypothesis of @xcite . then we treat training with scheduled sampling as a form of progressive fine-tuning, by assuming that the model-generated prefixes can be viewed as a downstream task we adapt the model to after it is trained with the ground-truth prefixes. to this end, we apply teacher-forcing at inference time to quantify the impact of catastrophic forgetting on models trained with scheduled sampling. ## experiments in this section we apply the methods discussed in section 3 to examine if training with scheduled sampling leads to models ignoring the prefixes, and also determine whether scheduled sampling causes catastrophic forgetting. ## elastic weight consolidation for scheduled sampling section 4 shows that even though scheduled sampling addresses exposure bias by increasing model reliance on the input sequence, it also leads to output degradation due to catastrophic forgetting @xcite when the model-generated prefix is correct at inference time. elastic weight consolidation (ewc) @xcite is an effective regularization-based approach for mitigating catastrophic forgetting during fine-tuning with successful applications in nmt @xcite . to this end, we use ewc to regularize conditioning with the model-generated prefixes ŷ<t to avoid forgetting using the ground-truth @xmath0. ewc penalizes changes to the parameters which are important for the ground-truth prefixes, while adapting more the parameters that are less important: ## experiments with ewc in this section, we perform experiments with the proposed ewc-regularized scheduled sampling variant following the setup described in section 4. to further demonstrate the effectiveness of our method, we also conduct experiments on two large-scale wmt'14 translation datasets, the english-german (en-de) and english-french (en-fr). in addition, we report results using the sentence-oracle scheduled sampling variant proposed by @xcite , which conditions on model-generated prefixes obtained via beam search. ## conclusion in this work, we conduct systematic analyses to examine the strengths and weaknesses of scheduled sampling. by applying lrp to quantify the influence of the input sequence and the prefix during prediction, we show that models trained with scheduled sampling increase their reliance on the former to address exposure bias. however, we also demonstrate that as a side-effect, this leads to output degradation due to catastrophic forgetting when the prefix generated by the model is correct. accordingly, we propose using ewc to better balance mitigating exposure bias with retaining performance. our ewc-regularized scheduled sampling variant alleviates catastrophic forgetting and significantly improves performance over mle and scheduled sampling baselines on four iwslt'14 and wmt'14 translation datasets.