Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation (2021)

Abstract: Although teacher forcing has become the main training paradigm for neural machine translation, it usually makes predictions only conditioned on past information, and hence lacks global planning for the future. To address this problem, we introduce another decoder, called seer decoder, into the encoder-decoder framework during training, which involves future information in target predictions. Meanwhile, we force the conventional decoder to simulate the behaviors of the seer decoder via knowledge distillation. In this way, at test the conventional decoder can perform like the seer decoder without the attendance of it. Experiment results on the Chinese-English, English-German and English-Romanian translation tasks show our method can outperform competitive baselines significantly and achieves greater improvements on the bigger data sets. Besides, the experiments also prove knowledge distillation the best way to transfer knowledge from the seer decoder to the conventional decoder compared to adversarial learning and L2 regularization.

## introduction neural machine translation (nmt) @xcite @xcite @xcite has achieved great success and is drawing larger attention recently. most nmt models are under the attention-based encoder-decoder framework which assumes there is a common semantic space between the source and target languages. the encoder encodes the source sentence to the common space to get its meaning, and the decoder projects the source meaning to the target space to generate corresponding target words. whenever generating a target word at a time step, the decoder * the code: https://github.com/ictnlp/seerforcingnmt needs to retrieve the attended source information and then decodes into a target word. the underline principle which makes sure the framework works is that the information hold by the source sentence and its target counterpart is equivalent. thus the translation procedure can be considered to decompose source information into different pieces and then to convert each piece to a proper target word according to bilingual context. when all the information encoded in the source sentence is throughly processed, the whole translation has been generated. neural machine translation models are usually trained via maximum likelihood estimation (mle) @xcite and the operation form is known as teacher forcing @xcite . the teacher forcing strategy performs one-step-ahead predictions with the past ground truth words fed as context and forces the distribution of the next prediction to approach a 0-1 distribution where the probability of the next ground truth word corresponds to 1 and others to 0. in this way, the predicted sequence is trained to be close to the ground truth sequence. from the perspective of information division, the function of teacher forcing is to teach the translation model how to segment source information and derive the ground truth word from the source information at a maximum probability. however, teacher forcing can only provide upto-now ground truth words for one-step-ahead predictions and hence lacks global planning for the future. this will result in local optimization especially when the next prediction is highly related to the future. besides, as the translation grows, the previous prediction errors will be accumulated and affect later predictions @xcite . this is the important reason why nmt models cannot always produce the ground truth sequence during training. therefore, it is more possible to achieve global optimization by getting to know the future ground truth words. this can lead to better crossattention to the source sentence and thus better information devision. but unfortunately, ground truth can be only obtained during training and we cannot inference with future ground truth at test. to address this problem, we introduce an additional seer decoder into the encoder-decoder framework to integrate future information. during training, the seer decoder is used to guide the behaviors of the conventional decoder while at test the translation model only inferences with the conventional decoder without introducing any extra parameters and calculation cost. specifically, the conventional decoder only gets past information participating in the next prediction, while the seer decoder has both the past and future ground truth words engaged in the next prediction. both decoders are trained to generate ground truth via mle and meanwhile the conventional decoder is forced to simulate the behaviors of the seer decoder via knowledge distillation @xcite . in this way, at test the conventional decoder can perform like the seer decoder as if it knew the future translation. we conducted experiments on two small data sets (chinese-english and english-romanian) and two big data sets (chinese-english and english-german) and the experiment results show that our method can outperform strong baselines on all the data sets. in addition, we also compared different mechanisms of transferring knowledge and found that knowledge distillation is more effective than adversarial learning and l2 regularization. to the best of our knowledge, this paper is the first to explore the effects of the three mechanisms simultaneously in machine translation. 1 (n-1) ⇥ < l a t e x i t s h a 1 _ b a s e 6 4 = " c 5 v 4 y v c k q x a j v e e f + d g 0 c k j 6 l y @xmath0 we introduce our method on the basis of transformer which is under the encoder-decoder framework @xcite . our model consists of three components: the encoder, the conventional decoder and the seer decoder. the architecture is shown in figure 1 . the encoder and the conventional decoder work in the same way as the corresponding components of transformer do. the seer decoder integrates future ground truth information into its self-attention representation and calculates cross-attention over source hidden states with the self-attention representation as the query. during training, the encoder is shared by the two decoders and both decoders perform predictions to generate ground truth. the behaviors of the conventional decoder are guided by the seer decoder via knowledge distillation. if the conventional decoder can predict a similar distribution as the seer decoder, we think the conventional decoder performs like the seer decoder. then we can only use the conventional decoder for test. the details of the encoder and the conventional decoder can be got from @xcite . assume the input sequence is @xmath1, ..., x j ), the ground truth sequence is y * = (y * 1 , ..., y * i ) and the generated translation is @xmath2, ..., y i ). we will give more description to the seer decoder and the training in what follows. ## training in our method, only the conventional decoder is employed for test and the seer decoder is only used to guide the conventional decoder during training. given a sentence pair x, y * in the training set, the conventional decoder and the seer decoder can predict a distribution for target position i as p c (y i |y * <i , x) and p s (y i |y * >i , y * <i , x), respectively. the two decoders are both trained by comparing its predicted distribution with the 0-1 distribution of the ground truth word by minimizing the cross entropy, that is to maximize the likelihood of the corresponding ground truth word. as the two decoders involve different information for next prediction, we call the training strategy teacher forcing and seer forcing, respectively. the cross-entropy loss for the conventional decoder is the conventional decoder is further trained to get close to the distribution of the seer decoder via knowledge distillation. in knowledge distillation, the conventional decoder (the student) has to not only match the one-hot ground truth word, but fit the distribution over the target vocabulary v drawn by the seer decoder (the teacher). the knowledge distillation loss can be formalized as different from the conventional knowledge distillation which first trains the teacher via cross entropy against ground truth, then fixes the teacher and only trains the student, we train all the parameters from the scratch, but we still follow the above rule to keep the teacher (i.e. the seer decoder) unchanged in the process of distillation. to do this, we do not update the parameters of the seer decoder through the loss l kd , that is, we only back propagate gradients to the seer decoder through l s , but not through l kd . ## related work reinforcement-learning-based methods also encode future information in the rewards to supervise fine-tuning of the translation model. the rewards are worked out either by sampling future translation with the reinforce algorithm @xcite @xcite , or by directly calculating a value with the actor-critic algorithm @xcite . this set of methods only give a weak supervision to the nmt model through rewards and suffer from unstable training. in contrast, @xcite propose to train autoregressive nmt with the probabilistic n-gram based gleu @xcite and @xcite propose to minimize the bag-of-ngrams difference for nonautoregressive nmt so that the two methods can abandon reinforcement learning and perform training directly by gradient descent. another set of methods introduce future information into inference with additional pass of decoding or extra components at test. niehues et al. et al. (2019a) , zhou et al. (2019b) and zhang et al. (2019b) also maintain a forward decoder and a backward decoder to decode simultaneously but they interact to each other when making predictions. @xcite introduce a future-aware vector at test which is learned via the knowledge distillation framework during training. the difference between this set of methods and our method is that our method does not require any other cost at test and is easy to use. there are some other works which integrate future information during training while only perform one-pass decoding. @xcite introduce a twin network to perform bidirectional decoding simultaneously during training and force the hidden states generated by the two decoders to be consistent, then at inference it can only use the forward decoder. but in this method the two decoders act as a counterpart to each other and no decoder plays a role of teacher, which determines that it can only be trained via l 2 regularization, not knowledge distillation which has proven in the experiments more effective than l 2 regularization. @xcite introduce an evaluation module to give each translation more reasonable evaluation when it cannot match the ground truth. the evaluation is conducted from the perspective of fluency and faithfulness which both need the participation of past and future information. the difference from the method proposed in this paper is their method uses self-generated translation as past information and does not train with knowledge distillation. some researchers work in another perspective by introducing future information. @xcite propose to employ future source information to guide simultaneous machine translation with knowledge distillation, so that the incompleteness of source can be mitigated. @xcite and @xcite propose to model past and future information for the source to help the decoder focus on untranslated source information. ## conclusion in order to help the nmt model to make good global planning at inference, we propose to introduce a seer decoder which embodies future ground truth to guide the behaviors of the conventional decoder. to this end, we employ the method of knowledge distillation to transfer future information from the seer decoder to the conventional decoder. at test, the conventional decoder can perform translation on its own as if it knew some future information. the experiments indicate our method can outperform strong baselines significantly on four data sets. we are also the first to explore learning mechanisms of knowledge distillation, adversarial learning and l 2 regularization and knowledge distillation has proven to be the most effective one.