Improving Text Generation with Student-Forcing Optimal Transport (2020)

Abstract: Neural language models are often trained with maximum likelihood estimation (MLE), where the next word is generated conditioned on the ground-truth word tokens. During testing, however, the model is instead conditioned on previously generated tokens, resulting in what is termed exposure bias. To reduce this gap between training and testing, we propose using optimal transport (OT) to match the sequences generated in these two modes. We examine the necessity of adding Student-Forcing scheme during training with an imitation learning interpretation. An extension is further proposed to improve the OT learning for long sequences, based on the structural and contextual information of the text sequences. The effectiveness of the proposed method is validated on machine translation, text summarization, and text generation tasks.

## introduction natural language generation is an essential component of many nlp applications, such as machine translation @xcite , image captioning @xcite ), text summarization @xcite , dialogue systems @xcite , and machine comprehension @xcite . generating human-like natural language is typically cast as predicting a sequence of consecutive words in a recurrent manner. maximum likelihood estimation (mle) is commonly employed as the objective to train such textgeneration models, maximizing the log-likelihood of producing the ground-truth tokens within a sentence or paragraph @xcite . in recurrent neural network (rnn) models, this is also known as teacher-forcing (tf) @xcite , due to the use of ground-truth tokens for next-token prediction. however, in the maximum likelihood paradigm, previous observed tokens are usually provided during training, giving rise to an issue termed exposure bias @xcite : the ground-truth * * equal contribution tokens seen by the model during training are not available at inference time. during inference, the model is required to use outputs from the last step instead of the unseen ground-truth, which is often referred to as student-forcing (sf). as a result, there is a discrepancy between training and inference, accumulating errors along the sequencegeneration trajectory @xcite . this challenge has been addressed by incorporating model-generated text at training time. @xcite proposed scheduled sampling (ss), where the training samples are systematically tampered with by replacing some of the ground-truth tokens with model-predicted tokens. other works regard text generation as a sequential decision making problem, applying reinforcement learning (rl) techniques @xcite . in particular, quantitative evaluation metrics such as bleu and rogue are used as sequence-level rewards for modelgenerated texts. despite the encouraging results reported, concerns have been raised w.r.t. the above strategies. scheduled sampling is known to be statistically inconsistent and fails to address the fundamental issues @xcite . rl solutions usually suffer from slow and unstable training due to the high variance of reinforce-based policy gradients; consequently, specific training techniques are often needed @xcite . additionally, the results from rl models often do not correlate well with human evaluations, as the rewards used are typically biased towards specific aspects of a language model @xcite . on the other hand, recent developments in likelihood-free modeling techniques, most prominently the generative adversarial networks (gans), tackle exposure bias in a more principled fashion @xcite @xcite . gan-based text models, however, suffer from a number of severe dif-ficulties, including mode collapse, where generated text looks real but lacks necessary diversity @xcite . additionally, the training of gan-based models is often unstable, and model learning easily breaks down in the event of vanishing or exploding gradients @xcite . therefore, existing adversarial methods may not be able to match sentences generated by student-forcing with ground-truth sentences. to mitigate the challenges from the adversarial methods, we utilize a sequence-matching loss based on optimal transport (ot), which avoids a neural discriminator and delivers a smoother gradient for the generator. recently, @xcite leverage ot loss based on a teacher-forcing scheme, however, it degenerates to wordlevel matching, making it difficult to capture temporal semantic information. in this work, in order to enable sequence-level matching, we instead propose an ot-based sequence-level training scheme to directly optimize the discrepancy loss between the ground-truth and free-running text samples. further, we introduced various ot cost functions for loss calculation. the significance of this work is two-fold: firstly, this approach alleviates the exposure bias, boosting model performance at the inference stage by using a sequencelevel objective between free-running output and reference. secondly, with the use of ot, this approach provides a direct objective that is easy and robust to optimize, without biasing towards a specific, manually-defined metric. our work provides the following contributions: i) we introduce a novel method for text generation called student-forcing ot (sfot), leveraging ot loss to improve long-term sequence sampling. ii) a new context-preserving ot approach is proposed to effectively match a text sequence with order information. iii) we examine the necessity of integrating ot with student-forcing via imitation learning. iv) the proposed models are robust demonstrated by extensive empirical evaluations on neural machine translation (nmt), text summarization, and neural text generation (nlg). ## student forcing optimal transport to reduce exposure bias, the output sequences of the generator in the teacher-forcing (training) and student-forcing (inference) stages should be indis-tinguishable. therefore, we propose to use ot loss to measure sequence matching distance between the two stages, in conjunction with the maximum likelihood estimate. ## related work text generation natural language generation (nlg) is a challenging nlp task. neural language models parameterized by autogressive architectures are widely used for nlg. to improve the global control ability of generated sentences, variational auto-encoders are considered for language generation (bowman et al., 2016; fu et al., 1 the discount factor is set as one for simplicity 2019; @xcite . recently, gpt-2 @xcite and @xcite improve the generation fluency via pre-training on massive text corpus. all of them are trained with mle using teacher-forcing, which are known to suffer from exposure bias in principle @xcite . several methods have been proposed to solve the problem, including @xcite . adversarial training techniques were also proposed @xcite @xcite @xcite @xcite @xcite . however, adversarial-based nlg models can suffer from gradient vanishing and unstable training. indeed, @xcite argues that a temperature sweeping approach on mle can outperform gan-based models. our model further improves this work by adopting a principled sequence-matching loss via optimal transport and achieve state-of-the-art results on nlg tasks. optimal transport optimal transport is widely employed for a variety of nlp tasks, including document classification @xcite , word embedding space alignment @xcite , and generative adversarial networks @xcite . the most related work to ours is @xcite . we discuss the difference between the proposed sfot and tfot as follows: while the overall performance of sfot is superior to tfot only by a decent margin on standard datasets, we emphasize that the main advantage of sfot is for long sentences. as shown in the break-down analysis (figure 4 ), sfot is significantly better than tfot when sentence length is larger than 60. this highlights the critical contribution of this paper to addressing the exposure bias problem, which longer sentences usually suffer from. in section 2.3, we provide theoretical justification on why sfot can reduce exposure bias, while tfot still suffers from it: tfot is based on partial expert trajectories and induces a bias occupancy measure, while our proposed method sfot uses previous self-generated words and can obtain an optimal policy. sequence matching direct sequence matching has been explored widely in various machine learning tasks. jaccard distance has been used to retrieve prototypes for sentence generation @xcite . chernoff distance has been applied to image classification @xcite . however, these distances assume each instance in the sequence is independent, ignoring temporal information. these distances consequently measure sequence alignment poorly, as they miss semantic relationships inside a sentence (e.g.cause and effect). dynamic time warping (dtw) @xcite and connectionist temporal classification (ctc) @xcite ) consider temporal information, and have been adopted widely in speech recognition. however, these losses preserve strict ordered alignment, and hence cannot be directly applied to text sequences. ## experiments we perform experiments on neural machine translation (nmt), abstractive text summarization and unconditional natural language generation (nlg) tasks. algorithms are implemented in tensorflow and trained on an nvidia titan x gpu. ## conclusions we have introduced sfot to mitigate exposure bias in text generation. the proposed model captures positional and contextual information of word tokens in ot matching. experiments on neural machine translation, text summarization, and text generation have demonstrated the effectiveness of our sfot algorithm, yielding improved performance over strong baselines on these tasks.