R - T ea F or: Regularized Teacher-Forcing for Abstractive Summarization (2022)

Abstract: Teacher-forcing is widely used in training sequence generation models to improve sampling efficiency and to stabilize training. However, teacher-forcing is vulnerable to the exposure bias problem. Previous works have attempted to address exposure bias by modifying the training data to simulate model-generated results. Nevertheless, they do not consider the pairwise relationship between the original training data and the modified ones, which provides more information during training. Hence, we propose Regularized Teacher-Forcing (R-TeaFor) to utilize this relationship for better regularization. Empirically, our experiments show that R-TeaFor outperforms previous summarization state-of-the-art models, and the results can be generalized to different pre-trained models.

## introduction recently, the encoder-decoder models have made a giant leap in sequence generation tasks. these models are primarily trained with the teacher-forcing method @xcite , which has been praised for enhancing sampling efficiency and training stability. however, as teacher-forcing only exposes models to the training data distribution rather than their prediction distribution, models suffer from the exposure bias problem @xcite . previous works for solving exposure bias can be roughly split into four categories: regularization, contrastive learning, reinforcement learning, and sampling-based methods. regularization methods such as dropout @xcite and perturbation @xcite focus on problems of data distribution differences between training data and inference data, which is also related to exposure bias. @xcite propose r-drop, which refines dropout by forcing predictions of different sub-models generated by dropout to be consistent with each other. @xcite figure 1 : an illustration of r-teafor. the output distributions are generated both by the original target sequence and the augmented target sequence. r-teafor aims to reduce the distance of two distributions by additional symmetric kl-divergence loss. contrastive learning is widely used in either pretraining or fine-tuning sequence generation models to provide more information inside the training data. word vectors @xcite and language models are the early applications for contrastive learning in natural language processing. for summarization tasks, both extractive-based approach @xcite and abstractive-based approach can be combined with contrastive learning. these methods mostly entail an additional contrastive loss @xcite @xcite , or use the contrastive data to learn an additional model for reranking @xcite . the challenge of contrastive learning is to find a suitable contrastive pair that is neither too easy to distinguish nor too hard to learn. reinforcement learning based on policy gradient @xcite and their variations have been used extensively for sequence generation @xcite @xcite . pang and he (2021) formulate the sequence generation learning problem as an off-line reinforcement learning with expert demonstrations that addresses exposure bias by training the model on its state/history distribution. however, reinforcement learning still requires a warm start phase or regularization powered by traditional teacher-forcing. scheduled sampling @xcite is a representative sampling-based approach that schedules the sampling rate between ground-truth tokens and model-generated tokens according to training steps. @xcite propose another variation by refining the sampling rate with beam search. @xcite further suggest scheduling the sampling rate according to the decoding steps. although sampling-based approaches can simulate the data distribution in the inference stage during training, the modified training data might deviate from the original one. such deviation will mislead the model to learn from noisy training data. we, therefore, present regularized teacher-forcing (r-teafor), which considers the pairwise relationship between the original and modified data. figure 1 illustrates the general framework of r-teafor. during training, the model generates the token distribution twice-once with the ground-truth sequence as input and once with the augmented sequence modified from the same ground-truth. r-teafor forces the above two distributions to be consistent with each other by introducing the symmetric kl-divergence loss. the augmented sequence serves as a regularizer to prevent the model from exposure bias. meanwhile, the model can still learn from the original ground-truth sequences. though the concept of r-teafor is straightforward, we find it surprisingly effective through ex-tensive experiments on two summarization benchmarks. moreover, the results can be generalized to different pre-trained models such as bart @xcite and pegasus @xcite . ## r-teafor before elaborating on r-teafor, we first formulate the abstractive summarization problem in a more general sequence generation manner. sequence generation models take source sequence @xmath0, s 2 , ..., s |s| as input, and generate target sequence @xmath1, t 2 , ..., t |t| autoregressively. in other words, the generating process can be modeled by the following probability: during training, the previous target sequence @xmath2. however, as the ground-truth tokens become unavailable during inference, we apply model predictions in @xmath3. this discrepancy leads to the exposure bias problem. r-teafor handles exposure bias by forcing the model to learn from both ground-truth sequence t and augmented sequence t ′ simultaneously. for constructing the augmented sequence t ′ = t ′ 1 , t ′ 2 , ..., t ′ |t ′ | , we randomly mask the ground-truth tokens t i with probability β. while decoding the i-th token, r-teafor generates the probability distribution of t i twice-once with the ground-truth sequence and once with the augmented sequence. the basic negative loglikelihood loss is calculated as finally, we combine negative log-likelihood loss and symmetric kl-divergence loss with hyperparameter α controlling the weight of two losses. by this measure, r-teafor encourages the model to make a consistent prediction regardless of the input sequence being augmented or not. unlike sampling-based methods which use original training data only in the warm-up phase, r-teafor always trains the model with pairwise original/augmented data. during training, the distribution generated based on the ground-truth sequence can serve as a guide when the model attempts to regularize on a much noisy augmented sequence. without such guidance, models may regularize to a sub-optimal point as models are bounded with fewer constraints. ## conclusion we present regularized teacher-forcing (r-teafor), which utilizes the pairwise relationship between the original training data and augmented data for model training. the original data can serve as a guide when the model attempts to regularize on much noisy augmented data. with extensive experiments, we discuss the necessity of the pairwise relationship in r-teafor. in addition, we show that r-teafor outperforms previous state-of-the-art models on cnn/dailymail and xsum benchmarks. we believe that r-teafor has strong potential to apply to other sequence generation tasks such as machine translation and question answering.