P rophet N et: Predicting Future N-gram for Sequence-to- S equence P re-training (2020)

Abstract: This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.

## introduction large-scale pre-trained language models @xcite @xcite and sequence-to-sequence models @xcite @xcite have achieved remarkable success in downstream tasks. autoregressive (ar) language modeling, which estimates the probability distribution of the text corpus, is widely used for sequence modeling and sequence-to-sequence (seq2seq) learning @xcite . recently, it also becomes one of the successful self-supervised objectives for large-scale pre-training as used in gpt- * work is done during internship at microsoft research asia. however, as discussed in previous works @xcite @xcite , ar-based models may prefer to focus on the latest tokens rather than capture longterm dependencies for the next token prediction. the reasons are as follows: (a) local correlations such as bigram combination are usually stronger than long-term dependencies. (b) teacher forcing, where the model focus on one-step-ahead prediction for each time step, has no explicit bias toward future token planning and modeling. as a result, the model may learn a bias for language modeling; that is, the local token combinations' modeling is overfitting, but the global coherence and long-term dependency are underfitting @xcite @xcite . during inference, the generations tend to maintain local coherence but lack meaningful global structure @xcite , especially when we use greedy decoding instead of beam search. in this paper, we present a new large-scale pretrained seq2seq model called prophetnet with a novel self-supervised objective future n-gram prediction. in addition to the traditional language model (lm) or seq2seq model that optimizes onestep-ahead prediction, the prophetnet also learns nstep ahead predictionthis future n-gram prediction is served as extra guidance that explicitly encourages the model to plan for future tokens and prevents overfitting on strong local correlations. the hidden states of prophetnet are forced to contain useful information for the next token and further help predict multiple future tokens. there are two goals when designing prophetnet: (a) the model should be able to simultaneously predict the future n-gram at each time step in an efficient way during the training phase, and (b) the model can be easily converted to predict the next token only as original seq2seq model for inference or fine-tuning phase. to achieve that, we extend the two-stream self-attention proposed in xlnet @xcite to n-stream self-attention. prophet-net contains a main stream self-attention, which is the same as the self-attention in the original transformer. besides, we introduce n extra self-attention predicting streams for future n-gram prediction, respectively. during training, the i-th predicting stream attends to the main stream's hidden states to predict the next i-th future token, which guarantees every n continuous tokens in the target sequence are trained to predict at one time step. since the main stream parameters are shared with every predicting stream, we can disable the n-stream selfattention during inference. only the next first token is predicted for each time step, which is same as the original transformer seq2seq model. for experiments, we use the proposed future ngram prediction with the mask based auto-encoder denoising task @xcite which has been proved to be effective for seq2seq pre-training as compared in @xcite for prophetnet pre-training. we use two scale pre-trained datasets to pre-train prophetnet, respectively: the base scale (16gb) dataset as used in bert @xcite , and the large scale (160gb) similar to bart @xcite . the pre-trained prophetnet is further fine-tuned on several nlg tasks. experimental results show that prophetnet has achieved the best performance on cnn/dailymail, gigaword, and squad 1.1 question generation tasks compared to the models using the same base scale pre-training dataset. for the large scale dataset pre-training experiment, prophetnet achieves new state-of-the-art results on cnn/dailymail and gigaword, using only about 1/3 pre-training epochs of bart and about 1/5 pre-training corpus of t5 @xcite and pegasus @xcite . ## prophetnet we propose a new seq2seq pre-training model called prophetnet, which is based on transformer @xcite encoder-decoder ar-chitecture. compared to the original transformer seq2seq model, prophetnet introduces three modifications: (a) the novel self-supervised objective called future n-gram prediction as described in § 2.2. (b) the n-stream self-attention mechanism as described in § 2.3. (c) the mask based autoencoder denoising task for seq2seq pre-training as described in § 2.4. figure 1 shows the architecture of prophetnet. before we describe our model in detail, we first introduce the notations and sequence-to-sequence learning. ## experiments and results in this section, we describe the experimental details and results. we first describe the details of prophet-net pre-training in § 3.1. then we fine-tune the prophetnet on two downstream nlg tasks, including text summarization as described in § 3.2 and question generation as reported in § 3.3. we report the experiment of large-scale pre-training in § 3.4. results without pre-training are compared in § 3.5. we set predicting future gram length into 2 according to the analysis in § 3.6. ## conclusion in this paper, we introduce prophetnet, a sequenceto-sequence pre-training model that learns to predict future n-gram at each time step. prophetnet achieves the best performance on both abstractive summarization and question generation tasks. furthermore, prophetnet achieves new state-of-the-art results on cnn/dailymail and gigaword using only about 1/3 the pre-training epochs of the previous model.