Pre-training for Abstractive Document Summarization by Reinstating Source Text (2020)

Abstract: Abstractive document summarization is usually modeled as a sequence-to-sequence (SEQ2SEQ) learning problem. Unfortunately, training large SEQ2SEQ based summarization models on limited supervised summarization data is challenging. This paper presents three sequence-to-sequence pre-training (in shorthand, STEP) objectives which allow us to pre-train a SEQ2SEQ based abstractive summarization model on unlabeled text. The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document. These objectives include sentence reordering, next sentence generation and masked document generation, which have close relations with the abstractive document summarization task. Experiments on two benchmark summarization datasets (i.e., CNN/DailyMail and New York Times) show that all three objectives can improve performance upon baselines. Compared to models pre-trained on large-scale data (larger than 160GB), our method, with only 19GB text for pre-training, achieves comparable results, which demonstrates its effectiveness.

## introduction automatic document summarization is the task of condensing a document into its shorter form with important content preserved, which requires widecoverage understandings of the document, rather than specific words or phrases. this task can be typically classified into two categories: extractive and abstractive document summarization. extractive summarization @xcite @xcite aims to extract important sentences from the input document and concatenates such extracted sentences as the corresponding output summary. thus, the relative orders of the selected sentences in the summary is the same as their relative orders in the input document. differently, abstractive summarization @xcite @xcite rewrites the source text and generates the corresponding summary which may contain novel words and phrases not featured in the input. the output summary is closely related to the input document. also, summary sentences, paraphrased from the input by the abstractive summarizers, might have a different relative order compared to the source text. in other words, contents of the original document may be reordered in its summary. such a phenomena is defined as content reordering (see section 3.2 for detailed definition). statistically, we observed that around 40% instances of the training split of our summarization dataset have this content reordering phenomena. therefore, it is necessary to design a model that is capable of reordering content. however, as far as we know, relatively rare prior work has studied this for abstractive summarization. abstractive summarization is usually framed as a sequence-to-sequence (seq2seq) learning problem @xcite . in this paper, we adopt the seq2seq transformer @xcite , which has been demonstrated to be the state-of-the-art for seq2seq modeling @xcite . recent studies @xcite @xcite @xcite have proven effectiveness of pretrained seq2seq transformer models on the natural language generation tasks, such as abstractive summarization. based on the above observations, with regard to abstractive summarization, this work proposes three sequence-to-sequence pre-training (in shorthand, step) objectives which can be used to pretrain a seq2seq model on unlabeled text, namely sentence reordering (sr), next sentence generation (nsg), and masked document generation (mdg). all three objectives are designed to reinstate the original source text. sr learns to recover a document with randomly shuffled sentences. given the first segment of a document, nsg generates the next segment of the original document. mdg learns to recover a masked document to its original form. after pre-training a model with our proposed objective(s) on unlabeled documents, we finetune it on supervised summarization datasets (i.e., cnn/dailymail and new york times). experiments show that, even pre-training on documents from the training split of a summarization dataset, our method can improve performance upon a heavily tuned large seq2seq transformer model which already includes a strong pre-trained encoder by a large margin. by involving more data (19gb) for pre-training, the performance is further improved. compared to models pre-trained with much more data (≥160gb), we can still achieve comparable or even higher rouge scores. ## related work extractive summarization this task aims to find the informative sentences in a document as its summary. this task is usually viewed as a sentence ranking problem @xcite using scores from a binary (sequence) classification model, which predicts whether a sentence is in the summary or not. extractive neural models @xcite @xcite employ hierarchical lstms/cnns as the feature learning part of the binary (sequence) classifier, which largely outperform discrete feature based models @xcite @xcite . very recently, the feature learning part was replaced again with pre-trained transformer encoders @xcite that lead to another huge performance gain. however, extractive models have their own limitations. for example, the extracted sentences might be too long and redundant. besides, manually written summaries in their nature are abstractive. therefore, we focus on abstractive summarization in this paper. abstractive summarization this task aims to generate a summary by rewriting a document, which is a seq2seq learning problem. seq2seq attentive lstms @xcite are employed in @xcite that have been extended with copy mechanism @xcite , coverage model @xcite and reinforcement learning @xcite . @xcite used a seq2seq transformer model with only its encoder initialized with a pre-trained transformer encoder (i.e., bert; @xcite . this work proposes to pre-train the decoder together with the encoder and then initialize both the encoder and decoder of a summarization model with the pre-trained transformer model. there is also a line of work that bridges extractive and abstractive models with attention mechanisms @xcite and reinforcement learning @xcite , while our model is simpler. pre-training pre-training methods draw a lot of attentions recently. @xcite and @xcite pre-trained lstm and transformer using language modeling objectives. to leverage the context in both directions, bert @xcite is trained with the masked language modeling and next sentence prediction objectives. spanbert @xcite applied only the masked language modeling objective that masks contiguous random spans, rather than random tokens. xlnet @xcite proposed a permutation language modeling objective that removes the independence assumption of masked tokens in bert. roberta @xcite extends bert with more training data and better training strategies. the above models focus on pre-training an encoder or a decoder, while we propose methods to pre-train a seq2seq model (i.e., the encoder together with the decoder) for abstractive summarization. @xcite (unilm) proposed a unified language model that can be used for both natural language understanding and generation tasks, which is pre-trained using masked, unidirectional and seq2seq language modeling objectives. the encoder and decoder parameters are shared. by contrast, we pre-train a seq2seq transformer with separate parameters for the encoder and decoder. @xcite (mass) proposed a method to pre-train a seq2seq transformer by masking a span of text and then predicting the masked tokens. their pre-training task is similar to our mdg task, but we apply a different masking strategy and predict the original text. @xcite tested their model on sentence-level tasks (e.g., machine translation and sentence compression), while we aim to solve document-level tasks (e.g., abstractive document summarization). @xcite (bart) adopted the combination of text infilling and sentence permutation as a single objective for seq2seq transformer pretraining. differently, we propose three objectives and use them individually. specifically, mdg replaced each selected token with a masked token in the input sequence. raffel et al. (2019) (t5 ) studies different pre-training objectives, model architectures, and unlabeled datasets. prophetnet @xcite predicts the next n tokens simultaneously. @xcite (pegasus) proposed to remove/mask sentences from an input document and learn to generate such removed/masked sentences for pre-training, while nsg predicts the following sentences of the input sequence and mdg masks randomly selected tokens. ## conclusion we proposed three sequence-to-sequence pretraining objectives, including sentence reordering, next sentence generation, and masked document generation. all those objectives have relations with abstractive summarization task and are designed based on reinstating the source text. a seq2seq model for abstractive document summarization can be pre-trained using such objectives and then fine-tuned on the summarization dataset. compared to models pre-training on the even larger corpora (≥160gb), our method, with only 19gb for pre-training, can still achieve comparable and even better performance. in the future, we would like to investigate other objectives to pre-train seq2seq models for abstractive summarization.