Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers (2020)

Abstract: Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. In this work, we find that transformer attentions can be used to rank sentences for unsupervised extractive summarization. Specifically, we first pre-train a hierarchical transformer model using unlabeled documents only. Then we propose a method to rank sentences using sentence-level self-attentions and pre-training objectives. Experiments on CNN/DailyMail and New York Times datasets show our model achieves state-of-the-art performance on unsupervised summarization. We also find in experiments that our model is less dependent on sentence positions. When using a linear combination of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results.

## introduction document summarization is the task of transforming a long document into its shorter version while still retaining its important content. researchers have explored many paradigms for summarization, while the most popular ones are extractive summarization and abstractive summarization @xcite . as their names suggest, extractive summarization generates summiries by extracting text from original documents, and abstractive summarization rewrites documents by paraphrasing or deleting some words or phrases. most summarization models require labeled data, where documents are paired with human written summaries. unfortunately, human labeling for summarization task is expensive and therefore high quality large scale labeled summarization datasets are rear @xcite compared to growing web documents created everyday. it is also not possible to create summaries for documents in all text domains and styles. in this paper, we focus on unsupervised summarization, where we only need unlabeled documents during training. many attempts for unsupervised summarization are extractive @xcite @xcite @xcite @xcite @xcite . the core problem is to identify salient sentences in a document. the most popular approaches among these work rank sentences in the document using graph based algorithms, where each node is a sentence and weights of edges are measured by sentence similarities. then a graph ranking method is employed to estimate sentence importance. for example, tex-trank @xcite utilizes word co-occurrence statistics to compute similarity and then employs pagerank @xcite to rank sentences. sentence similarities in @xcite are measured with bert @xcite and sentences are sorted w.r.t. their centralities in a directed graph. recently, there has been increasing interest in developing unsupervised abstractive summarization models @xcite @xcite . these models are mostly based on sequence to sequence learning @xcite and sequential denoising auto-encoding @xcite . unfortunately, there is no guarantee that summaries produced by these models are grammatical and consistent with facts described original documents. @xcite propose an unsupervised method to pre-train a hierarchical transformer model (i.e., hibert) for document modeling. the hierarchical transformer has a token-level transformer to learn sentence representations and a sentence-level transformer to learn interactions between sentences with self-attention. in @xcite , hibert is applied to supervised extractive summarization. however, we believe that after pre-training hibert on large scale unlabeled data, the self-attention scores in the sentence-level transformer becomes meaningful for estimating the importance of sentences. intuitively, if many sentences in a document attend to one particular sentence with high attention scores, then this sentence should be important. in this paper, we find that (sentence-level) transformer attentions (in a hierarchical transformer) can be used to rank sentences for unsupervised extractive summarization, while previous work mostly leverage graph based (or rule based) methods and sentence similarities computed with off-the-shelf sentence embeddings. specifically, we first introduce two pretraining tasks for hierarchical transformers (i.e., extended hibert) to obtain sentence-level selfattentions using unlabled documents only. then, we design a method to rank sentences by using sentence-level self-attentions and pre-training objectives. experiments on cnn/dailymail and new york times datasets show our model achieves stateof-the-art performance on unsupervised summarization. we also find in experiments that our model is less dependent on sentence positions. when using a linear combination of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results. our code and models are available at https://github.com/xssstory/stas . ## related work in this section, we introduce work on supervised summarization, unsupervised summarization and pre-training. supervised summarization most summarization models require supervision from labeled datasets, where documents are paired with human written summaries. as mentioned earlier, extractive summarization aims to extract important sentences from documents and it is usually viewed as a (sentence) ranking problem by using scores from classifiers @xcite or sequential labeling models @xcite . summarization performance of this class of methods are greatly improved, when human engineered features @xcite @xcite are replaced with convolutional neural networks (cnn) and long shortterm memory networks (lstm) @xcite @xcite . abstractive summarization on the other hand can generate new words or phrases and are mostly based on sequence to sequence (seq2seq) learning @xcite . to better fit in the summarization task, the original seq2seq model is extended with copy mechanism @xcite , coverage model @xcite , reinforcement learning @xcite as well as bottom-up attention @xcite . recently, pretrained transformers @xcite achieve tremendous success in many nlp tasks @xcite . pre-trained methods customized for both extractive @xcite and abstractive @xcite summarization again advance the state-of-the-art in supervised summarization. our model also leverages pre-trained methods and models, but it is unsupervised. unsupervised summarization compared to supervised models, unsupervised models only need unlabeled documents during training. most unsupervised extractive models are graph based @xcite @xcite @xcite @xcite @xcite . for example, textrank @xcite treats sentences in a document as nodes in an undirected graph, and edge weights are measured with cooccurrence based similarities between sentences. then pagerank @xcite is employed to determine the final ranking scores for sentences. @xcite builds directed graph by utilizing bert @xcite to compute sentence similarities. the importance score of a sentence is the weighted sum of all its out edges, where weights for edges between the current sentence and preceding sentences are negative. thus, leading sentences tend to obtain high scores. unlike @xcite , sentence positions are not explicitly modeled in our model and therefore our model is less dependent on sentence positions (as shown in experiments). there are also an interesting line of work on unsupervised abstractive summarization. yang et al. pre-training pre-training methods in nlp learn to encode text by leveraging unlabeled text. early work mostly concentrate on pre-training word embeddings @xcite @xcite . later, sentence encoder can also be pre-trained with language model (or masked language model) objectives @xcite @xcite . @xcite propose a method to pre-train a hierarchical transformer encoder (document encoder) by predicting masked sentences in a document for supervised summarization, while we focus on unsupervised summarization. in our method, we also propose a new task (sentence shuffling) for pre-training hierarchical transformer encoders. @xcite propose a contrastive pre-training objective to predict relative distances of surrounding sentences to the anchor sentence, while our sentence shuffling task predicts original positions of sentences from a shuffled docuemt. besides, pre-training methods mentioned above focus on learning good word, sentence or document representations for downstream tasks, while our method focuses on learning sentence level attention distributions (i.e., sentence associations), which is shown in our experiments to be very helpful for unsupervised summarization. ## model in this section, we describe our unsupervised summarization model stas (as shorthand for sentencelevel transformer based attentive summarization). we first introduce how documents are encoded in our model. then we present methods to pre-trained our document encoder. finally we apply the pretrained encoder to unsupervised summarization. ## experiments in this section we assess the performance of stas on the document summarization task. we firstly introduce datasets we used and then give our implementation details. finally we compare our method against previous methods.