Discourse-Aware Neural Extractive Text Summarization (2020)

Abstract: Recently BERT has been adopted for document encoding in state-of-the-art text summarization models. However, sentence-based extractive models often result in redundant or uninformative phrases in the extracted summaries. Also, long-range dependencies throughout a document are not well captured by BERT, which is pre-trained on sentence pairs instead of documents. To address these issues, we present a discourse-aware neural summarization model - DiscoBert. DiscoBert extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity. To capture the long-range dependencies among discourse units, structural discourse graphs are constructed based on RST trees and coreference mentions, encoded with Graph Convolutional Networks. Experiments show that the proposed model outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.

## introduction neural networks have achieved great success in the task of text summarization @xcite . there are two main lines of research: abstractive and extractive. while the abstractive paradigm @xcite @xcite focuses on generating a summary word-by-word after encoding the full document, the extractive approach @xcite @xcite directly selects sentences from the document to assemble into a summary. the abstractive approach is more flexible and generally produces less redundant summaries, while the extractive approach enjoys better factuality and efficiency @xcite . recently, some hybrid methods have been proposed to take advantage of both, by designing a two-stage pipeline to first select and then rewrite (or compress) candidate sentences @xcite @xcite . compression or rewriting aims to discard uninformative phrases in the selected sentences. however, most of these hybrid systems suffer from the inevitable disconnection between the two stages in the pipeline. meanwhile, modeling long-range context for document summarization remains a challenge @xcite . pre-trained language models (devlin et al., 2019) are designed mostly for sentences or a short paragraph, thus poor at capturing longrange dependencies throughout a document. empirical observations @xcite show that adding standard encoders such as lstm or transformer @xcite on top of bert to model inter-sentential relations does not bring in much performance gain. in this paper, we present discobert, a discourse-aware neural extractive summarization model built upon bert. to perform compression with extraction simultaneously and reduce redundancy across sentences, we take elementary discourse unit (edu), a sub-sentence phrase unit originating from @xcite 2 as the minimal selection unit (instead of sentence) for extractive summarization. figure 1 shows an example of discourse segmentation, with sentences broken down into edus (annotated with brackets). by operating on the discourse unit level, our model can discard redundant details in sub-sentences, therefore retaining additional capacity to include more concepts or events, leading to more concise and informative summaries. furthermore, we finetune the representations of discourse units with the injection of prior knowledge to leverage intra-sentence discourse relations. more specifically, two discourse-oriented graphs are proposed: rst graph g r and coreference graph g c . over these discourse graphs, graph convolutional network (gcn) (kipf and welling, 2017) is imposed to capture long-range interactions among edus. rst graph is constructed from rst parse trees over edus of the document. on the other hand, coreference graph connects entities and their coreference clusters/mentions across the document. the path of coreference navigates the model from the core event to other occurrences of that event, and in parallel explores its interactions with other concepts or events. the main contribution is threefold: (i) we propose a discourse-aware extractive summarization model, discobert, which operates on a subsentential discourse unit level to generate concise and informative summary with low redundancy. (ii) we propose to structurally model 2 we adopt rst as the discourse framework due to the availability of existing tools, the nature of the rst tree structure for compression, and the observations from @xcite . other alternatives includes graph bank @xcite and pdtb @xcite . ## discobert model and today, the pulitzer prize for … south carolina, which has a tiny … 85,000. ## experiments in this section, we present experimental results on two popular news summarization datasets. we compare our proposed model with state-of-the-art baselines and conduct detailed analysis to validate the effectiveness of discobert. ## related work neural extractive summarization neural networks have been widely used in extractive summarization. various decoding approaches, including ranking @xcite , index prediction @xcite and sequential labelling @xcite @xcite , have been applied to content selection. our model uses a similar configuration to encode the document with bert as @xcite did, but we use discourse graph structure and graph encoder to handle the long-range dependency issue. neural compressive summarization text summarization with compression and deletion has been explored in some recent work. xu and durrett (2019 graph-based summarization graph approach has been explored in text summarization over decades. lexrank introduced a stochastic graphbased method for computing relative importance of textual units @xcite . @xcite employed a gcn on the relation graphs with sentence embeddings obtained from rnn. @xcite also proposed graphbased attention in abstractive summarization model. @xcite developed a framework to reason long-distance relationships for text summarization. ## conclusion in this paper, we present discobert, which uses discourse unit as the minimal selection basis to reduce summarization redundancy and leverages two types of discourse graphs as inductive bias to capture long-range dependencies among discourse units. we validate the proposed approach on two popular summarization datasets, and observe consistent improvement over baseline models. for future work, we will explore better graph encoding methods, and apply discourse graphs to other tasks that require long document encoding.