Neural Extractive Summarization with Hierarchical Attentive Heterogeneous Graph Network (2020)

Abstract: Sentence-level extractive text summarization is substantially a node classification task of network mining, adhering to the informative components and concise representations. There are lots of redundant phrases between extracted sentences, but it is difficult to model them exactly by the general supervised methods. Previous sentence encoders, especially BERT, specialize in modeling the relationship between source sentences. While, they have no ability to consider the overlaps of the target selected summary, and there are inherent dependencies among target labels of sentences. In this paper, we propose HAHSum (as shorthand for Hierarchical Attentive Heterogeneous Graph for Text Summarization), which well models different levels of information, including words and sentences, and spotlights redundancy dependencies between sentences. Our approach iteratively refines the sentence representations with redundancy-aware graph and delivers the label dependencies by message passing. Experiments on large scale benchmark corpus (CNN/DM, NYT, and NEWSROOM) demonstrate that HAHSum yields ground-breaking performance and outperforms previous extractive summarizers.

## introduction single document extractive summarization aims to select subset sentences and assemble them as informative and concise summaries. recent advances @xcite @xcite focus on balancing the salience and redundancy of sentences, i.e. selecting the sentences with high semantic similarity to the gold summary and resolving the redundancy between selected sentences. taking table most of the previous approaches utilize autoregressive architecture @xcite @xcite , which just models the unidirectional dependency between sentences, i.e., the state of the current sentence is based on previously sentence labels. these models are trained to predict the current sentence label given the ground truth labels of the previous sentences, while feeding the predicted labels of the previous sentences as input in inference phase. as we all know, the autoregressive paradigm faces error propagation and exposure bias problems @xcite . besides, reinforcement learning is also introduced to consider the semantics of extracted summary @xcite , which combines the maximum-likelihood cross-entropy loss with the rewards from policy gradient to directly optimize the evaluation metric for the summarization task. recently, the popular solution is to build a summarization system with two-stage decoder. these models extract salient sentences and then rewrite @xcite , compress @xcite @xcite , or match @xcite these sentences. previous models generally use top-k strategy as an optimal strategy: for different documents, the number of selected sentences is constant which conflicts with the real world. for example, almost all previous approaches extract three sentences from the source articles (top-3 strategy @xcite @xcite ), although 40% documents in cnn/dm contain more or less than 3-sentences oracle summary. that's because these approaches are difficult to measure the salience and redundancy simultaneously with error propagation. notably, @xcite introduces the length variable into the decoder and @xcite can choose any number of sentences by match candidate summary in semantic space. in order to address above issues, we construct the source article as a hierarchical heterogeneous graph (hhg) and propose a graph attention net @xcite based model (hahsum) to extract sentences by simultaneously balancing salience and redundancy. in hhg, both words and sentences are constructed as nodes, the relations between them are constructed as different types of edges. this hierarchical graph can be viewed as a two-level graph: word-level and sentencelevel. for word-level graph (word-word), we design an abstract layer to learn the semantic representation of each word. then, we transduce the word-level graph into the sentence-level one, by aggregating each word to its corresponding sentence node. for sentence-level graph (sentencesentence), we design a redundancy layer, which firstly pre-labels each sentence and iteratively updates the label dependencies by propagating redundancy information. the redundancy layer restricts the scale of receptive field for redundancy information, and the information passing is guided by the ground-truth labels of sentences. after obtaining the redundancy-aware sentence representations, we use a classifier to label these sentence-level nodes with a threshold. in this way, the whole framework extracts summary sentences simultaneously instead of autoregressive paradigm, taking away the top-k strategy. the contributions of this paper are as below: 1) we propose a hierarchical attentive heterogeneous graph based model(hahsum) to guide the redundancy information propagating between sentences and learn redundancy-aware sentence representation; 2) our architecture is able to extract flexible quantity of sentences with a threshold, instead of top-k strategy; 3) we evaluate hahsum on three popular benchmarks (cnn/dm, nyt, newsroom) and experimental results show that hahsum outperforms the existing state-of-the-art approaches. our source code will be available on github foot_0 . ## conclusion in this paper, we propose hierarchical attentive heterogeneous graph, aiming to advance text summarization by measuring salience and redundancy simultaneously. our approach model redundancy information by iteratively update the sentence information with message passing in redundancy-aware graph. as a result, hahsum produces more focused summaries with fewer superfluous and the performance improvements are more pronounced on more extractive datasets.