Heterogeneous Graph Neural Networks for Extractive Document Summarization (2020)

Abstract: As a crucial step in extractive document summarization, learning cross-sentence relations has been explored by a plethora of approaches. An intuitive way is to put them in the graph-based neural network, which has a more complex structure for capturing inter-sentence relationships. In this paper, we present a heterogeneous graph-based neural network for extractive summarization (HETERSUMGRAPH), which contains semantic nodes of different granularity levels apart from sentences. These additional nodes act as the intermediary between sentences and enrich the cross-sentence relations. Besides, our graph structure is flexible in natural extension from a single-document setting to multi-document via introducing document nodes. To our knowledge, we are the first one to introduce different types of nodes into graph-based neural networks for extractive document summarization and perform a comprehensive qualitative analysis to investigate their benefits. The code will be released on Github.

## introduction extractive document summarization aims to extract relevant sentences from the original documents and reorganize them as the summary. recent years have seen a resounding success in the use of deep neural networks on this task @xcite @xcite @xcite . these existing models mainly follow the encoder-decoder framework in which each sentence will be encoded by neural components with different forms. to effectively extract the summary-worthy sentences from a document, a core step is to model the cross-sentence relations. most current models capture cross-sentence relations with recurrent neural networks (rnns) @xcite @xcite . however, rnns-based models are usually hard to capture sentence-level long-distance dependency, especially in the case of the long document or multidocuments. one more intuitive way is to model the relations of sentences using the graph structure. nevertheless, it is challenging to find an effective graph structure for summarization. efforts have been made in various ways. early traditional work makes use of inter-sentence cosine similarity to build the connectivity graph like lexrank @xcite and textrank @xcite . recently, some works account for discourse inter-sentential relationships when building summarization graphs, such as the approximate discourse graph (adg) with sentence personalization features @xcite and rhetorical structure theory (rst) graph @xcite . however, they usually rely on external tools and need to take account of the error propagation problem. a more straightforward way is to create a sentence-level fully-connected graph. to some extent, the transformer encoder @xcite used in recent work @xcite ) can be classified into this type, which learns the pairwise interaction between sentences. despite their success, how to construct an effective graph structure for summarization remains an open question. in this paper, we propose a heterogeneous graph network for extractive summarization. instead of solely building graphs on sentence-level nodes, we introduce more semantic units as additional nodes in the graph to enrich the relationships between sentences. these additional nodes act as the intermediary that connects sentences. namely, each additional node can be viewed as a special rela-tionship between sentences containing it. during the massage passing over the heterogeneous graph, these additional nodes will be iteratively updated as well as sentence nodes. although more advanced features can be used (e.g., entities or topics), for simplicity, we use words as the semantic units in this paper. each sentence is connected to its contained words. there are no direct edges for all the sentence pairs and word pairs. the constructed heterogeneous wordsentence graph has the following advantages: (a) different sentences can interact with each other in consideration of the explicit overlapping word information. (b) the word nodes can also aggregate information from sentences and get updated. unlike ours, existing models usually keep the words unchanged as the embedding layer. (c) different granularities of information can be fully used through multiple message passing processes. (d) our heterogeneous graph network is expandable for more types of nodes. for example, we can introduce document nodes for multi-document summarization. we highlight our contributions as follows: ## related work extractive document summarization with the development of neural networks, great progress has been made in extractive document summarization. most of them focus on the encoderdecoder framework and use recurrent neural networks @xcite @xcite or transformer encoders @xcite for the sentential encoding. recently, pre-trained language models are also applied in summarization for contextual word representations @xcite @xcite . another intuitive structure for extractive summarization is the graph, which can better utilize the statistical or linguistic information between sentences. early works focus on document graphs constructed with the content similarity among sentences, like lexrank @xcite and textrank @xcite . some recent works aim to incorporate a relational priori into the encoder by graph neural networks (gnns) @xcite . methodologically, these works only use one type of nodes, which formulate each document as a homogeneous graph. heterogeneous graph for nlp graph neural networks and their associated learning methods (i.e. message passing @xcite , selfattention @xcite ) are originally designed for the homogeneous graph where the whole graph shares the same type of nodes. however, the graph in the real-world application usually comes with multiple types of nodes @xcite , namely the heterogeneous graph. to model these structures, recent works have made preliminary exploration. @xcite introduced a heterogeneous graph neural network to encode documents, entities and candidates together for multihop reading comprehension. @xcite focused on semi-supervised short text classification and constructed a topic-entity heterogeneous neural graph. for summarization, @xcite proposes a heterogeneous graph consisting of topic, word and sentence nodes and uses the markov chain model for the iterative update. @xcite modify textrank for their graph with keywords and sentences and thus put forward heterorank. inspired by the success of the heterogeneous graph-based neural network on other nlp tasks, we introduce it to extractive text summarization to learn a better node representation. ## methodology given a document @xmath0, • • • , s n } with n sentences, we can formulate extractive summarization as a sequence labeling task as @xcite ; word encoder cnn encoder bilstm word word 𝑤 3 𝑤 1 𝑤 2 sentence sentence graph layer 𝑠 1 𝑠 2 sentence selector tf-idf edge feature word node sentence node figure 1: model overview. the framework consists of three major modules: graph initializers, the heterogeneous graph layer and the sentence selector. green circles and blue boxes represent word and sentence nodes respectively. orange solid lines denote the edge feature (tf-idf) between word and sentence nodes and the thicknesses indicate the weight. the representations of sentence nodes will be finally used for summary selection. liu and lapata, 2019b). our goal is to predict a sequence of labels y 1 , • • • , y n (y @xmath1, 1}) for sentences, where y @xmath2. the ground truth labels, which we call oracle, is extracted using the greedy approach introduced by @xcite with the automatic metrics rouge @xcite . generally speaking, our heterogeneous summarization graph consists of two types of nodes: basic semantic nodes (e.g. words, concepts, etc.) as relay nodes and other units of discourse (e.g. phrases, sentences, documents, etc.) as supernodes. each supernode connects with basic nodes contained in it and takes the importance of the relation as their edge feature. thus, high-level discourse nodes can establish relationships between each other via basic nodes. in this paper, we use words as the basic semantic nodes for simplicity. hetersumgraph in section 3.1 is a special case which only contains one type of supernodes (sentences) for classification, while heterdocsumgraph in section 3.5 use two (documents and sentences). based on our framework, other types of supernodes (such as paragraphs) can also be introduced and the only difference lies in the graph structure. ## experiment we evaluate our models both on single-and multidocument summarization tasks. below, we start our experiment with the description of the datasets. nyt50 nyt50 is also a single-document summarization dataset, which was collected from new york times annotated corpus @xcite and preprocessed by @xcite . it contains 110,540 articles with summaries and is split into 100,834 and 9706 for training and test. following @xcite , we use the last 4,000 examples from the training set as validation and filter test examples to 3,452. multi-news the multi-news dataset is a largescale multi-document summarization introduced by @xcite . it contains 56,216 articlessummary pairs and each example consists of 2-10 source documents and a human-written summary. following their experimental settings, we split the dataset into 44,972/5,622/5,622 for training, validation and test examples and truncate input articles to 500 tokens. ## conclusion in this paper, we propose a heterogeneous graphbased neural network for extractive summarization. the introduction of more fine-grained semantic units in the summarization graph helps our model to build more complex relationships between sentences . it is also convenient to adapt our singledocument graph to multi-document with document nodes. furthermore, our models have achieved the best results on cnn/dailymail compared with non-bert-based models, and we will take the pretrained language models into account for better encoding representations of nodes in the future.