Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks (2020)

Abstract: Text summarization aims to compress a textual document to a short summary while keeping salient information. Extractive approaches are widely used in text summarization because of their fluency and efficiency. However, most of existing extractive models hardly capture inter-sentence relationships, particularly in long documents. They also often ignore the effect of topical information on capturing important contents. To address these issues, this paper proposes a graph neural network (GNN)-based extractive summarization model, enabling to capture inter-sentence relationships efficiently via graph-structured document representation. Moreover, our model integrates a joint neural topic model (NTM) to discover latent topics, which can provide document-level features for sentence selection. The experimental results demonstrate that our model not only substantially achieves state-of-the-art results on CNN/DM and NYT datasets but also considerably outperforms existing approaches on scientific paper datasets consisting of much longer documents, indicating its better robustness in document genres and lengths. Further discussions show that topical information can help the model preselect salient contents from an entire document, which interprets its effectiveness in long document summarization.

## introduction text summarization is an important task in natural language processing, which can help people rapidly acquire important information from a large sum of documents. previous summarization approaches can be mainly classified into two categories, which are abstractive and extractive. neural-based abstractive models usually use a seq2seq framework @xcite to generate a word-by-word summary after encoding a full document. by contrast, extractive models directly select important sentences from the original document and then aggregate them into a summary. abstractive models are generally more flexible but may produce disfluent or ungrammatical summary texts @xcite , whereas extractive models have advantages in factuality and efficiency @xcite . despite their success, modeling long-range inter-sentence relationships for summarization remains a challenge @xcite . hierarchical networks are usually applied to tackle this problem by modeling a document as a sequence of sequences @xcite . however, empirical observations @xcite showed that the use of such a paradigm to model intersentence relationships does not provide much performance gain for summarization. hierarchical approaches are also slow to train and tend to overfit @xcite . most recently, graph neural networks (gnns) are widely explored to model cross-sentence relationships for summarization task. the critical step of this framework is to build an effective document graph. several studies @xcite built document graphs based on discourse analysis. however, this approach depends on external tools and may lead to other problems, such as semantically fragmented output @xcite . @xcite built a word-sentence document graph based on word appearance, but such statistical graph-building approach hardly captures semantic-level relationships. therefore, how to model a document as a graph for summarization effectively remains an open question. another critical point of summarization is modeling global information, which plays a key role in sentence selection @xcite . pre-trained language models can considerably boost the performance of summarization @xcite since they effectively capture context features. however, they are poor at modeling document-level information, particularly for long documents, because most of them are designed for sentences or a short paragraph @xcite . to tackle the abovementioned weaknesses, this paper proposes a novel graph-based extractive summarization model. first, we encode an entire document with a pre-trained bert @xcite to learn contextual sentence representations, and discover latent topics with a joint neural topic model (ntm; @xcite . second, we build a heterogeneous document graph consisting of sentence and topic nodes, and simultaneously update their representations with a modified graph attention network (gat; @xcite . third, the representations of sentence nodes are extracted to compute the final labels. intuitively, our topic-sentence document graph has the following advantages: 1) during the graph propagation, sentence representations can be enriched by topical information, which can be considered as a kind of document-level feature and help our model distil important contents from an entire document. 2) topic nodes can act as intermediary to bridge longdistance sentences; hence, our model can efficiently capture inter-sentence relationships. we evaluate our model on four standard datasets, including news articles and scientific papers. the experimental results show its effectiveness and superiority. to summarize, our contributions are threefold. ## model this section describes our model, namely, topic-aware graph neural network for document summarization (topic-graphsum). figure 1 presents the overview architecture. given an arbitrary document 𝐷𝐷 = {𝑠𝑠 1 , 𝑠𝑠 2 , … , 𝑠𝑠 𝑁𝑁 } that consists of 𝑁𝑁 sentences, the objective of our model is to learn a sequence of binary labels {𝑦𝑦 1 , 𝑦𝑦 2 , … , 𝑦𝑦 𝑁𝑁 }, where 𝑦𝑦 𝑖𝑖 ϵ{0, 1} represents whether the 𝑖𝑖-th sentence should be included in summary. our model generally consists of three parts, which are the 1) document encoder, 2) neural topic model, and 3) graph attention layer. given the input document, the document encoder learns contextual representations of each sentence with a pre-trained bert. the ntm aims to learn the document topic distribution and a group of topic representations. the graph attention layer builds a heterogeneous document graph with topics and sentences and then simultaneously update their node representations. after graph encoding, sentence representations are further combined with topics and then sent to a sentence classifier to compute the final labels. we elucidate each part below. ## result and analysis this section reports our experimental results. we evaluate our model on two criteria: 1) whether it can achieve state-of-the-art results? 2) what benefits does the latent topic contribute to summarization? to this end, we first compare our model with state-of-the-art approaches on two widely used benchmark datasets cnn/dm and nyt. then, we evaluate our model on two scientific paper datasets to verify whether discovering latent topics can help summarize long documents. lastly, we present ablation and case studies for further analysis. ## conclusion and future work in this paper, we systematically explore the effects of latent topics for document summarization, and propose a novel graph-based extractive summarization model, which allows joint learning of latent topics and leverages them to enrich sentence representations via a heterogeneous graph neural network. the experimental results on four well-studied datasets demonstrate that our model not only achieves results on par with state-of-the-art summarization models on news article datasets but also significantly outperforms existing approaches on scientific paper datasets, indicating its strong robustness in various document genres and lengths. further explorations on incorporating more types of semantic units (e.g. keywords and entities) into document graph for enhancing the performance of summarization will be addressed in our future work.