Leveraging Graph to Improve Abstractive Multi-Document Summarization (2020)

Abstract: Graphs that capture relations between textual units have great benefits for detecting salient information from multiple documents and generating overall coherent summaries. In this paper, we develop a neural abstractive multi-document summarization (MDS) model which can leverage well-known graph representations of documents such as similarity graph and discourse graph, to more effectively process multiple input documents and produce abstractive summaries. Our model utilizes graphs to encode documents in order to capture cross-document relations, which is crucial to summarizing long documents. Our model can also take advantage of graphs to guide the summary generation process, which is beneficial for generating coherent and concise summaries. Furthermore, pre-trained language models can be easily combined with our model, which further improve the summarization performance significantly. Empirical results on the WikiSum and MultiNews dataset show that the proposed architecture brings substantial improvements over several strong baselines.

## introduction multi-document summarization (mds) brings great challenges to the widely used sequence-tosequence (seq2seq) neural architecture as it requires effective representation of multiple input documents and content organization of long summaries. for mds, different documents may contain the same content, include additional information, and present complementary or contradictory information @xcite . so different from single document summarization (sds), cross-document links are very important in extracting salient information, detecting redundancy and generating overall coherent summaries for mds. graphs that capture relations between textual units have great benefits to mds, which can help generate more informative, concise and coherent summaries from multiple documents. moreover, graphs can be easily constructed by representing text spans (e.g. sentences, paragraphs etc.) as graph nodes and the semantic links between them as edges. graph representations of documents such as similarity graph based on lexical similarities @xcite and discourse graph based on discourse relations @xcite , have been widely used in traditional graph-based extractive mds models. however, they are not well studied by most abstractive approaches, especially the end-to-end neural approaches. few work has studied the effectiveness of explicit graph representations on neural abstractive mds. in this paper, we develop a neural abstractive mds model which can leverage explicit graph representations of documents to more effectively process multiple input documents and distill abstractive summaries. our model augments the end-toend neural architecture with the ability to incorporate well-established graphs into both the document representation and summary generation processes. specifically, a graph-informed attention mechanism is developed to incorporate graphs into the document encoding process, which enables our model to capture richer cross-document relations. furthermore, graphs are utilized to guide the summary generation process via a hierarchical graph attention mechanism, which takes advantage of the explicit graph structure to help organize the summary content. benefiting from the graph modeling, our model can extract salient information from long documents and generate coherent summaries more effectively. we experiment with three types of graph representations, including similarity graph, topic graph and discourse graph, which all significantly improve the mds performance. additionally, our model is complementary to most pre-trained language models (lms), like bert @xcite , roberta @xcite and xlnet @xcite . they can be easily combined with our model to process much longer inputs. the combined model adopts the advantages of both our graph model and pre-trained lms. our experimental results show that our graph model significantly improves the performance of pre-trained lms on mds. the contributions of our paper are as follows: ## model description in order to process long source documents more effectively, we follow @xcite in splitting source documents into multiple paragraphs by line-breaks. then the graph representation of documents is constructed over paragraphs. for example, a similarity graph can be built based on cosine similarities between tf-idf representations of paragraphs. let g denotes a graph representation matrix of the input documents, where g[i][j] indicates the relation weights between paragraph p i and p j . formally, the task is to generate the summary s of the document collection given l input paragraphs p 1 , . . . , p l and their graph representation g. our model is illustrated in figure 1 , which follows the encoder-decoder architecture @xcite . the encoder is composed of several token-level transformer encoding layers and paragraph-level graph encoding layers which can be stacked freely. the transformer encoding layer follows the transformer architecture introduced in @xcite , encoding contextual information for tokens within each paragraph. the graph encoding layer extends the transformer architecture with a graph attention mechanism to incorporate explicit graph representations into the encoding process. similarly, the decoder is composed of a stack of graph decoding layers. they extend the transformer with a hierarchical graph attention mechanism to utilize explicit graph structure to guide the summary decoding process. in the following, we will focus on the graph encoding layer and graph decoding layer of our model. ## conclusion in this paper we explore the importance of graph representations in mds and propose to leverage graphs to improve the performance of neural abstractive mds. our proposed model is able to incorporate explicit graph representations into the document encoding process to capture richer relations within long inputs, and utilize explicit graph structure to guide the summary decoding process to generate more informative, fluent and concise summaries. we also propose an effective method to combine our model with pre-trained lms, which further improves the performance of mds significantly. experimental results show that our model outperforms several strong baselines by a wide margin. in the future we would like to explore other more informative graph representations such as knowledge graphs, and apply them to further improve the summary quality.