Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters (2021)

Abstract: This paper presents an efficient graph-enhanced approach to multi-document summarization (MDS) with an encoder-decoder Transformer model. This model is based on recent advances in pre-training both encoder and decoder on very large text data (Lewis et al., 2019), and it incorporates an efficient encoding mechanism (Beltagy et al., 2020) that avoids the quadratic memory growth typical for traditional Transformers. We show that this powerful combination not only scales to large input documents commonly found when summarizing news clusters; it also enables us to process additional input in the form of auxiliary graph representations, which we derive from the multi-document clusters. We present a mechanism to incorporate such graph information into the encoder-decoder model that was pre-trained on text only. Our approach leads to significant improvements on the Multi-News dataset, overall leading to an average 1.8 ROUGE score improvement over previous work (Li et al., 2020). We also show improvements in a transfer-only setup on the DUC-2004 dataset. The graph encodings lead to summaries that are more abstractive. Human evaluation shows that they are also more informative and factually more consistent with their input documents.

## introduction abstractive multi-document summarization (mds), the task of writing a consolidated summary of the main information from multiple documents, has seen advancements with the introduction of large-scale datasets and powerful transformer-based models @xcite @xcite . however, some of the key challenges of mds include lack of proper inter-document context-aware information, improper logical flow of information, and need 1 all our code publicly available at: https://github. com/amazon-research/bartgraphsumm . decoder cluster texts truncated concatenated @xmath0 ## related work researchers have been interested in automatically summarizing multiple documents since the late 1990s. first works @xcite cited the gaining popularity of the world wide web (www) as a motivation for the task. they modeled multi-document collections as graph structures -perhaps influenced by the link structure of the www itself. @xcite summarized pairs of documents by building a graph representation of each and performing graph matching to find salient regions across both documents. radev and mckeown (1998) summarized multiple documents by mapping them to abstract template representations, then generating text from the templates. in the early 2000s, datasets from the document understanding conference (duc), which included human-written summaries for multi-document clusters, sparked increased research interest. in lexrank, @xcite extracted the most salient sentences from a multi-document cluster by constructing a graph representing pairwise sentence similarities and running a pagerank algorithm on the graph. subsequent approaches followed the same paradigm while improving diversity of the extracted sentences @xcite or adding document-level information into the graph @xcite . @xcite incorporated dependency graph features into their sentence relation graphs. @xcite built graphs over sets of terms, rather than sentences. @xcite built a graph over event mentions and their relationships, in order to summarize news events using sentence extraction techniques. @xcite and @xcite leveraged amr formalism to convert source text into amr graphs and then generate a summary using these graphs. more recently, the introduction of larger datasets for mds has enabled researchers to train neural models for multi-document summarization. @xcite introduced a large-scale dataset for mds called wikisum, based on wikipedia articles. @xcite introduced a hierarchical transformer model to better encode global and local aspects in multiple documents and showed improvements on wikisum. @xcite introduced an mds dataset of human-written abstracts from the newser.com website, along with the source articles that are cited from these abstracts. further, they also proposed a hierarchical neural model for mds with an additional maximal marginal relevance (mmr) module that calculates sentence ranking scores based on relevancy and redundancy. @xcite further showed the usefulness of pre-trained language models to improve the performance on mds. however, this approach lacks a pre-trained decoder, and it also limits the document length that can be encoded by the pre-trained language models. in contrast, our work utilizes the pre-trained seq2seq bart @xcite model to improve the performance on mds. we have also incorporated the longformerbased attention mechanism @xcite into bart model to encode long documents. to encode graphs into an mds neural model, @xcite constructed a semantic graph representing key phrases and entities from the documents, as well as their expressed relationships; they used linearized forms of these graphs as inputs to their transformer model. in contrast, we use dual encoders for encoding both documents text and linearized graph text information. recently, @xcite constructed a similarity graph, topic graph, and discourse graph between input documents and encoded this information directly, rather than in linearized form, into a transformer. in our work, we build semantic graphs at the sentence level and create a consolidated graph representation by efficiently removing less useful information. ## models in this section, we first discuss our baseline mds model utilizing the pre-trained bart sequence-tosequence model @xcite . next, we integrate a longformer approach @xcite into the bart model for encoding long documents. finally, we discuss our integration of graph encodings into the bart model. ## experimental setup multi-news dataset. the multi-news dataset @xcite consists of english news articles and the corresponding summaries written by professionals on the newser.com website. the articles in this dataset are curated from a diverse set of news sources (over 1, 500 sites). in this work, we use the same splits provided by @xcite , i.e., 44, 972/5, 622/5, 622 examples for training/validation/test, respectively. following @xcite , we truncate n documents to a total length of l tokens such that 4 we set @xmath1. we choose l/n tokens from each document and concatenate the truncated documents as input. ## conclusion we presented an efficient graph-enhanced approach to mds that achieves state-of-the-art results on the multi-news dataset using the pre-trained encoderdecoder transformer model along with an efficient encoding mechanism. we also show improvements in a transfer-only setup on the duc-2004 dataset. the graph encodings lead to summaries that are more abstractive. human evaluation shows that they are also more informative and factually more consistent with their input documents. finally, we present extensive ablations to better understand the usefulness of our method.