Topic-Guided Abstractive Multi-Document Summarization (2021)

Abstract: A critical point of multi-document summarization (MDS) is to learn the relations among various documents. In this paper, we propose a novel abstractive MDS model, in which we represent multiple documents as a heterogeneous graph, taking semantic nodes of different granularities into account, and then apply a graph-to-sequence framework to generate summaries. Moreover, we employ a neural topic model to jointly discover latent topics that can act as cross-document semantic units to bridge different documents and provide global information to guide the summary generation. Since topic extraction can be viewed as a special type of summarization that “summarizes” texts into a more abstract format, i.e., a topic distribution, we adopt a multi-task learning strategy to jointly train the topic and summarization module, allowing the promotion of each other. Experimental results on the Multi-News dataset demonstrate that our model outperforms previous state-of-the-art MDS models on both Rouge scores and human evaluation, meanwhile learns high-quality topics.

## introduction multi-document summarization (mds) is the task to create a fluent and concise summary for a collection of thematically related documents. compared to single document summarization, it requires the ability to incorporate the perspective from multiple sources and therefore is arguably more challenging @xcite . broadly, existing studies can be classified into two categories: extractive and abstractive. extractive approaches directly select important sentences from the input documents, which is usually regarded as a sentence labeling @xcite @xcite or sentence ranking task @xcite . by contrast, abstractive models typically use the natural language generation technology to produce a word-by-word summary. in general, extractive methods are more efficient and can avoid grammatical errors @xcite , while abstractive methods are more flexible and human-like because they can generate absent words @xcite . recently, with the development of representation learning for nlp @xcite and large-scale datasets @xcite , some studies have achieved promising results on abstractive mds @xcite . nevertheless, we found there are two limitations that have not been addressed by previous studies. first, some works simply concatenate multiple documents into a flat sequence and then apply single-document summarization approaches @xcite . however, this paradigm fails to consider the hierarchical document structures, which plays a key role in mds task @xcite . also, the concatenation operation inevitably produces a lengthy sequence, and encoding long texts for summarization is a challenge @xcite . second, when dealing with multiple documents, a critical point is to learn the cross-document relations. some studies address this problem by mining the co-occurrence words or entities @xcite , which can hardly capture implicit associations due to the diverse language expressions. some other studies @xcite first generate low-dimensional vectors in sentence-or paragraph-level and then build interaction based on these highly compressed representations. these methods inevitably result in the loss of large amounts of fine-grained interaction features and would damage the interpretability of models. therefore, how to learn the relation across documents effectively remains an open question. to shed lights on these missing points, this paper proposes a novel abstractive mds model that marries topic modeling into abstractive summary generation. the motivation is that both tasks aim to distil salient information from massive text and therefore could provide complementary features for each other. concretely, we jointly optimize a neural topic model (ntm) @xcite that learns topic distribution of source documents and corpus-level topic representations, and an abstractive summarizer that incorporates latent topics to summary generation process. in the encoding process, we represent multiple documents as a heterogeneous graph consisting of word, topic, and document nodes and encode it with a graph neural network to capture the interactions among different semantic units. in the decoding process, we devise a topic-aware decoder that leverages learned topics to guide the summary generation. we train the two modules with a multitask learning framework, where an inconsistency loss is applied to penalize the difference between the topic distribution of source documents and that of generated summaries. it encourages the summarizer to generate a summary that is thematically consistent with its source documents and also helps the two modules learn from each other. in this manner, our model is learned such that better topics can yield better summaries and vice versa. we conduct throughout experiments on the recently released multi-news dataset @xcite . the results demonstrate the effectiveness and superiority of our model. to sum up, the contributions of this paper are threefolds: datasets for multi-document summarization recently, @xcite released the first largescale news dataset for mds. each article is collected from real-life scenarios and the golden summaries are written by human, which ensures the data quality. prior to them, some studies tried to construct dataset in automatic manners. for example, @xcite and @xcite built datasets based on wikipedia pages, regarding the first section as the summary and others as different documents. however, modeling the relations among different documents is a different task from modeling that of paragraphs from a same document. therefore, the generalization ability of models built on such data could be questionable. for this reason, we do not consider such auto-constructed datasets but focus on the multi-news dataset curated by human. topic modeling for text summarization topic model is widely used for document modeling. nevertheless, few studies have applied it in summarization task. previous studies regarded topical distributions as additional features to enrich word or sentence representations @xcite @xcite . however, these methods use a pipeline process where topic extraction and summary generation are separately performed. in comparison, we adopt a multi-task learning strategy so that the two tasks can learn complementary features from each other. recently, @xcite has applied ntm to extractive summarization. though inspired by it, the motivation and proposed method of this study differ from it by a large margin. @xcite use latent topics to preselect salient sentences, while we use them to capture cross-document relations for abstractive mds. besides, @xcite solely explores the effect of topic modeling on summarization, while we systematically explore their interplay. ## model this section describes our model, named as topic-guided multi-document summarization (tg-multisum). the overall architecture is presented in figure 1 . given a set of documents {d 1 , d 2 , ..., d n }, the goal of our model is to generate a word sequence @xmath0, y 2 , ..., y s } as the summary. our model consists of three major components: 1) the neural topic model aims to learn the topical information of source documents; 2) the document graph encoder builds the interaction among different documents and various semantic units. 3) the topic-aware decoder generates summary words based on the learned node representations. the entire model is trained in an end-to-end manner. we explain each part below. ## conclusion and future work this study proposes a novel abstractive mds model that integrates a joint ntm to discover latent topics. experimental results demonstrate that our model not only achieves the-state-of-the art results on summarization but also produce high-quality topics. further discussions show that topic inference and summary generation can promote each other. in the future, we will explore how to apply latent topics in controllable summarization.