Modeling Endorsement for Multi-Document Abstractive Summarization (2021)

Abstract: A crucial difference between single- and multi-document summarization is how salient content manifests itself in the document(s). While such content may appear at the beginning of a single document, essential information is frequently reiterated in a set of documents related to a particular topic, resulting in an endorsement effect that increases information salience. In this paper, we model the cross-document endorsement effect and its utilization in multiple document summarization. Our method generates a synopsis from each document, which serves as an endorser to identify salient content from other documents. Strongly endorsed text segments are used to enrich a neural encoder-decoder model to consolidate them into an abstractive summary. The method has a great potential to learn from fewer examples to identify salient content, which alleviates the need for costly retraining when the set of documents is dynamically adjusted. Through extensive experiments on benchmark multi-document summarization datasets, we demonstrate the effectiveness of our proposed method over strong published baselines. Finally, we shed light on future research directions and discuss broader challenges of this task using a case study.

## introduction multi-document abstractive summarization, i.e. mudas, remains a challenging problem compared to its single-document counterpart @xcite @xcite @xcite . the task poses a substantial challenge to modern neural models: when the set of source documents is concatenated into a flat sequence, it may exceed the maximum length allowed by the gpu memory. there are also fewer datasets available to train mudas models in an end-to-end fashion. recent work tackles this problem by selecting representative sentences from the source documents to reduce the task to singledocument summarization @xcite @xcite . nevertheless, there could be substantial information loss if only representative sentences are used for mudas. it becomes unclear what information is reiterated and salient, resulting in unimportant sentence parts being included in the summary. e.g., when the sentence "world leaders join to pledge @xmath08 billion" or "u.s. sits out," is more salient given the topic of discussion. the neural representations also treat different quantities, e.g., "@xmath15 million," indiscriminately @xcite . consequently, there is an urgent need for summarization systems to acquire fine-grained, segment-level textual salience. without that, a neural abstractive system can miss out on salient details and favor fluency over information accuracy. in this paper, we present a conceptual framework that leverages the endorsement effect to model finegrained segment salience for multi-document summarization. when an analyst reads a document, he retains a synopsis of the key ideas of the document in his mind. the synopsis later serves as an endorser to identify segments in other documents that reiterate the same ideas @xcite . we call the synopsis an "endorser" and the document a "candidate." segments of the candidate documents that are frequently endorsed by synopses suggest high salience and are to be consolidated into an abstractive summary. our synopses are generated from a state-of-the-art summarizer @xcite and a variety of methods are investigated to quantify the level of endorsement from a text synopsis to a document. figure 1 provides an overview of synopsis-document endorsement. our contributions in this paper include: ## related work redundancy is essential in multi-document summarization. without repetition and redundancy, even humans cannot agree on what information is salient and should be included in the summary @xcite . optimizing summaries for frequency-based saliency has attained success prior to the era of deep learning @xcite @xcite . these extractive systems strive to include the most frequently occurring concepts in the summary. however, when it comes to abstractive summarization systems, the frequency of concepts is not fully utilized by modern neural models. recent studies on mudas implicitly estimate frequency using hierarchical encoders / decoders. liu and lapata (2019) encode the documents using hierarchical transformers where cross-document relationships are characterized by attention weights. perez-beltrachini et al. (2019) explore structured convolutional decoders. li et al. ( 2020 ) leverage similarity and discourse graphs to alter the attention mechanism of encoder-decoder models. researchers have also attempted optimization algorithms such as maximal margin relevance and determinantal point processes combined with contextualized representations and reinforcement learning @xcite . despite promising progress, modeling frequency for multidocument summarization remains an open problem, in part because neural summarization models are often pretrained on single documents that contain little or no redundant content @xcite @xcite @xcite . named entities and quantities that represent salient information details are not properly accounted for @xcite ). if we do not explicitly model frequency, abstractive summarizers may fail to adequately recognize such salient details. we are particularly interested in reducing multiple input documents to a single document, then consolidate the content into a succinct abstract @xcite . our method enhances the single document with fine-grained segment salience to offset the lead bias @xcite , which hinders the development of multiple-document summarization. our salience estimates are obtained from a frequency-driven endorsement model. below we present details of the proposed method. ## summarization with endorsement we approach the mudas problem in two stages. first, we obtain fine-grained segment-level endorsement for any candidate document. by excluding unendorsed sentences from consideration, we reduce the set of documents to a single input document. we next present an enhanced abstractive summarization model to consolidate the document into a succinct abstract, analogously to how an editor would consolidate text with emphasis on endorsed segments. this process involves non-trivial design decisions. in this section, we start by presenting the second stage in our approach -the summarization model with endorsement. ## modelling endorsement in this section, we present the first stage in our approach -modelling endorsement -whose outputs are passed to the abstractive summarization model in the second stage. modelling endorsement serves two main purposes. it allows us to identify salient segments of text using a frequency-driven endorsement model, and the level of endorsement guides the summarizer to consolidate salient content. further, it helps us reduce the source input from multiple documents to a single pseudodocument, whereby any unendorsed sentences are removed from consideration. a fragment of text is considered to be endorsed if its information is observed in the endorser. we obtain a set of synopses from the source documents; they are used as endorsers to identify salient segments from a candidate source document. a segment that is endorsed only once indicates its information is considered important by only one source document. frequent endorsement by multiple endorsers suggests the information is reiterated in multiple source documents, and reiteration implies increased salience. any information that is present among multiple sources is likely to be important. thus, our method identifies salient segments considering both within-and cross-document saliency. our approach is in spirit similar to those of building semantic concept graphs for multi-document summarization @xcite @xcite in that frequently reiterated concepts are likely to be captured. however, we do not explicitly construct semantic concept graphs, but focus on modeling synopsis-document endorsement and incorporating it into summary generation, which distinguishes our work from these studies. we investigate two variants to compute segment-level endorsement. ## data we experiment with a large-scale multi-document summarization dataset @xcite whose data are gathered from the wikipedia current events portal (wcep). 3 the dataset contains an archive of important news events happening around 2016-2019. each event is associated with a succinct summary of 30-40 words written by the editor and an average of 1.2 source articles linked from the event page. additional source articles are retrieved from the commoncrawl-news dataset using an event classifier. these articles are published within a window @xmath2://en.wikipedia.org/wiki/portal: current _ events event date. we sample from these additional articles to ensure each event has 10 source articles. all summaries and source articles are in english. the dataset contains 8,158, 1,020 and 1,022 clusters respectively in the train, validation and test splits. our method aims to produce an abstractive summary from a cluster of news articles discussing a given event or topic. to assess the generality of our method, we apply the model trained on wcep to three different test sets, i.e., the test split of wcep and two benchmark multi-document summarization datasets, duc-04 and tac-11. the duc/tac datasets contain 50 and 44 clusters, respectively. they each comprise a set of news events collected over a period of time, and thus are suitable for evaluation of the model's generality in out-of-domain scenarios. duc and tac datasets contain four reference summaries per cluster created by nist evaluators. wcep has a single reference summary per cluster written by editors. the target summary length is 100 words for duc/tac and 40 words for wcep, following the convention of previously published results. endorsement-related statistics for these datasets are presented in synop num seg % endorse @xmath3.9 14.2 100.0 12.6 5.6 duc-04 58 6.1 11.7 100.0 9.7 2.3 tac-11 60 6.7 11.8 100.0 14.5 4.1 hi-map @xcite introduces an endto-end hierarchical attention model to generate abstracts from multi-document inputs. we compare our system to these baselines and report results on wcep, duc-04, and tac-11 datasets foot_2 . sequential vs. reciprocal endorsement. we investigate two endorsement patterns: (a) reciprocal endorsement allows any two documents of the same cluster to endorse each other, and (b) sequential endorsement arranges source documents in chronological order and only later documents are allowed to endorse earlier ones. the endorsement mechanism provides the flexibility needed for many domains to exploit cross-document relationships to generate abstractive summaries. for our variants, the highest-scoring sentences are consolidated to form an input document which, along with the endorsement scores, are passed to our endorsementaware abstractor to be condensed into a summary. endorsement-aware abstractor. we employ bart, a state-of-the-art encoder-decoder model as our base abstractor @xcite . the model has 24 layers in the encoder and decoder, a hidden size of 1024, 16 heads, with a total of 406m parameters. it was fine-tuned on the train split of wcep for an average of two epochs with a batch size of 4. we use the adam optimizer (kingma and ba, 2015) and a learning rate of 3 - foot_3 with warm-up. at inference time, we use a beam size of @xmath4, with a minimum decoding length of 10 and a maximum of 50 tokens. our implementation is based on fairseq 5 and it takes about two hours to train the model on a nvidia v100 32gb gpu card. for the endorsement-aware abstractor, we add two sets of companion heads to the decoder, for a total of 48 attention heads. the τ values for each set of heads are 0/1/2. 12% of the tokens receive level-1 attention (τ = 1), 4% receive level-2 attention (τ = 2). the λ τ values are set to be 0.8, 0.1, and 0.1-this gives more influence to the original attention heads, so the model is not confused by the addition of the new heads that attend to endorsed segments. we use a maximum of 1024 tokens for the input document. synopsis-document endorsement. to enable soft alignment between a synopsis and a candidate document, we use bertscore @xcite with the following hash code: roberta-large_l17_no-@xmath5.3.2(@xmath6.8.0)rescaled. it suggests that the token representations are drawn from the 17th layer of roberta-large. our maximum sum subarray algorithm requires the scores to contain a mix of positive/negative values. thus, we subtract all scores by δ. the δ values are 0.85 and 0.8 for the soft and hard alignment, respectively. these values are tuned on validation data, where a larger δ indicates fewer tokens will be endorsed. we proceed by presenting summarization results on our datasets, including an ablation study to examine the contribution of each part of our method. we also present a case study showcasing the potential of our endorsement method. ## conclusion we presented a novel framework to model asynchronous endorsement between synopses and documents for multi-document abstractive summarization. we introduced an endorsement method to enrich the encoder-decoder model with fine-grained endorsement. our method was evaluated on benchmark multi-document datasets and we discussed challenges and shed light on future research. mike lewis, yinhan liu, naman goyal, marjan ghazvininejad, abdelrahman mohamed, omer levy, veselin stoyanov, and luke zettlemoyer. 2020. bart: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871-7880, online. association for computational linguistics. xintong li, aleksandre maskharashvili, symon jory stevens-guille, and michael white. 2020. leveraging large pretrained models for webnlg 2020. in proceedings of the 3rd international workshop on natural language generation from the semantic web (webnlg+), pages 117-124, dublin, ireland (virtual). association for computational linguistics.