Proposition-Level Clustering for Multi-Document Summarization (2022)

Abstract: Text clustering methods were traditionally incorporated into multi-document summarization (MDS) as a means for coping with considerable information repetition. Particularly, clusters were leveraged to indicate information saliency as well as to avoid redundancy. Such prior methods focused on clustering sentences, even though closely related sentences usually contain also non-aligned parts. In this work, we revisit the clustering approach, grouping together sub-sentential propositions, aiming at more precise information alignment. Specifically, our method detects salient propositions, clusters them into paraphrastic clusters, and generates a representative sentence for each cluster via text fusion. Our summarization method improves over the previous state-of-the-art MDS method in the DUC 2004 and TAC 2011 datasets, both in automatic ROUGE scores and human preference.

## introduction common information needs are most often satisfied by multiple texts rather than by a single one. accordingly, there is a rising interest in multi-document summarization (mds) -generating a summary for a set of topically-related documents. inherently, mds needs to address, either explicitly or implicitly, several subtasks embedded in this summarization setting. these include salience detection, redundancy removal, and text generation. while all these subtasks are embedded in single-document summarization (sds) as well, the challenges are much greater in the multi-document setting, where information is heterogeneous and dispersed, while exhibiting substantial redundancy across linguistically divergent utterances. figure 1 : an example of a cluster of propositions, shown within their source sentence context, from tac 2011 (topic d1103). clustering these as sentences would yield noisy unaligned information, however grouping together only the marked propositions keeps information alignment clean. the first sentence is illustratively divided into propositions, where only one of them is aligned to those in the other sentences. in such an approach, the goal is to cluster redundant paraphrastic pieces of information across the texts, which roughly convey the same meaning. repetition of information across texts, as captured by paraphrastic clustering, typically indicates its importance, and can be leveraged for salience detection. moreover, representing a paraphrastic cluster may facilitate generating a corresponding summary that eliminates repetitions while fusing together complementary details within the cluster. traditionally, clustering-based approaches were widely used for summarization, mostly in extractive and unsupervised settings @xcite @xcite . notably, most of these works generated sentence-based clusters, which tend to be noisy since a sentence typically consists of several units of information that only partially overlap with other cluster sentences. as a result, such clusters often capture topically related sentences rather than paraphrases. figure 1 exemplifies such a noisy cluster, which does contain paraphrastic propositions (marked in blue) within their full sentences (marked in black). another line of research in summarization coped with such noisy sentence-based setting, and looked into the use of sub-sentential units for summarization, e.g., @xcite summarizes with elementary discourse units (edus), while @xcite endorse using open information extraction (ope-nie) -based propositions @xcite for summarization. in this paper, we revisit and combine the clustering-based approaches along with subsentential setting, two research lines that were explored only individually and rather scarcely in recent years. specifically, we apply clustering-based summarization at the more fine-grained propositional level, which avoids grouping non-aligned texts, yielding accurate paraphrastic clusters. these clusters also provide better control over the generated summary sentences -as the generation component is only required to fuse similar propositions. our model ( §3) leverages gold reference summaries to derive training datasets for several summarization sub-tasks. first, salient document propositions were extracted, to train a salience model, by greedily maximizing alignment with the reference summaries. then, an available proposition similarity model, trained from summarysource alignments @xcite , provides the basis for agglomerative clustering @xcite . finally, we created training data for a bart-based model for sentence fusion @xcite by aligning reference summary propositions with source proposition clusters. similar to many other works, we leave inter-sentence coherence and sentence planning and ordering outside the scope of the current paper. accordingly, our process produces a bullet-style summary of individual concise and coherent sentences. overall, our experiments ( §4) show that this multi-step model outperforms strong recent end-toend solutions, which do not include explicit modeling of propositions and information redundancy. to the best of our knowledge, our approach achieves state-of-the-art results in our setting on the duc 2004 and tac 2011 datasets, with an improvement of more than 1.5 and 4 rouge-1 f1 points respectively, over the previous best approach. finally, we also suggest ( §5) that clustering-based methods provide "explanations", or supporting evidence, for each generated sentence, in the form of the source cluster propositions (see an example in ## background and related work clustering-based summarization. clusteringbased summarization approaches typically involve salience detection while avoiding redundancy. one such approach clustered topically-related sentences, after which cluster properties were leveraged for rating sentence salience @xcite @xcite . another approach rated sentence salience and clustered sentences simultaneously, iteratively improving the two objectives @xcite @xcite . recently, however, clustering methods have been gradually marginalized out, being replaced by neural techniques. more recently though, some approaches @xcite presented abstractive clustering-based summarization, where topically-related sentences in each cluster are fused together to generate a summary sentence candidate. while most of previous clustering approaches operated at the noisy sentence level, in our work we present more accurate proposition-level clustering that eventually enhances summarization. ## method this section first provides an overview of our method, followed by subsections describing its components. we follow previous clustering-based approaches, where text segments are first clustered into semantically similar groups, exploiting redundancy as a salience signal. then, each group is fused to generate a merged sentence, while avoiding redundancy. as we operate at the propositionlevel, we first extract all propositions from the input documents ( §3.1). then, to facilitate the clustering step, we filter out non-salient propositions using a salience model ( §3.2). next, salient propositions are clustered based on their semantic similarity ( §3.3). the largest clusters, whose information was most repeated, are selected to be included in the summary ( §3.4). finally, each cluster is fused to form a sentence for a bullet-style abstractive summary ( §3.5). in addition, we provide an extractive version where a representative (source) proposition is selected from each cluster (3.6). overall, clustering explicit propositions induces a multi-step process that requires dedicated training data for certain steps. to that end, we derive new training datasets for the salience detection and the fusion models from the original gold summaries. the full pipeline is illustrated in figure 2 , where additional implementation details are in §b in the appendix. ## paraphrastic clusters as summary evidence a unique advantage of a cluster-based summary is that each summary sentence is linked explicitly to a group of propositions from which the sentence was generated, in so providing an "explanation", or support evidence, for the output. these cluster explanations can expand the reader's knowledge and provide complementary facts from the nearby source context regarding the information from the generated sentence. such a feature may be incorporated in interactive summarization systems, as applied in @xcite , where a user can choose to expand on the facts within a sentence of the presented summary. to assess the reliability of such feature, we verified that clusters indeed "explain" their generated sentences. to that end, we conducted a crowdsourced annotation, where a worker marked whether a cluster proposition mentions the main idea of its corresponding generated sentence. each pair was examined by three workers, with the majority vote used for the final decision. on a random selection of 25% of the clusters, we found that, on average, 89% and 84% of a cluster's propositions in duc 2004 and tac 2011 support their corresponding generated sentence, with an average cluster size of 3.4 and 4.8 propositions, respectively. furthermore, given this strong alignment of a cluster to its generated sentence, a cluster facilitates effective verification of faithfulness of its corresponding generated abstractive sentence. since the output sentence is based solely on its cluster propositions, the sentence's correctness can be verified against the "explaining" cluster instead of against the full document set. an example of an unfaithful abstraction is marked in red in