Multi-Document Summarization with Centroid-Based Pretraining (2023)

Abstract: In Multi-Document Summarization (MDS), the input can be modeled as a set of documents, and the output is its summary. In this paper, we focus on pretraining objectives for MDS. Specifically, we introduce a novel pretraining objective, which involves selecting the ROUGE-based centroid of each document cluster as a proxy for its summary. Our objective thus does not require human written summaries and can be utilized for pretraining on a dataset consisting solely of document sets. Through zero-shot, few-shot, and fully supervised experiments on multiple MDS datasets, we show that our model Centrum is better or comparable to a state-of-the-art model. We make the pretrained and fine-tuned models freely available to the research community https://github.com/ratishsp/centrum .

## introduction in multi-document summarization (mds), the input is a set of documents, and the output is a summary that describes important information in a coherent and non-redundant manner @xcite . in recent years, there have been significant improvements in mds due to the availability of mds datasets @xcite @xcite and advances in pretraining approaches @xcite @xcite . in particular, @xcite introduced a pretraining approach called primera (pyramidbased masked sentence pretraining) adapted for mds. to create synthetic summaries, they used the pyramid scheme @xcite , incorporating a process of identifying and ranking entities, followed by grouping sentences containing these entities in the input documents. the sentences with the highest overlap with other documents (measured using rouge) in each group were masked in the input and integrated into the output, forming a synthetic summary. @xcite show that an encoder-decoder model trained on such a corpus attains strong zero-shot, few-shot, and fully supervised results on multiple datasets. however, these synthetic summaries may lack coherence as the sentences are derived from various positions within the input documents. furthermore, there is potential for redundancy, as sentences encapsulating similar information could be selected for inclusion in the summary. in this paper, we propose centrum, a pretraining objective for mds, which is conceptually simple and overcomes these problems. the key intuition is that among a set of documents in a news cluster, the document which shares the most content with the other documents in the cluster can serve as a proxy for the summary of the document set. such a cluster centroid is inherently coherent as it is a humanwritten document. furthermore, because it isn't artificially assembled, it avoids content repetition. in this paper, we pretrain centrum on newshead @xcite corpus and perform zero-shot, fewshot and fully-supervised experiments on various mds datasets. we show that centrum performs favorably compared to primera, especially in the zero-shot and few-shot settings, where there are none or very few training examples available. ## centroid-based selection of document as summary background on @xcite leveraged the newshead corpus @xcite , a compilation of 369,940 news clusters, for pretraining. using the pyramid scheme @xcite , they created synthetic summaries through a multi-step procedure. they gathered the entity mentions in the input documents and rank the entities by the count of documents in which an entity is mentioned. next, they divide the sentences from the documents into distinct groups, such that the sentences containing an entity belong to the same group. they then extracted the sentence with the highest overlap (as quantified by rouge @xcite ) with other documents from each group. this sentence was replaced with a mask token in the input, and copied to the output document. the idea here was to leverage information from other documents to reconstruct the masked sentence. the sentences thus obtained were concatenated to form a synthetic summary. @xcite applied the method to the new-shead corpus @xcite containing news articles clustered by topic. to accommodate long document lengths, they use longformer encoder-decoder (led) architecture @xcite . led supports sparse global attention along with dense local attention on the input. primera employs global attention on specialized tokens (<doc-@xmath0, which act as separators between the documents within the input cluster. the pretrained primera model was then used for zeroshot evaluation, few-shot or full finetuning across multiple mds datasets. problems with primera pretraining primera's reference summaries consist of sentences extracted from varying positions within different documents in a cluster. this method can yield incoherent summaries, as it can be unclear which entities the sentences refer to. we illustrate this with an example of a synthetic summary created using primera in our model we propose an alternate pretraining objective for mds called as centrum. we hypothesize that a document exhibiting the highest similarity with the rest of the documents in a cluster could serve as a proxy for its summary. this method inherently filters out documents that bear she's a fantastic person, solid as a rock and i'm very proud of her success because i really believe i had at least something to do with it." it was unclear exactly what type of records he was referring to -the attendance of 6,500 fell far short of many elton john concerts. . . . (4 sent) donald trump made his way to great falls, montana, on thursday (july 5), primarily to slam democratic sen. jon tester and accuse him of failing to live up to his promises in washington. "i've broken more elton john [attendance] records, and i don't have a musical instrument," he boasted. this is my only musical instrument-the mouth-and hopefully the brain is attached to the mouth. during a rally in great falls, montana, where president trump derided the #metoo movement and attacked individual democratic lawmakers, the president once again bragged about the size of his supporter turnout. "i've broken more elton john [attendance] records, and i don't have a musical instrument," trump said according to yahoo news. "i don't have a guitar, or an organ. . . . this is my only musical instrument -the mouth -and hopefully the brain is attached to the mouth. the brain is so much more important." . . . drawing inspiration from @xcite , we designate a document as the summary if it maximizes the semantic match with other documents in the cluster. specifically, from each document set d in an instance, the best candidate summary ŷ is selected as: vogler et al. ( 2022 ) recently employed a comparable strategy for unsupervised mds. however, our approach differs from theirs by applying this strategy for mds task-specific pretraining. moreover, following @xcite , we employ the led architecture for handling long document context in the input.