How “Multi” is Multi-Document Summarization? (2022)

Abstract: The task of multi-document summarization (MDS) aims at models that, given multiple documents as input, are able to generate a summary that combines disperse information, originally spread __across__ these documents. Accordingly, it is expected that both reference summaries in MDS datasets, as well as system summaries, would indeed be based on such dispersed information. In this paper, we argue for quantifying and assessing this expectation. To that end, we propose an automated measure for evaluating the degree to which a summary is “disperse”, in the sense of the number of source documents needed to cover its content. We apply our measure to empirically analyze several popular MDS datasets, with respect to their reference summaries, as well as the output of state-of-the-art systems. Our results show that certain MDS datasets barely require combining information from multiple documents, where a single document often covers the full summary content. Overall, we advocate using our metric for assessing and improving the degree to which summarization datasets require combining multi-document information, and similarly how summarization models actually meet this challenge.

## introduction multi-document summarization (mds) consists of creating a short and concise summary that includes the salient information in a set of related documents. beyond the challenges in single-document summarization, a summary of multiple texts is expected to combine and assemble information spread across several input texts. summary: global warming has caused the arctic ice to melt considerably. these changes are threatening the indigenous arctic population and could raise ocean levels worldwide. in order to develop mds models that effectively merge information from various sources, it is necessary that reference summaries in mds datasets should be based on such dispersed information across the source documents. however, to the best of our knowledge, while existing datasets assume that this property is realized, measuring (automatically) the degree of multi-text merging was not investigated in the literature. in this work, we suggest quantifying the degree to which a summary is "disperse" in terms of the minimum number of documents needed to cover its content. accordingly, we develop an automated method for measuring this aspect for any mds summary. to that end, we first identify the potential provenance of the summary information in all source documents. then, for each possible number of documents, we form the subset of documents that includes the largest amount aligned informa-tion with the summary. finally, we define the degree of multi-text merging of an mds summary as a function of the amount of summary information not covered by each subset of documents. we apply our automated measure to evaluate the degree of multi-text merging in four prominent mds datasets (duc, tac, multinews and wcep) as well as the output of five recent systems. our results show that some existing datasets barely involve multi-text merging because the reference summary information mostly appears in a single document. unsurprisingly, the length of the summary has a substantial impact on the amount of multi-text merging since longer summaries cover more detailed information which tends to be spread across documents. taken together, our work is the first to measure and empirically analyze multi-text merging in mds datasets and model summaries. we suggest that future work will use our methodology to develop better datasets and to improve the degree of multitext merging in mds models. ## empirical analysis reference summaries we compute the aac scores on the reference summaries of @xcite @xcite , tac 2008 to 2011 (nist), multinews @xcite and @xcite . we report the aac score for each dataset, as well as the standard deviation of the individual aac scores across the dataset topics, in ## conclusion we propose a measure to evaluate the degree of multi-text merging, an essential aspect in multidocument summarization. using this measure, we found that some prominent mds datasets contain summaries that hardly combine information from multiple input sources. furthermore, we show that fine-tuning on a single effective document, achieves better summary performance than finetuning on the full set of documents.