Analyzing Multi-Sentence Aggregation in Abstractive Summarization via the Shapley Value (2023)

Abstract: Abstractive summarization systems aim to write concise summaries capturing the most essential information of the input document in their own words. One of the ways to achieve this is to gather and combine multiple pieces of information from the source document, a process we call aggregation. Despite its importance, the extent to which both reference summaries in benchmark datasets and system-generated summaries require aggregation is yet unknown. In this work, we propose AggSHAP, a measure of the degree of aggregation in a summary sentence. We show that AggSHAP distinguishes multi-sentence aggregation from single-sentence extraction or paraphrasing through automatic and human evaluations. We find that few reference or model-generated summary sentences have a high degree of aggregation measured by the proposed metric. We also demonstrate negative correlations between AggSHAP and other quality scores of system summaries. These findings suggest the need to develop new tasks and datasets to encourage multi-sentence aggregation in summarization.

## introduction abstractive summarization aims to gather important information from some source text and to synthesize this information into a brief, informative, and factually correct summary. summary-worthy information on a topic can be located in multiple parts of the document or even in different documents in the multi-document summarization case. they may appear in multiple sentences with either overlapping content or complementary information that is related in discourse. therefore, aggregation, the process of combining multiple related pieces of information, is necessary to generate more useful and concise abstractive summaries. ## method in this section, we propose aggshap, an automatic metric to quantify the degree of aggregation of a summary sentence using the shapley value. shapley value is a concept from cooperative game theory used to determine the contributions of individual players to the outcome of a coalition game. we consider how much information in a summary sentence is covered as a coalition game in which source sentences are players. the shapley value of a source sentence can be interpreted as its contribution to covering information in the summary sentence. the aggshap score of a summary sentence captures the dispersion of source sentences' shapley values. the degree of aggregation of a multi-sentence summary is the mean aggshap scores of summary sentences. ## evaluations of aggshap we validate the effectiveness of aggshap through two experiments. first, we show fusional sentences can be distinguished from extractive ones. next, we compute correlations between ag-gshap or word overlap metrics on the one hand and direct assessment of aggregation on the other. ## analysis given our automatic tool for measuring aggregation, we can use it to investigate the current state of multi-sentence aggregation in abstractive summarization. first, we study whether widely used datasets have sufficient signals to train summarization systems to perform multi-sentence aggregation (sec. 5.1). next, sec. 5.2 presents how well summarizers that are trained or fine-tuned on one of these datasets (cnn/dm) perform aggregation. finally, we are interested in whether the quality of a summary is affected by its degree of aggregation. (sec. 5.3) ## conclusion in this paper, we propose aggshap to quantify aggregation operations in abstractive summarization. our metric effectively distinguishes sentences that require multiple points of dependencies from those that do not in a dataset containing fusional summary sentences. moreover, it has a stronger correlation with human ratings of aggregation than existing n-grams overlap measures. we use ag-gshap to compare the levels of aggregation in summarization datasets and conclude that most recent summarization datasets from the news domain contain limited instances of reference summaries that require aggregation. we show that abstractive summarization models rarely perform semantic aggregation beyond simple concatenation of text units. finally, we find improvements in the dimension of aggregation may not be rewarded by current evalu-ation schemes of general summarization qualities. future evaluations should thus focus specifically on the issue of aggregation, ideally in a domain or setting whether aggregation is necessary to derive a reference summary or a useful conclusion.