Source-summary Entity Aggregation in Abstractive Summarization (2022)

Abstract: In a text, entities mentioned earlier can be referred to in later discourse by a more general description. For example, Celine Dion and Justin Bieber can be referred to by Canadian singers or celebrities . In this work, we study this phenomenon in the context of summarization, where entities from a source text are generalized in the summary. We call such instances source-summary entity aggregations . We categorize these aggregations into two types and analyze them in the Cnn/Dailymail corpus, showing that they are reasonably frequent. We then examine how well three state-of-the-art summarization systems can generate such aggregations within summaries. We also develop techniques to encourage them to generate more aggregations. Our results show that there is significant room for improvement in producing semantically correct aggregations.

## introduction the quality of abstractive summarization systems has improved substantially in the past few years. an important next research question is to better understand the specific linguistic and semantic operations which can lead to high-quality abstractive text. in this work, we focus on how entities can be referred to in summaries, especially with an expression more general than in the source. for example, there are few existing studies about the semantics of text generated by current abstractive systems. some have focused on summary n-grams that are not found in the source text @xcite , and others that look at problems document (cnn) comic books of the past few years have seen a lot of changes (a female thor, anyone?) but not quite so many at one time. three major characters -superman, wonder woman (both of dc comics, a time warner company, like cnn) and archie andrews -came out with new looks (...) summary superman, wonder woman and archie all debuted new looks thursday. three of the most well-known comic book characters of all time look radically different. we focus on a specific semantic operation that summary writers can perform in order to change the level of detail in a summary: the semantic aggregation of named entities in context, as in we then explore how well existing systems can generate copy and novel aggregations that match those found in reference summaries. specifically, given a document, the models must generate a summary, and the aggregations within the generated summary are evaluated against the aggregations in the reference summary. we evaluate three state-of-the-art transformerbased abstractive summarization systems: bart @xcite , pegasus @xcite , and t5 @xcite . the experimental results show that the task is hard, especially for generating novel aggregations. we also explore how to fine-tune bart @xcite to generate more summary-worthy aggregations without compromising the overall summary quality. the performance of all the summarization models is still far below the upper bounds posed by our oracles, showing that there is room for improvement. ## related work semantic generalization in automatic summarization is receiving increasing interest, both in terms of the data and the models. in @xcite , the authors fuse concepts within sentences using hypernymy relations from taxonomies such as wordnet. @xcite focuses on abstracting single concepts. roughly, they train an encoder-decoder architecture on documents where single nouns are replaced with hypernyms, to produce more general summaries. contrary to these approaches based on taxonomies, @xcite use a mixed objective for training encoderdecoder architectures to encourage abstraction in summaries. the level of abstraction was defined in terms of novel n-grams. the surface-level novel n-grams definition of abstractiveness has also been used in recent summarization datasets @xcite . this approximation is convenient for generation since it measures any kind of rewriting. however, being able to explicitly measure different types of abstraction is important for tracking progress. our work is based on this motivation. the closest idea towards entity aggregation is @xcite and we draw heavily from their work. they introduce a dataset and task (tesa) which consists of producing a non-enumerating noun phrase ('former us presidents') that aggregates a set of entities ('clinton', 'bush') in a textual context (a new york times article). their data was collected using crowd annotators and does not specifically focus on any task. our work explores entity aggregations in the context of abstractive summarization. our task can be seen as a referring expression generation problem @xcite where a general phrase in the summary stands in for a set of entities in the source. the task is also related to multi-antecedent coreference resolution and split-antecedent anaphora @xcite @xcite , but most resources and approaches here are aimed at pronominal coreference. some studies address entity-driven summarization @xcite , with the aim of focusing the summaries on the most salient entities. differently to our work, this work does not focus explicitly on the aggregation expressions in the summaries. currently, abstractive summarization systems have vastly improved generation capabilities, achieved by using pre-trained transformers @xcite as a backbone. among the state of the art of text summarization benchmarks, bart @xcite , pegasus @xcite , prophetnet @xcite and t5 (raffel et al., 2020) stand out in terms of rouge. some challenges with these models, such as mitigating hallucinations or ensuring factual consistency, are of great interest. different methods have been proposed here @xcite . of these @xcite ; @xcite incorporate text planning using a sequence of entities to prompt faithful generation. we use such ideas to encourage systems to produce more aggregations by jointly training the models to identify summary-worthy aggregations while learning to generate summaries. ## experimental setup our experiments aim to evaluate the capabilities of state-of-the-art summarization models to generate summary-worthy aggregations. in this section, we describe the task, our oracles, models and evaluation measures. ## results the results of the models and the oracles, at aggregation and summary levels, are presented in tables 7 and 8 respectively. note that copy and novel results are on a subset of the cnn/dailymail test set where aggregations are present in the summary. the full test set results are also reported separately. all the models struggle at aggregation-level with the novel test set, especially when they are evaluated using the most restrictive metric (aggregation exact match). in this case, the models obtain approximately 10 times lower performance than with the copy test set. even with the copy test set, where the models can copy the aggregations from the source to the summary, the results are almost always lower than 50 precision, recall, and f 1 , which shows the difficulty of the task for current summarization approaches. at the aggregation-level, bart+aggregationchains systematically outperforms all the other systems, showing that content planning with aggregation chains is an appropriate strategy to generate more summaryworthy aggregations. however, its performance at the summary-level is worse than that of bart, especially on the cnn/dailymail test set. this regarding bart+pretrainingaggregations, its performance does not significantly differ from bart neither at aggregation-level nor at summary-level. the upper bounds of the performance at aggregation and summary levels are posed by bart+perfectaggregationchains and copy pe-gasus oracles respectively. at the aggregation level, the performance of all the models is far below the upper bound posed by bart+perfectaggregationchains, which suggests that there is great room for improvement of summarization systems. in addition, although the copy sentence oracles also have access to the reference aggregations, they obtain significantly worse results than the bart+perfectaggregationchains. it suggests that summarization systems that have not been trained to consider aggregations properly will struggle in the aggregation-level evaluation. at the summary-level, the performance of the models is more similar to the oracles than in the aggregation-level evaluation, which suggests that there is smaller room for improvement here compared to aggregation-level. ## conclusion and future work we studied source-summary entity aggregation, a frequent phenomenon in the cnn/dailymail corpus. we analyzed the capabilities of state-of-theart summarization systems to generate summaryworthy aggregations, and explored different ways of fine-tuning bart to generate more aggregations. our results suggest that summarization models can improve greatly along these lines. in future work, we would like to explore how to leverage knowledge about the entities to generate better aggregations. another important direction is other types of semantic generalization, such as aggregations of sequences of events. also, we plan to explore source-summary entity aggregation on more abstractive summarization datasets such as xsum and wikihow, which could reflect better the aggregation phenomenon. finally, we would like to investigate deeper questions about semantic aggregation in summarization, e.g., when is generality preferred over specificity in summaries?