E nt SUM : A Data Set for Entity-Centric Extractive Summarization (2022)

Abstract: Controllable summarization aims to provide summaries that take into account user-specified aspects and preferences to better assist them with their information need, as opposed to the standard summarization setup which build a single generic summary of a document. We introduce a human-annotated data set EntSUM for controllable summarization with a focus on named entities as the aspects to control. We conduct an extensive quantitative analysis to motivate the task of entity-centric summarization and show that existing methods for controllable summarization fail to generate entity-centric summaries. We propose extensions to state-of-the-art summarization approaches that achieve substantially better results on our data set. Our analysis and results show the challenging nature of this task and of the proposed data set.

## introduction automatic summarization is a core nlp problem that aims to extract key information from a large document and present it to the user with the role of assisting them to digest the core information in the document faster and more easily. however, each user may have a distinct information need and generating a single summary for a document is not suitable for all readers of the document. recently, various setups for summarization were proposed such that user preferences can be taken into account in the summarization process. these include providing guidance signals such as summary length @xcite , allowing users to provide terms of interest such as aspects @xcite or entities @xcite or providing users the flexibility to interact with the summary and explore new facets of interest @xcite . the development of such methods may be paramount in enabling the wide-spread usability of summarization technology. figure 1 shows an example of a document, its generic summary and summaries controlled through salient named entities in the original document. high quality reference data sets are needed to foster development and facilitate benchmarking. most summarization data sets are obtained using opportunistic methods such as using abstracts written by editors or librarians when indexing documents. these are by default generic, thus not applicable to controllable summarization. initial research in this area used small scale human annotations to compare between controllable and generic summarization methods @xcite , but these can be prone to biases or qualitative issues, offer only relative quality measurement and do not allow for replicable comparisons between multiple methods or model tuning. thus, this paper introduces a new data set for controllable summarization focusing on entities as control aspects given these are usually key aspects in documents and their summaries. the data set consists of 2,788 human-generated entity-centric summaries across 645 documents that are obtained using a strict quality control process mechanism involving several intermediate annotation steps which can be further used in modelling and analyses such as identifying sentences relevant to an entity. the summaries are elicited largely to merge the most important content in a coherent way, while maintaining factuality during the summary creation process. our data set demonstrates the distinct nature of the entity-centric summarization as opposed to generic summarization and that methods proposed to date for controllable summarization fail at this task. we propose adaptations of state-of-the-art extractive and abstractive summarization methods that significantly improve performance when compared to generic summaries. our contributions are: • the first annotated data set for controllable summarization with entities as targets for control (entsum -entity summarization); • quantitative data set analysis that highlights the challenges and distinctiveness of this task; • evaluation of generic and also controllable summarization methods on the entsum data set; • adaptations of extractive and abstractive summarization methods for performing entity-centric summarization when trained with generic summaries only. opinion summarization is the task of automatically generating summaries for a set of reviews about a specific target and usually involves inferring the aspects of interest, predicting sentiment towards them and generating a summary from the extracted sentences @xcite . amplayo and lapata (2021) studied zero-shot controllability to generate needspecific summaries for movie reviews and evaluated using human comparison judgments. contemporaneous to this work, controllable multi-document summarization for aspects in reviews was introduced @xcite . this work created two data sets used for testing, one focusing on six aspects in hotel reviews (space) and another focusing on 18 aspects for product reviews (oposum+), both obtained using a multi-step annotation process related to the one we use in this paper. interactive summarization is a technique which aims to provide to an interactive faceted summarization of a set of documents and help the user inquire for more information via suggested or free-text queries @xcite @xcite . this setup is focused on a multi-document scenario where relevant content to a target concept is retrieved, then fed to a generic abstractive summarization method. recently, @xcite proposed decisionfocused summarization, where the goal is to summarize information across multiple documents with the goal of aiding a human to forecast an outcome. ## the entsum data set this section details the collection and annotation process for data set creation. we focus on entities as the aspect to control because named entities are central actors in most news articles and entities are key aspects that make good summaries, together with events and facts. initial work on controllable summarization considered entities as one of the target for controls @xcite . most large-scale summarization data sets were obtained opportunistically by mining existing sources of documents and their generic summaries expressed either as titles @xcite , bullet points @xcite summaries created for indexing purposes @xcite or tl;dr's created by scientific paper authors @xcite . however, we could not identify any similar proxies for entity-centric summaries. thus, we created the entsum data set through a manual annotation process. ## methods for an initial modelling attempt for the entsum data set, we evaluate all controllable summarization approaches proposed to date, generic summarization methods, strong heuristics for summarization and a couple of adaptations of state-of-the-art methods for abstractive @xcite and extractive summarization @xcite to the entitycentric summarization task. some of the methods described in this section involve detecting the entity mentions in documents unlabeled with entities in training and/or at inference time. for this, we use a combination of standard methods for ner based on flair @xcite and their coreferent mentions as identified through the spanbert coreference system @xcite . ## results we benchmark all methods described above on the newly proposed entsum data set in order to establish baseline performance of both abstractive and extractive methods for this new task and data set. the results show the following trends across all four evaluation metrics: entity-centric summarization is very different to generic summarization given that methods that do not take entity information into account (lead3 ovr , gsum ovr ) perform significantly lower than the best methods in the same class which use entity information. previously introduced methods (convnet, ctrlsum) for controllable summarization can not perform well on entity-centric summarization with their results being over 17 bertscore and 29 rouge-l lower than the proposed adaptation for abstractive summarization on entity-centric summaries. further, these methods actually obtain lower results by 4.93 bertscore and 7.43 rouge-l than the entityagnostic gsum ovr method, which shows these methods are not effective at modelling entitycentric information through their training and inference process. our proposed adaptations to both abstractive and extractive methods perform well on entity-centric evaluation, despite they were trained on a data set that used proxies for entity-centric summaries. for extractive summarization bertsum ent-top3 performs better than extractive approaches perform better than abstractive methods, which is expected due to the extractive nature of the entsum data set, the gap between the best performing methods (bertsum ent-top3 and gsum ent-sent ) is clear, when using bertscore (+2.02) which better estimates semantic similarity opposed to the n-gram matches used in rouge (+7.66 on rouge-2, +6.03 on rouge-l). lead3 ent is a very strong baseline as expected, because this is a strong baseline for document summarization in general and especially because entsum is by design a more extractive summarization data set. lead3 using oracle selected sentences perform much better than lead3 and shows the benefits of selecting salient sentences (+7.36 rouge-l, +5.16 bertscore) and the benefits of selecting the most important sentences used in writing the summary (further +9.82 rouge-l, +6.26 bertscore compared to top salient sentences). the absolute results also show there is further room for improvement in entity-centric summarization approaches, given that performance of automated methods still lags behind lead3 ent , whereas this is currently surpassed by automated methods in generic summarization. ## conclusion we introduced the first annotated data set (entsum) for controllable summarization where entities are targets for control. we conducted a quantitative analysis of the newly created resource and highlighted how this is different to generic summarization methods. we used the entsum data set for benchmarking state-of-the-art generic abstractive and extractive summarization methods, as well as initial methods for controllable summarization. further, we proposed a new setup for learning entity-centric summaries from generic summarization data sets and, extending previous methods, demonstrated good performance on the newly proposed entsum data set. in the future, we aim to propose new methods for both extractive and abstractive summarization performance through modelling information about the document and the entity in a more complex way. we also plan to create a data set for entity-centric summarization that is more abstractive in nature.