E nt SUM v2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization (2023)

Abstract: Entity-centric summarization is a form of controllable summarization that aims to generate a summary for a specific entity given a document. Concise summaries are valuable in various real-life applications, as they enable users to quickly grasp the main points of the document focusing on an entity of interest. This paper presents ENTSUMV2, a more abstractive version of the original entity-centric ENTSUM summarization dataset. In ENTSUMV2 the annotated summaries are intentionally made shorter to benefit more specific and useful entity-centric summaries for downstream users. We conduct extensive experiments on this dataset using multiple abstractive summarization approaches that employ supervised fine-tuning or large-scale instruction tuning. Additionally, we perform comprehensive human evaluation that incorporates metrics for measuring crucial facets. These metrics provide a more fine-grained interpretation of the current state-of-the-art systems and highlight areas for future improvement.

## introduction controllable summarization is a rapidly expanding field of research that deals with creating summaries tailored to different elements @xcite @xcite . the controllable elements include entities @xcite , aspects @xcite , users' preferred style @xcite and length @xcite . controllable summarization has the promise to increase the utility and usability of summarization systems by enabling users to obtain summaries that align with their specific needs and preferences @xcite . further, controllable summaries can be used in downstream applications like search @xcite , entity salience @xcite , aspect-based sentiment classification @xcite or question answering. abstractive summarization methods aim to produce new summaries @xcite , which can be obtained through selection, compression and reformulation of the given source document. compared to extractive summarization, abstractive summarization can produce concise summaries that capture the essence of the source text using fewer words, making them more efficient for users to consume. however, abstractive summarization is prone to suffer from issues in consistency with the source document (or factual errors), coherence or fluency @xcite @xcite . to evaluate abstractive summaries, automatic metrics have been proposed, although their correlation with human evaluation on the desirable facts for a summary are not always high or consistent @xcite . in this paper, we focus on the task of abstractive entity-centric summarization. past research on this topic was limited by the ability to comprehensively evaluate models, relying either on singlefaceted human quality judgments @xcite @xcite or reference entity-centric summaries which were very extractive @xcite . to this end, we release an updated version of the entsum dataset @xcite , named entsumv2, where summaries are deliberately made shorter and more abstractive. moreover, we enhance the evaluation process of entity-centric summarization methods by incorporating a comprehensive multi-faceted human evaluation, specifically designed for this task. this human evaluation complements the standard automatic metrics, including rouge and bertscore. by incorporating both automatic metrics and human evaluation, we aim to provide a thorough and robust evaluation of summarization model performance and show the path forward to improving models for this task. separately, we explore training several model architectures on this task and propose several improvements to the training process, which substantially outperform the existing state-of-the-art (+2.5 bertscore, +4.4 rouge-l), instructiontuned models and the strong entity-centric lead3 heuristic. ## data in this paper, we introduce the entsumv2 dataset which contains more compressed abstractive summaries when compared to the original entsum dataset. we build the entsum dataset on top of the new york times' summarization corpus (hereafter referred to as nyt) which is available to use via the ldc. 1 the dataset shares the same set of documents as entsum, but with a stricter length constraint of up to 60 words, half of entsum's. the annotations are performed by annotators trained over multiple rounds on a proprietary annotation platform. the annotators are presented with the original document, the target entity and the salient sentences for the target entity as annotated in the original entsum dataset. a diagram with the annotation process is presented in appendix a. for quality control, we additionally calculated the inter-annotator agreement for the entsumv2 dataset in the final training round using rouge- @xcite and bertscore between the abstractive summary and the proxy summary (entity salient sentences) provided to annotators at annotation time. the entsumv2 krippendorff's alpha for rouge-[1,2,l] and bertscore are 0.75, 0.84, 0.85 and 0.81 respectively, indicating a high overlap. we collect a single summary for each document and entity pair. ## methods we experiment with several methods for abstractive summarization as follows: ## experimental setup training data we employ proxy summaries from the nyt corpus during the fine-tuning process of gsum and t5 models for entity-centric summarization. the original corpus comprised 44,382 training and 5,523 validation pairs (document, summary) for generic summary. to generate entitycentric summaries, we select the first three sentences that mention the target entity. this selection is based on the entity recognition and coreference resolution methods as described in @xcite . given that each document in the corpus contained multiple entities, the training set expanded to 464,339 pairs, while the validation set grows to 58,991 pairs. test data we use the entsumv2 dataset for evaluation only, following @xcite . we conduct experiments by splitting this dataset into training and test sets. however, training on this dataset, even when combined with the additional proxy summaries, does not result in any performance improvements. therefore, to ensure more accurate and dependable evaluations of model performance, we utilize the entire entsumv2 dataset exclusively for testing purposes. implementation details the t5-base foot_1 , t5-v1.1 foot_2 , and flan-t5-base foot_3 models are obtained from the huggingface model repository. we use the gsum implementation provided by the authors. 6 in our gsum experiments, we adhere to the hyperparameters and implementation details outlined in the gsum framework. we conduct fine-tuning of the t5 and t5 v1.1 models for 2 epochs with a learning rate of 2e-5. the batch size is set to 32, and the experiments are performed on nvidia tesla v100 gpus. during inference, we impose a constraint on the t5 models to limit the generated output to a maximum of 60 tokens. ## results we evaluate all models using both automatic and human evaluation for a more comprehensive view on model performance. ## conclusions this paper presents a comprehensive analysis of abstractive entity-centric summarization. we introduce a new dataset -entsumv2 with summaries that are more abstractive and almost half the length of the summaries in entsum, posing additional challenges to summarization models. we explore different model types, improving upon previous top-performing models through data insights and training techniques, as well as surpassing the strong lead3 entity-centric baseline. finally, we conduct the first multi-faceted human evaluation on entitycentric summarization, revealing detailed insights into model behavior and trade-offs, suggesting potential avenues for further enhancement.