GUM sley: Evaluating Entity Salience in Summarization for 12 E nglish Genres (2024)

Abstract: As NLP models become increasingly capable of understanding documents in terms of coherent entities rather than strings, obtaining the most salient entities for each document is not only an important end task in itself but also vital for Information Retrieval (IR) and other downstream applications such as controllable summarization. In this paper, we present and evaluate GUMsley, the first entity salience dataset covering all named and non-named salient entities for 12 genres of English text, aligned with entity types, Wikification links and full coreference resolution annotations. We promote a strict definition of salience using human summaries and demonstrate high inter-annotator agreement for salience based on whether a source entity is mentioned in the summary. Our evaluation shows poor performance by pre-trained SOTA summarization models and zero-shot LLM prompting in capturing salient entities in generated summaries. We also show that predicting or providing salient entities to several model architectures enhances performance and helps derive higher-quality summaries by alleviating the entity hallucination problem in existing abstractive summarization.

## introduction the task of salient entity extraction (see) is to identify entities that are central to a document's overall meaning. previous work on see has relied on crowdsourcing @xcite or user statistics on the web (e.g. clickstream data, @xcite to derive salience labels for entities. in this study, we extend an approach from @xcite , who considered an entity salient if it also appears in a human-written summary or abstract of a news article, and we cover many further genres rather than just news. figure 1 shows an example of salient entities in a conversation annotated according to our definition of salience. see is increasingly important as nlp systems move from understanding document 'aboutness' at the word level (e.g. keyword extraction) @xcite to entity level document understanding @xcite . therefore, a dataset with see labels can benefit downstream applications such as information retrieval and summarization, which extract salient information from large documents and prioritize specific entities in controllable models. although several see datasets already exist @xcite @xcite @xcite , most are predominantly collected from news articles and derive labels using crowdsourcing or "found" information such as hyperlinks, which are not intended to annotate salience per se. this has two major limitations: first, crowdsourcing see without rigorous training and clear definitions of salience may be biased towards individuals and inconsistent interpretations of what is considered salient. second, focusing on news limits system performance on more diverse data (e.g. conversation, vlogs, etc.). to investigate the role of see in tasks such as summarization, previous entity-centric work @xcite @xcite has compared summaries generated by entity-aware methods with generic summarization methods qualitatively. as part of our evaluation, we combine manual and automatic, qualitative and quantitative analyses to assess see impact on several approaches to summarization. in this paper, we therefore present and evaluate a gold standard dataset manually annotated with see labels, by identifying all entities that appear in a human-written summary as salient, making the task less subjective. our dataset, called gumsley (gum salient linked entity corpus) is based on the existing ud english gum corpus (georgetown university multilayer corpus, zeldes 2017) and goes beyond other entity salience datasets in covering all named and non-named salient entities for 12 genres of english text. gumsley also enables the evaluation of see annotations in a broad spectrum of genres and tasks, since the data contains wikification identifiers for named entities, as well as comprehensive coreference resolution. our results show that a significant amount of salient entities are not captured by sota abstractive summarization models or out-of-the-box llms (section 5.1). we also conduct a quantitative analysis to show that providing gold or even predicted salient entities to models helps to generate a higher quality summary (section 5.2). ## related work entity salience datasets the growing interest in see is demonstrated by increasing numbers of annotated datasets, with different approaches to recognizing entities and assigning labels. the first step is usually entity identification. while some datasets @xcite apply a multi-step nlp pipeline (np extraction, coreference resolution, possibly a named entity resolver) to pinpoint entities, others @xcite @xcite have done so manually. since pipelines may propagate errors to later steps, full manual annotation is used in our study to avoid such issues. to collect salience labels, most studies @xcite prefer human annotation using crowdsourcing. although crowdsourcing may outperform automated methods, it is inevitably noisy and can suffer from subjective bias issues @xcite since people have different judgments on what they consider salient. in this study, we follow a more regi-mented approach similar to the nyt salience corpus @xcite , which considers entities that also appear in an abstract or summary as salient. unlike dunietz and gillick, which uses automatic coreference resolution to detect mentions in newspaper summaries, we use fully manual annotation coupled with gum's carefully written summaries which have consistent guidelines and style across 12 english genres @xcite , rather than found abstracts or teasers limited to news or academic data. ## annotation process our dataset, gumsley, is based on the open access gum @xcite , a manually annotated multilayer corpus with universal dependencies (ud) parses @xcite , entity information (entity types, wikification links and more), coreference resolution and discourse parses, as well as a human-written summary for each document. data comes from 12 text types and covers over 200k tokens (see gumsley adds a layer of entity salience labels to all named and non-named entities in gum, annotated by three trained experts (phds/phd students in computational linguistics). the main goal is to annotate a subset of entities as salient if they are mentioned in the summaries, regardless of subjective judgments about their importance. annotators are asked to look at the source document as well as the human-written summary first, and then make a binary judgment on every mention as to whether it is in both. in this way, annotators mark a judgment for every mention in both documents. this approach, which goes back to dunietz and gillick 2014, assumes that if something is salient it should appear in the summary, and conversely, if it appears in the summary, it must be salient, since summaries are meant to be as short and informative as possible. this assumption is mirrored in gum's summary guidelines foot_0 . we reason that while this approach could over-generate, it should have high recall, since it would be hard to summarize a document while omitting salient entities. despite this, we note that our approach still flags only a fraction of entities as salient (around 7%, see we also double-annotated 21,770 tokens of the data corresponding to the 24 test documents of the ud release of gum, containing 3,283 entities (≈ 10% of the data in in this case, "dancing lewdly" will not be marked as salient because "the whole dance" is not coreferred to by a pronoun in the document. these are thus all marked as salient. ## experimental setup in order to evaluate the usefulness of see annotations we apply our data to the task of automatic summarization and test 1) whether system summaries capture gold salient entities identified by humans, and 2) whether see information can improve summarization quality. we evaluate the following models: brio brio @xcite is a recent sota abstractive summarization model, trained and finetuned on three newswire datasets: the cnn/daily mail dataset (cnn/dm, hermann et al. 2015), xsum @xcite , and the nyt dataset @xcite . it uses a novel training paradigm that introduces a contrastive learning component to estimate the probability of the generated summaries more accurately. we chose the pre-trained xsum brio model foot_3 , which most closely resembles the style of gum's single sentence summaries (cf. figure 1 ). we test whether the summaries generated by the model are able to capture gold salient entities in gumsley using the ud test partition (see ctrlsum ctrlsum @xcite is a summarization model used for generating abstractive summaries. it is considered a controllable summarization method because it produces summaries based on user input, which can specify entities of interest (in the form of keywords), summary length, and questions that the summary should answer. the system is a fine-tuned version of the bart large model @xcite based on three training datasets: cnn/dm, arxiv scientific papers @xcite , and bigpatent (patent documents, @xcite . ctrlsum is designed to separate test-time user control of summarization and the training process. during training, summaries are conditioned on the source document and automatically extracted keywords. at test time, a control function is applied to map control aspects to keywords, while model parameters from training remain unchanged. thus ctrlsum differs from other controllable summarization methods in not requiring separate models for each control aspect, generalizing to new keywords at test time. we use the pre-trained ctrlsum model foot_4 in three scenarios: gold, pred and zero. for gold we use the 3 most frequently mentioned gold salient entities in each document foot_5 as "keywords" (all unique mentions of these entities are used, excluding pronouns); in pred we generate predicted salient entities using gpt-4 (openai 2023), a generative llm that achieves humanlevel performance on a range of benchmarks, using the prompt 'find the top 3 salient entities in the following document.', and in zero we test without adding salient entities. gpt-4 gpt-4 (openai, 2023) is the latest version of generative pre-trained transformers at the time of writing. although training details for gpt-4 are not released (incl. model size, architecture, dataset, training method, etc.), we know from technical reports (openai, 2023) that it was trained using masking and reinforcement learning from human feedback (rlhf). for a more robust comparison between the finetuned models (brio and ctrlsum) and promptbased models, we control the length of gpt-4 prompts using the following prompt: summarize the following article in n sentences. since brio's xsum model produces one-sentence outputs and ctrlsum summaries are mostly 2-3 sentences, we can compare gpt-4 summaries with both systems using the sentence-count length prompt n. in order to test whether adding gold or predicted entities to the model helps generate better summaries, we use the following prompt format: summarize the following article in n sentences. in your summary, make sure to include the following words: <gold or predicted entity 1,2,3>. in this section, we evaluate model performance and the impact of see on two aspects of summarization: section 5.1 shows a manual evaluation of entity-level performance (are all salient entities included in summaries?) on the test set foot_6 (24 documents/ 6k entity mentions, ## conclusion this paper presented gumsley, the first manually annotated entity salience dataset covering all named and non-named entities for 12 genres of english text. gumsley achieves a high level of agreement in all 12 genres, creating a high-quality entity salience dataset that allows the evaluation of see annotations in diverse genres. our evaluation shows that a significant amount of salient entities are not captured by sota abstractive summarization models and prompt-based llms and that adding salient entities to model inputs substantially enhances the coverage. we also show that adding such entities helps reduce hallucina-tions in less common genres (e.g. textbooks and travel guides) to a large extent, generating higherquality summaries. we hope that gumsley will enable further research on entity salience and can serve as a challenging dataset for testing text summarization methods in a wide range of genres focusing on entities.