Leveraging Contextual Information for Effective Entity Salience Detection (2024)

Abstract: In text documents such as news articles, the content and key events usually revolve around a subset of all the entities mentioned in a document. These entities, often deemed as salient entities, provide useful cues of the aboutness of a document to a reader. Identifying the salience of entities was found helpful in several downstream applications such as search, ranking, and entity-centric summarization, among others. Prior work on salient entity detection mainly focused on machine learning models that require heavy feature engineering. We show that fine-tuning medium-sized language models with a cross-encoder style architecture yields substantial performance gains over feature engineering approaches. To this end, we conduct a comprehensive benchmarking of four publicly available datasets using models representative of the medium-sized pre-trained language model family. Additionally, we show that zero-shot prompting of instruction-tuned language models yields inferior results, indicating the task’s uniqueness and complexity.

## introduction many nlp studies have highlighted the importance of entities to understanding the semantics of a document @xcite . automatically identifying entities in unstructured text documents and linking them to an underlying knowledge base, such as wikipedia, is one of the core nlp tasks, with multiple shared tasks @xcite , benchmarks @xcite @xcite @xcite , and studies @xcite dedicated to solving them. although an entity may play a crucial semantic role in document understanding, not all entities in † work was done while the author was affiliated with bloomberg figure 1 : an example of a document with salient and non-salient entities. entity mentions are highlighted in text. a text document play equal roles. some entities are the central subjects or actors within a document, around which the content and the key events revolve. others are mentioned only to provide additional context to the main event. for example, some entities may be actors in peripheral events, while others are deemed uninformative to the understanding of the document. thus, entity salience in a text is defined as a binary or ordinal rating to quantify the extent to which a target entity is central to a given piece of text @xcite . figure 1 provides an example text along with the mentioned entities and their salience. we note that the salience of an entity to a text is independent of the user's interest when reading or searching the document @xcite , which is usually referred to as entity relevance. it is also distinct from entity importance, which quantifies the overall importance of the entity independent of the document. automatically inferring entity salience was shown to aid search @xcite , improve ranking results @xcite , entity detection @xcite , and enable entity-centric applications such as entity-centric summarization @xcite . in this paper, we study the effectiveness of transformer-based pre-trained language models (plms) in the task of entity salience detection. prior work on determining entity salience relied on heavy feature engineering to craft features explicitly covering relevant aspects, such as entity frequency @xcite , position of entity mentions within a document @xcite , relations to other entities @xcite , document features, such as its length @xcite and lexical features, such as the name of the entity or its context. only a single recent work attempted to use plms in a pipeline which included key entity detection, albeit the scope of the evaluation was limited to a single high performing dataset @xcite . in contrast, our proposed method uses a cross-encoder architecture where a target entity's name or alias and its contextual mentions in a text document are encoded by a plm encoder. the classifier uses the contextual representation and, optionally, positional information about the entity encoded through the decile position embedding vector of mentions to determine the salience score of a target entity. we conduct experiments on four publicly available datasets, two of which were human annotated and two that were curated semi-automatically. we fine-tune several cross-encoders using plms and demonstrate that these yield consistent and significant improvements over feature-based methods, as well as prompting instruction-tuned plms. the latter shows the novelty and complexity of the task of entity salience detection, which requires the model to learn significant task-specific semantic knowledge for this natural language understanding task. our contributions in this paper are the following: ## related work understanding the aboutness of a document is one of the long-standing goals of research in both information retrieval and natural language processing @xcite . several types of approaches have been proposed, including extracting key-terms @xcite , identifying latent topics @xcite , or generating text summaries @xcite . there has been a recent focus in using entities to understand the content of a document. towards this goal, the task of entity salience has been first described for web pages in @xcite and for news content in @xcite . this task can be viewed as a restricted form of keyword or keyphrase extraction @xcite if salience is binary. for the rest of this study, we will use the concept of salience as described in @xcite . the salience labels for entities were obtained either by crowdsourcing labels from multiple raters to identify salient entities @xcite @xcite or by using proxies. for example, @xcite hypothesize that salient entities are those that appear in the article's abstract. @xcite identifies an entity as salient if the wikinews category that corresponds to the entity is also labeled as the category of the article. past studies mostly proposed machine learning methods to infer the salience of a given entity that relied on hand-crafted features. features that can be computed from the target entity mentions and document alone can be categorized into the following: positional (e.g., position in the document, if entity is in the abstract) @xcite , count-based (e.g., number of references to the entity) @xcite , local context @xcite , or global context @xcite . further, joint entity salience resolution can be performed by creating features using the entity graph (e.g., centrality in the entity graph) @xcite . finally, past work also showed that incorporating external knowledge about entities from knowledge bases can boost predictive performance @xcite . automatically inferring salience for entities can directly benefit multiple downstream applications, such as improving ranking results for queries containing entities @xcite or improv-ing the performance of entity detection by joint modelling @xcite . moreover, by inferring salience, new entity-centric applications can be built, such as highlighting salient entities in search @xcite , improving the interpretability of news trends through salient entities @xcite , or identifying entities for creating entity-centric summaries of news stories @xcite . ## problem definition we use the concept of salience as introduced in @xcite : salient entities are entities explicitly mentioned in the document that are objectively important as a function of the structure of the text. the goal of the salience model is to produce a single salience score ψ(e) for the entity e using only the document d and the explicit entity mentions m e . we consider using external knowledge, such as information about entities from knowledge bases, to be outside the scope and leave integration of such knowledge for future work. ## methods pre-trained language models (plms) have shown a remarkable ability to encode syntactic and semantic knowledge in their parameters @xcite that can be leveraged when fine-tuned on downstream natural language understanding (nlu) tasks. we postulate that plms can be harnessed to help in entity salience detection, a targetbased document-level nlu task. in this section, we present an architecture based on the cross-encoder setup adapted to the task of entity salience detection. ## datasets in this section, we describe our entity salience benchmark, which consists of four datasets: two datasets were curated using semi-automated methods and two used human annotations. we provide summary statistics of these datasets and label collection methods in nyt-salience this dataset is introduced in @xcite and is the largest dataset to date for entity salience detection. the dataset is curated with an assumption that salient entities are mentioned in the abstract of a news article in the nyt corpus @xcite . entities and their mentions are identified using a classical nlp pipeline involving pos tagging, dependency parsing, and noun phrase extraction. despite being large-scale, the automatic dataset creation process could introduce noise as corroborated by moderate agreement numbers with human annotators on a subset of the data. the dataset contains a binary salience label for each entity. wn-salience introduced in @xcite , this is another automatically curated dataset consisting of wikinews articles. these are annotated with wikinews categories by their authors. wn-salience identifies salient entities by using the hypothesis that an entity is salient if the wikinews category that corresponds to the entity is also labeled as a category of the article. similar to nyt-salience, this dataset has binary salience labels. sel this is another dataset based on wikinews released by @xcite . however, unlike wn-salience, this dataset is human annotated, where multiple human annotators ranked the salience of entities into one of four categories. to conform with the binary labels of the other datasets, we map the 4 categories into binary labels of {0, 1} by mapping the bottom two classes to not salient and the top two classes to salient. entsum this dataset was introduced in @xcite . to construct this dataset, a randomly selected set of entities spanning a subset of 693 articles from the nyt corpus were assigned salience labels by human annotators on a four-point scale, ranging between [0, 3]. for each document entity pair, two independent annotations were collected, which were increased up to 5 in case of disagreements. if the average annotation score is greater than 1.5 for an entity, it is assigned a positive salience label. ## experiments we experiment on our entity salience benchmark with our proposed plm-based method, other ml and heuristic-based approaches used in past research, as well as an instruction-tuned plm. ## analysis in this section, we perform an analysis of model predictions in order to gain more insights into model behavior and understand potential avenues for further improvement. we thus break down performance by different factors including: the importance of inferring all entity mentions, the position of the first entity mention, and entity mention frequency. ## conclusion this paper aims to leverage the semantic knowledge encoded in pre-trained language models for entity salience detection. we propose the crossencoder method based on transformer-based plms with positional representations and compare its performance to several ml-based methods, heuristic methods, and instruction-tuned llms across four different datasets, two human-annotated and two automatically curated. across all our experiments, the cross-encoder model based on pre-trained language models outperforms all other methods, often with double digit gains in f-1 score. analyses of model behavior illustrate the important effects of mention frequency, mention position, and document length on performance, highlighting areas of future work.