Predicting Entity Salience in Extremely Short Documents (2024)

Abstract: A frequent challenge in applications that use entities extracted from text documents is selecting the most salient entities when only a small number can be used by the application (e.g., displayed to a user). Solving this challenge is particularly difficult in the setting of extremely short documents, such as the response from a digital assistant, where traditional signals of salience such as position and frequency are less likely to be useful. In this paper, we propose a lightweight and data-efficient approach for entity salience detection on short text documents. Our experiments show that our approach achieves competitive performance with respect to complex state-of-the-art models, such as GPT-4, at a significant advantage in latency and cost. In limited data settings, we show that a semi-supervised fine-tuning process can improve performance further. Furthermore, we introduce a novel human-labeled dataset for evaluating entity salience on short question-answer pair documents.

## introduction entity salience (es) is a natural language understanding task concerned with determining which entities mentioned in a passage of text are most salient to the passage. salience refers to the centrality of an entity to the content of a text rather than the intrinsic importance of the entity beyond the text or its relevance to the perspective of a particular reader @xcite . if entities are to be automatically extracted from the text, entity recognition and linking is performed before applying an entity salience model. the role of the es model is to score the entities so they can be filtered (or ranked) by downstream applications. in the example in figure 1 , there are three entities extracted from the question-answer (q/a) pair: popsicle, frank epperson and san francisco. the entity salience task is to classify the entities in the q/a pair as salient or non-salient. the ground truth labels for this example indicate that popsicle and frank epperson are salient while san francisco is non-salient. note that the extraction and linking of the entities is done by a separate entity recognition and linking model and is not considered part of the entity salience task. question: who invented the popsicle? answer: the popsicle was invented by frank epperson, an 11-year-old from san francisco. entities: in extremely short documents, many of the signals that are useful features for determining salience in longer documents, such as position, frequency and co-occurrence patterns are likely to be absent or attenuated @xcite . in this setting, a semantic understanding of the document and the entities is more critical. large language models and their ability to represent text through dense embeddings are a natural fit in this situation. in this paper, we propose to model the salience of an entity as the similarity of its embedding to the embedding of the overall text passage, where each embedding is generated by a common sentence encoder. specifically, this paper makes the following contributions: ## wikiqa-salience dataset existing es datasets @xcite @xcite focus on longer documents such as web pages and news articles (with many hundreds of words) rather than the short texts that are the focus of this work. in this section, we describe the creation of a dataset, wikiqa-salience, for evaluating entity salience on extremely short question-answer pair passages. we leveraged the wikiqa dataset as a starting point to create a new es dataset from publicly available data. the wikiqa corpus is an answer sentence selection (as2) dataset where the questions are derived from query logs of the bing search engine, and the answer candidates are extracted from wikipedia @xcite . the examples are q/a pairs in natural language with full-sentence (non-factoid) answers, which resemble the type of responses provided by conversational assistants. we first selected the q/a pairs in the corpus where the answer is labeled as correctly answering the question (positive pairs). then we applied the refined named entity resolution model @xcite to the combined question-answer text to extract named entities foot_1 . we augment each entity with the name, description and aliases of the entity from wikidata. since wikidata descriptions are typically extremely brief, we further augment the entities in the dataset with more detailed information from wikipedia pages (wherever these are available) including the wikipedia summary (i.e., the first section of the page) and the first 100 noun-phrases from the arti-cle extracted with the spacy nlp library. ground truth labels were generated by crowd workers on the amazon mechanical turk platform who rated the relevance of each entity to the q/a pair it was extracted from on a three level scale ("high", "moderate", "low"). five independent passes of annotation were performed for each entity. the finished dataset consists of 687 annotated q/a pairs with the linked entity data from refined, entity details from wikidata, and the (5-pass) crowd worker salience ratings. to aggregate the multiple annotator passes, we take the median rating (after mapping to numeric values), which unlike majority vote considers the inherent ordering of the labels. the 687 q/a pairs contain 2113 entities (unique at the q/a pair level), and the mean length of the question-answer text is just 190.6 characters and 32.9 words. the distribution of the es labels is significantly skewed towards salient entities with 1089 rated "high", 535 rated "moderate" and 489 rated "low". for the purpose of using the labels in a binary classification task, in this work we map high and moderate ratings to "salient" and low to "non-salient". the inter-rater agreement of the binary labels measured by the fleiss' kappa score is 0.230, which indicates "fair agreement" between the annotators @xcite . additional details on the construction of the dataset can be found in appendix a.6. ## experiments in the first set of experiments, we evaluate four pre-trained models in combination with several sets of entity text features. the four pre-trained models are composed of two model sizes selected from two families of sentence embedding models: all-minilm-v2 (sbert) @xcite and gte @xcite . we treat the concatenated question and answer text for entries in the dataset as the context for predicting salience. for entity text features, we consider the entity name (name), wikidata entity description (desc), the first section of the entity wikipedia page (fs) and the first 100 noun-phrases from wikipedia (np), as well as combinations of these. we measure performance using macro averaged f1 score (macro-f1) where the operating point of each model has been tuned to maximize the macro-f1 score. our reason for using macro averaging in this case, rather than the more conventional binary f1 score, is that the classes are highly imbalanced and we consider classification accuracy on the negative ("non-salient") class as important as classification on the positive class ("salient"). macro-f1 balances these considerations and makes the metric invariant to the assignment of labels to the classes. in the second set of experiments, we look at the performance benefits from fine-tuning the model. as a source of unlabeled training data, we use 23,843 q/a pairs published on the alexa answer crowd sourcing platform (alexaanswers, 2023). we also use 500 manually labeled examples from alexa answers as our validation dataset for selecting the best model checkpoint during fine-tuning. we evaluate these models using the same test set and metrics as in the first set of experiments. additional details about the model training process are included in appendix a.4. as a baseline, we compare our method to the entity salience prediction methods used in @xcite , which were adapted from @xcite . we focus on the position and frequency oriented features that previous works found to be useful but adapt them to the context of extremely short documents. we use the character position of the start of the first mention (first-mention-position) as a feature instead of the sentence of the first mention. we also use the rank order of the first mention (first-mention-order) and the number of times that the entity is explicitly mentioned (mention-count). we use a random forest classification model which we train and evaluate using three-fold cross-validation on the wikiqa-salience dataset. to provide additional benchmarks for assessing model performance, we also evaluate the following: ## deployed application our use case is for a conversational assistant, where entity salience is used to identify the most important entities in the current question-answer interaction. the most salient entity is used for retrieving explorable content to show on the screen accompanying the spoken answer. our baseline system relied on simple heuristics of entity count and position in the context with respect to a predicted answer span. we selected the gte-small-finetuned-name model to deploy due to runtime constraints on the availability of features and tuned the operating point, using a labeled dataset, to primarily improve the recall of salient entities while not harming the precision. we deployed the new model alongside the baseline system in an a/b test and monitored performance for two weeks. our key online performance indicators were the percentage of questionanswer interactions where we identified at least one salient entity (salient entity coverage) and the clickthrough-rate of explorable content shown on screen (ctr). compared to the baseline system, the entity salience model increased salient entity coverage by 25.7% (relative) and the ctr of explorable content increased 2%. ## related work a few previous studies have looked at the topic of entity salience. most have focused on longer documents and found that structural features (such as position, use in title, etc.) or statistical features (frequency of occurrence) are the most useful features for their models @xcite @xcite . some more recent works have also incorporated word or entity embed-dings as features @xcite . contemporaneous work by @xcite explored entity salience detection in news articles by fine-tuning pre-trained transformer models with classification heads that use contextualized entity embeddings. a closely related research area to es is keyword extraction -selecting a small number of words (or phrases) from a document which can concisely describe most important topics in the document. es can be conceived of as a keyword extraction task where the set of keywords to be considered is limited to the named entities in the document. while there is a large and diverse body of literature on keyword extraction techniques (hasan and ng, 2014), prior methods typically employ different combinations of statistical, graph-based, and embedding-based features @xcite @xcite @xcite . embedding based methods of keyword extraction generally work by comparing the similarly of keyword embedding to a passage embedding. this category of methods is most closely related to the work described in this paper. notable, examples include embedrank @xcite and keybert @xcite . also, sharma and @xcite uses an unsupervised embedding based approach to generate (noisily) labeled examples which are used to train a model. ## conclusion in this work, we propose a model for entity salience that works in the context of extremely short documents and introduce a new dataset for evaluating entity salience based on wikiqa. we show that this simple model can perform well in conjunction with pre-trained sentence transformers. we also demonstrate a data efficient approach to fine-tuning the model that achieves performance on-par with the far larger gpt-4 model on the entity salience task, while achieving far lower latency and cost, and is within a few percentage points of human performance on our dataset.