EDIN : An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing (2022)

Abstract: Existing work on Entity Linking mostly assumes that the reference knowledge base is complete, and therefore all mentions can be linked. In practice this is hardly ever the case, as knowledge bases are incomplete and because novel concepts arise constantly. We introduce the temporally segmented Unknown Entity Discovery and Indexing (EDIN)-benchmark where unknown entities, that is entities not part of the knowledge base and without descriptions and labeled mentions, have to be integrated into an existing entity linking system. By contrasting EDIN with zero-shot entity linking, we provide insight on the additional challenges it poses. Building on dense-retrieval based entity linking, we introduce the end-to-end EDIN-pipeline that detects, clusters, and indexes mentions of unknown entities in context. Experiments show that indexing a single embedding per entity unifying the information of multiple mentions works better than indexing mentions independently.

## introduction most existing works on entity linking (el) -the fundamental task of detecting mentions of entities in context and disambiguating them against a reference knowledge base (kb) -assume that such kb is complete, and therefore all mentions can be linked. in practice this is hardly ever the case, as kbs are incomplete when they are created and because novel concepts arise constantly. for example, english wikipedia, often used as the reference kb for large scale linking, is growing by more than 17k entities every month. 1 consequently, at the time of deployment el systems are quickly outdated and static evaluation overestimates performance. but as these systems play significant role in many real world industry applications, e.g., moderating discussions around recent events, a dynamic look on el is crucial. nonetheless, related work on this problem is sparse. available datasets @xcite @xcite and models @xcite are outdated and/or small scale and use features which are not readily available @xcite . most importantly, they approach the problem only in parts. we revisit this problem in context of dense-retrieval and large-scale el, e.g., el relying on bi-encoder architecture that runs a nearest neighbor search between mention encoding and a large-scale index of entity encodings. to this end, we introduce edin-benchmark and edinpipeline where unknown entities, that is entities with no available canonical names, descriptions and labeled mentions, have to be integrated into an existing el model in an end-to-end fashion. to the best of our knowledge, edin-pipeline is the first end-to-end pipeline tackling this problem. note that this setting is strictly more demanding than zero-shot (zs) entity linking @xcite , where a textual description of the zs entities is available at the time of training. the edin-benchmark is temporally segmented into two parts, one preceding time t 1 and one preceding time t 2 . with current approaches, an el system created at t 1 is unable to create a denseindex entry -and therefore successfully link -unknown entities introduced after t 1 . the task that we propose consists in adapting a model trained at t 1 using only an adaptation dataset -a set of new documents also mentioning unknown entities -and unsupervised techniques. there are therefore two parts to this task: i) discovery, which consists in detecting mentions of unknown entities in the adaptation dataset and classifying them as unknown and ii) indexing, consisting in mapping co-referring mentions of unknown entities to a single representation compatible with the entity index. by introducing a clear-cut temporal segmentation edin-benchmark targets unknown entities which are truly novel/unseen to all parts of an el system, specifically the pre-trained language model (plm). therefore, the el system cannot rely on implicit knowledge captured by the plm. this is, to the best of our knowledge, a setting that has not been explored before in the context of denseretrieval based el. temporal segmentation also lets us study effects of entity encoder and plm degradation. we observe that precision drops for known entities in novel contexts which points to a large problem of plm staleness also discussed by @xcite @xcite . we show that distinguishing known from unknown entities, arguably a key feature of an intelligent system, poses a major challenge to denseretrieval based el systems, as a model has to strike a delicate balance between relying on mention vs. context: context is crucial to distinguish unknown entities carrying the same name as known entities and to co-refer different mentions of the same unknown entities, while mentions are essential to distinguish unknown entities with different name but semantic similarity to existing ones. on the side of indexing, inserting unknown entities into a space of known entities poses problems of interference with known entities in their close proximity. for instance, when first encountering mentions of biontech we want to create an index entry in proximity of other biotech companies but in a way that linking can still differentiate between them. we find that adapting the el model to the updated index, is essential. we experiment with different indexing methods. in particular, we contrast single mention-level indexing @xcite with indexing clusters of mentions. we find that unifying the information of multiple mentions into a single embedding is beneficial. we summarize our contributions as follows: i) we introduce the edin-benchmark, a large scale end-to-end el dataset where unknown entities need to be discovered and integrated into an existing entity index in an unsupervised fashion. ii) we propose the edin-pipeline in the form of an extension of existing dense-retrieval architectures. iii) we contrast this task with zs el, and provide insight on the challenges it poses. iv) we show that indexing a single embedding per entity, unifying the information of multiple mentions, works better than indexing mentions independently. data and evaluation code is located here: https: //github.com/facebookresearch/edin ## task definition we formally define end-to-end el as follows: given a paragraph p and a set of known entities e @xmath0, each with canonical name, the title, t(e i ) and textual description d(e i ), our goal is to output a list of tuples, (e, [i, j]), where @xmath1,j spanning from the i th to j th token in p. we call a system that solves this task based on d(e i ) a descriptionbased entity linking system l. for edin-benchmark, after training a model l t1 at time step t 1 , a set of unknown entities e @xmath2, descriptions and labeled mentions is introduced between t 1 and t 2 > t 1 . the task is to adapt l t1 in an unsupervised fashion such that it can successfully link mentions of e u e k . we use three dataset splits: the training set d train to train l t1 , the adaptation dataset d adapt used to adapt l t1 and the test set d test to evaluate. both d adapt and d test include mentions between t 1 and t 2 . the model relies on d adapt to discover e u and extract representations to integrate e u into the entity index. we ensure that d adapt and d test are disjoint to prevent leakage of test mentions into entity representations extracted from d adapt . ## edin-pipeline our edin-pipeline is built on top an end-to-end extension of the dense-retrieval based model blink @xcite and is similar to @xcite . it is composed of a mention detection (md), entity disambiguation (ed) and rejection (r) components. md detects entity mention spans [i, j] in context relying on bert @xcite . ed links these mentions to @xmath3. it relies on biencoder architecture running a k-nearest-neighbor (knn) search between mention encoding and candidate entity encodings (the entity index). mention encodings are pooled from bert-encoded paragraph tokens p 1..n : to integrate these into the index of e k , mentions in singlesentence contexts are concatenated and mapped to a single embedding using the entity encoder. after adaptation, the updated entity index is used for standard el in an inductive setting. entities are represented using blink's frozen entity encoder: @xmath4. more information about architecture and training are detailed in appendix a. ## unknown entity discovery and indexing we introduce an end-to-end pipeline to encode e u into l t1 's entity index. the process is depicted in figure 1 . this pipeline is fully unsupervised and only relies on d adapt . it follows a two-step process: i) in discovery the el system detects mentions of unknown entities and recognises them as being unknown; ii) during indexing, co-referring mentions of unknown entities are mapped to a single embedding compatible with the entity index. after adaptation the updated model is tested on d test . ## evaluation as mentions of type e u are significantly less frequent than mentions of type e k , we report results on these two types separately. for discovery, we report precision and recall of e u classification and clustering metrics. to evaluate end-to-end el, we compute precision (p) and recall (r) following @xcite but using a hard matching criteria. to do so for cluster-based discovery, canonical names of indexed clusters need to be consistent with the set of test labels. our method of assigning canonical names to clusters based on pooling over mentions is not. to resolve this mismatch we pool over the gold labels associated with these mentions instead of the mentions themselves. this is only done for evaluation. unsupervised clustering of mentions in d adapt may suffer from two kinds of errors: i) clusters can be incomplete, e.g., mentions of a single entity can be split into multiple clusters which can lead to indexing the same entity multiple times and ii) clusters can be impure, e.g., mentions of different entities end in the same cluster, which leads to conflation of multiple entities into one representation. in our evaluation we use the gold labels for computing standard el metrics by associating possibly more than one cluster to each e u , and consider a prediction correct if a mention is linked to any of the clusters associated with the correct entity. el metrics could fail to capture shortcomings in establishing co-references between mentions though, therefore we report clustering metrics alongside el metrics. we follow @xcite and report normalized mutual information (nmi). ## edin-benchmark to construct the entity index, we download wikipedia dumps from t1 and t2 and extract entity titles and descriptions. setting t 1 to september 2019 (the date when blink was trained) the kb consists of 5.9m entities, setting t 2 to march 2022 an additional set of 0.7m entities is introduced. wikipedia and oscar data is created as follows. wikipedia: since usually only the first mention of an entity inside a wikipedia article is hyperlinked, we annotate a subset of wikipedia. we use a version of l that was trained at t2 on a labelled non-public dataset. while noisy, these predictions are significantly better than what our best discovery and indexing methods can achieve, therefore we adopt them as pseudo-labels for the purpose of comparing approaches. as discovery and indexing methods improve, manual labelling of the evaluation data will afford more accurate measures. wikipedia provides time stamps which enables us to separate two time splits. oscar news: this dataset is based on the common-crawl dataset oscar @xcite . we select a subset of english language news pages which we label automatically as described above. the dataset consists of 797k samples, which we split based on their publication date. we publish this dataset using stand-off annotations and code to download the relevant raw data. to enable evaluation of future versions of plms and el systems, we also publish our data processing scripts. for both types of datasets we publish two time splits: d 1 , containing samples preceding t1, which is used to train model l t1 and d 2 , with samples preceding t2, which is used to train an upper bound model l t2 . to adapt l t1 , we hold out a subset of data from between t1 and t2 to construct d adapt (d adapt ∩ d 2 = ∅). remaining samples are randomly split into train, dev, test. figure 2 illustrates the different data splits. overall dataset statistics are listed in to construct d adapt , we follow @xcite , and set the ratio of mentions of type e u to e k to 0.1. 2 as d t2 -test covers both known and unknown entities, we use this dataset for edinpipeline evaluation. in oscar d t2 -test, the average number of mentions per e u is 5.6 and it is ten times lower than for e k . covid-19 is the most occurring unknown entity with 12k mentions. 638k e u are not mentioned at all and only 733 are mentioned more than ten times. ## results and discussion in the following sections, we discuss results for oscar data. results on wikipedia data are consistent but lower and shown in appendix f. our main findings are shown in overall, our results show: in whats to come, we first discuss upper and lower performance bounds. then, we follow our two-step pipeline where we first present results on discovery and indexing separately and then assemble the full end-to-end pipeline. recall our terminology: ## related work el is an extensively studied task. prior to the introduction of plms, el systems used frequency and typing information, alias tables, tf-idf-based methods and neural networks to model context, mention and entity @xcite @xcite @xcite @xcite @xcite @xcite . @xcite present a plm-based dual encoder architecture that encodes mentions and entities in the same dense vector space and performs el via knn search. @xcite proposed the zs el task and show that domain adaptive training can address the domain shift problem. subsequently, @xcite showed that pre-trained zs architectures are both highly accurate and computationally efficient at scale. none of these works tackle the problem of unknown entities. recently, fitzgerald et al. ( 2021 ) model el entirely as mappings between mentions, where inference involves a nn search against all known mentions of all entities in the training set. in this setting mentions need to be labeled. they do not explore their approach in the setting of unknown entities. prior to dense retrieval-based el, unknown entity discovery work includes: @xcite train a classifier to determine whether the top ranked el candidate is unknown relying on local context, global wikipedia coherence, and additional manually crafted features. @xcite introduce a model for unknown entity discovery and typing leveraging incompatibilities and correlations among entity types. @xcite ; @xcite study a variety of features for unknown entity discovery: @xcite use perturbation-based confidence measures and key-phrase representations and @xcite explore different feature spaces, e.g., topical and search engine features. these features are not readily available and incorporating them into plmbased approaches is not straightforward; @xcite ; @xcite introduce shared tasks for discovery. these tasks are defined on comparatively small datasets and target only named entities; @xcite introduces a time sensitive method of discovering emerging entities relying on twitter data. none of these works consider unknown entities in an end-to-end setting including mention detection, unknown entity discovery and indexing. also, we cannot use their datasets to evaluate as these entities were part of training the plm. in the context of named entity tagging, @xcite showed that entity taggers can be effectively updated by incorporating contemporary unlabeled data using semi-supervised learning. closely related to el is the task of cross document entity co-reference (cdc), where no reference kb is present @xcite @xcite @xcite @xcite . @xcite benchmark methods for streaming cdc, where mentions are disambiguated in a scalable manner via incremental clustering. our work can be seen as bridging between the world of cdc and el. most recently, @xcite introduce a new el method using document-level supervised graph-based clustering. @xcite extend this work to cross-document el and entity discovery. in this work, we adopt a more standard bi-encoder architecture (i.e. blink), with better el scalability potential (memory linear in the number of entities and not in the number of mentions) and an existing end-to-end extension. we use a modified version of their discovery method. ## conclusion this work introduced edin-benchmark and edinpipeline. edin-benchmark is a large-scale, endto-end el benchmark with a clear cut temporal segmentation for unknown entity discovery and indexing. edin-pipeline detects and clusters mentions of unknown entities in context. these clusters of unknown mentions are then collapsed into single embeddings and integrated into the entity index of the original el system.