Zero-shot Entity Linking with Less Data (2022)

Abstract: Entity Linking (EL) maps an entity mention in a natural language sentence to an entity in a knowledge base (KB). The Zero-shot Entity Linking (ZEL) extends the scope of EL to unseen entities at the test time without requiring new labeled data. BLINK (BERT-based) is one of the SOTA models for ZEL. Interestingly, we discovered that BLINK exhibits diminishing returns, i.e., it reaches 98% of its performance with just 1% of the training data and the remaining 99% of the data yields only a marginal increase of 2% in the performance. While this extra 2% gain makes a huge difference for downstream tasks, training BLINK on large amounts of data is very resource-intensive and impractical. In this paper, we propose a neuro-symbolic, multi-task learning approach to bridge this gap. Our approach boosts the BLINK’s performance with much less data by exploiting an auxiliary information about entity types. Specifically, we train our model on two tasks simultaneously - entity linking (primary task) and hierarchical entity type prediction (auxiliary task). The auxiliary task exploits the hierarchical structure of entity types. Our approach achieves superior performance on ZEL task with significantly less training data. On four different benchmark datasets, we show that our approach achieves significantly higher performance than SOTA models when they are trained with just 0.01%, 0.1%, or 1% of the original training data. Our code is available at https://github.com/IBM/NeSLET .

## introduction entity linking is a fundamental task in the field of natural language processing (nlp) and plays an important role in numerous applications including knowledge base (kb) question answering, document understanding, dialogue systems, etc. consider the sentence -"she noticed a jaguar speeding on the highway." in this sentence, the phrase ## problem definition the zel task is akin to a cross-domain classification task where entities play the role of classes. it has two distinct characteristics -(i) number of classes (in both source and target domain) could be in the order of millions, e.g., wikipedia has more than 5 million entities, (ii) classes are not merely labels but have rich features in the form of short textual descriptions. formally, in a zel task, we are given an entity set e s , called seen (aka train) entities. each entity @xmath0. we are also given another entity set e u , called unseen (aka test) entities, where in these datasets, the first part m i (and m j , m k ) corresponds to the input text string along with the entity mention substring marked. the second part e i (and e j , e k ) corresponds to the gold entity that must be linked to this mention. a standard practice is to represent m i as the following tuple -(left context, mention, right context). for the sentence "she noticed a jaguar speeding on the highway", we have m @xmath1, jaguar, speeding on the highway) for any given mention @xmath2, it induces a score for each entity @xmath3. the zel model uses these induced scores f zel (m, e) to rank all the entities and highest-scoring entity is predicted as the final answer. the performance of a zel model is measured via hits@k for @xmath4. hits@k measures if the gold entity appears within the top-k elements. for @xmath5, it is called accuracy. ## neslet as mentioned earlier, neslet is an mtl-based neuro-symbolic approach where we use etp as an auxiliary task and jointly learn it with the primary task of el. both the primary and auxiliary tasks are classification tasks for a given mention m except that: (i) the corresponding classes are different -entities for the primary task and types for the auxiliary task, and (ii) auxiliary task is a hmlc task. in our modeling, we make a simplified assumption that the probability of entity e being the gold entity of a mention m is conditionally independent of the probability of any type t j being the valid type of the gold entity. in other words as shown in figure 3 , we use bert to obtain vector representation for both mention m and the entity e (as described in wu et al. ( 2020 )). parameters of these bert models are denoted by θ m and θ e , respectively. thus, we can say v @xmath6, θ m ) and v @xmath7, θ e ). neuro-symbolic model for p(t j | m): in practice, kb's organize entity types in the form of a logical hierarchy as shown in figure 2 . in such a hierarchy, if we traverse along a path from leaf node to the root, e.g. @xmath8, the corresponding types become coarse grained. to ensure this logical consistency property of the type hierarchy, we require that our proposed model satisfies the following path monotonicity property. we ensure this by exploiting gödel or łukasiewicz t-norm @xcite used in the field of fuzzy logic @xcite . as per these t-norms, if t j is an internal node in the hierarchy then we compute its probability purely in a symbolic fashion by using the probabilities assigned to its children nodes. we use logical or formula (given below) of the gödel (equation 2a) or łukasiewicz (equation 2b) t-norm for this purpose. c t j in these formulas denotes the set of children nodes for t j . for above model to work, we still need to address the issue of computing type probability for each leaf node t j in the hierarchy. we achieve this via a neural model where we obtain vector representations v t j for each leaf node by using a type network. the type network comprises an initial embedding vector for each type class and an optional linear layer followed by non-linearity. the parameters of this type network are denoted by θ t and hence, we have v t @xmath9, θ t ). type probability of each leaf node is computed using: because of equations ( 4 ) and (3), we call this model as neuro-symbolic. while training the above model, we need to prepare our data so as to ensure a property called upward closure of entity types. ## ethical considerations any zel technique, like ours, that advances the sota with less training data is a boon for applications such as kb question answering, document understanding, dialogue systems, etc. our solution, however, comes with its own limitations and risks as follows. 1) our assumption of entity types coming from the same hierarchy for both train and test domains is not always true in practice. moreover, if there is no entity type information available in the first place, then our approach can not even be used. we use publicly available datasets for training and testing our models. the wiki blink and conll-yago datasets are available under the mit and cc-by-3.0 licenses respectively. for the other datasets mentioned in ledell wu. 2020. blink. https://github. com/facebookresearch/blink . ledell wu, fabio petroni, martin josifoski, sebastian riedel, and luke zettlemoyer. 2020. scalable zeroshot entity linking with dense entity retrieval. in emnlp, pages 6397-6407. ikuya yamada, hiroyuki shindo, hideaki takeda, and yoshiyasu takefuji. 2016. joint learning of the embedding of words and entities for named entity disambiguation. in signll, pages 250-259.