A Read-and-Select Framework for Zero-shot Entity Linking (2023)

Abstract: Zero-shot entity linking (EL) aims at aligning entity mentions to unseen entities to challenge the generalization ability. Previous methods largely focus on the candidate retrieval stage and ignore the essential candidate ranking stage, which disambiguates among entities and makes the final linking prediction. In this paper, we propose a read-and-select (ReS) framework by modeling the main components of entity disambiguation, i.e., mention-entity matching and cross-entity comparison. First, for each candidate, the reading module leverages mention context to output mention-aware entity representations, enabling mention-entity matching. Then, in the selecting module, we frame the choice of candidates as a sequence labeling problem, and all candidate representations are fused together to enable cross-entity comparison. Our method achieves the state-of-the-art performance on the established zero-shot EL dataset ZESHEL with a 2.55% micro-average accuracy gain, with no need for laborious multi-phase pre-training used in most of the previous work, showing the effectiveness of both mention-entity and cross-entity interaction.

## introduction entity linking (el) is the task of aligning entity mentions in a document to their referent entity in a knowledge base (kb). considering the example in figure 1 , given the sentence "professor henry jones sr is indiana jones's father . . . ", the mention indiana jones should be linked to the entity indiana jones (minifigure). as a bridge that connects mentions in unstructured text and entities in structured kbs, el plays a key role in a variety of tasks, including kb population @xcite , semantic search @xcite , summarization @xcite , etc. previous el systems have achieved high performance when a large set of mentions and target entities is available for training. however, such labeled data may be limited in some specialized domains, such as biomedical and legal cases. therefore, we need to develop el systems that can generalize to unseen entity sets across different domains, highlighting the importance of zero-shot el @xcite . zero-shot el task is proposed by @xcite along with the zeshel dataset. there are two key properties in the zero-shot setting: (1) labeled mentions for the target domain are unavailable; (2) mentions and entities are only defined through textual descriptions, without additional resources (e.g., alias tables and frequency) which many previous el systems rely on. most zero-shot el systems consist of two stages @xcite : (1) candidate retrieval, where a small set of candi-dates are efficiently retrieved from kb; (2) candidate ranking, where candidates are ranked to find the most probable one. so far, many methods have been proposed the first-stage retrieval @xcite @xcite . however, the candidate ranking stage, which makes the final prediction, is important yet less discussed. this paper focuses on the candidate ranking stage. the cross-encoder, introduced by @xcite is a widely-used candidate ranking model that facilitates mention-entity matching. it takes as input the concatenation of the mention context and each entity description, and outputs whether or not the mention refers to the concatenated entity. however, since each entity is encoded independently with the mention, the cross-encoder lacks the capability to compare all candidates at once and select the most appropriate one. figure 1 shows an error case of the cross-encoder. when ranking candidates that share a high degree of lexical similarity with the mention, the cross-encoder tends to assign high scores to both candidates. it may even incorrectly rank an entity higher than the correct one. given the mention "indiana jones", the wrong indiana jones (theme) is rated higher than the gold indiana jones (minifigure)), highlighting the necessity for cross-entity comparison. motivated by the above example, we propose a read-and-select (res) framework, explicitly modeling both mention-entity matching and cross-entity comparison. the res framework consists of two modules: (1) in the reading module, we prepend prefix tokens @xcite to the concatenation of entity description and mention context, and obtain mention-aware entity representations, enabling mention-entity matching. (2) in the selecting module, the choice of candidates is framed as a sequence labeling problem, and all entity representations are fused together to enable cross-entity comparison. unlike previous pipeline approaches that contain independently trained models @xcite , our reading module and selecting module share parameters, and the whole model is trained end-to-end. we evaluate our approach on the widely-used zeshel dataset @xcite . based on the experimental results, our res framework achieves the state-of-the-art performance with a 2.55% micro-average accuracy improvement. besides its performance advantage, res comes with another benefit: it does not need laborious multi-phase pre-training used in most of previous work, showing its strong generalization ability. further ablation study and case study demonstrate the effectiveness of cross-entity interaction. our contributions are summarized as follows: entity linking (el) bridges the gap between knowledge and downstream tasks @xcite @xcite . there have been great achievements in building general el systems with wikipedia as the corresponding knowledge base (kb). among them, the bi-encoder has been particularly successful: two encoders are trained to learn representations in a shared space for mentions and entities @xcite . however, the actual disambiguation is only expressed via a dot product of independently computed vectors, neglecting mention-entity interaction. to this end, @xcite attempt to add a candidate ranking stage via stacking a cross-encoder after the biencoder. recent years have seen more transformerbased ranking models obtaining promising performance gains @xcite , but most of proposed works are built on the assumption that the entity set is shared among the train and test sets @xcite . in many practical cases, the train and test sets may come from different domain distributions and disjoint entity sets, emphasizing the necessity of zero-shot el. zero-shot el has recently attracted great interest from researchers. to enable progress on this task, @xcite propose a dataset called zeshel, where mentions in the test set are linked to unseen entities without in-domain labeled data. following this work, a number of methods that operate on zeshel have been proposed, but they only focus on the retrieval stage by mention context entity 1 entity 2 prefix prefix entity 3 entity n prefix prefix mention context mention context encoder mention context mention context reading module encoder encoder ... concat all encoder encoder ... selecting module ... encoder mention tokens candidate tokens ✔✔✔ ❌ ❌ ❌ ... ❌ ❌ ❌ ❌ ❌ ❌ sequence labels modifying the bi-encoder architecture, such as incorporating triplets in knowledge graphs @xcite , adding ultra-fine entity typing supervision @xcite , creating multi-view entity representations @xcite , and mining hard negatives for training @xcite . none of the above works considers the candidate ranking stage. however, the ranking stage is essential as it makes the final linking prediction among candidates, hence the need for further discussion about the ranking stage. the few works focusing on the ranking stage all follow the cross-encoder formulation. further improvements include extending to long documents @xcite and different pre-training strategies @xcite . in contrast, here we propose a sequence labeling formulation, where a model receives the mention, its context and all mention-aware entity representations as input, and marks out the the corresponding entity tokens. note that this differs from the cross-encoder where each entity is independently encoded with mention context. with our framework, mention-entity matching and crossentity comparison can be explicitly modeled. ## methodology in this section, we describe the read-and-select (res) framework, our proposed method for zeroshot el. as shown in figure 2 , the framework consists of the reading module and the selecting module, enabling mention-entity and cross-entity interaction respectively. first, we formally present the task formulation in section 3.1. next, we introduce our reading module in section 3.2. finally, we introduce our selecting module and end-to-end training objective in section 3.3. ## experiment in this section, we assess the effectiveness of res on candidate ranking of zero-shot el. we first introduce the experimental setup we consider, i.e., the data (section 4.1), evaluation metric (section 4.2), technical details (section 4.3), and baselines in comparison (section 4.4). then we present the main result of res, its category-specific results, and the ablation study in section 4.5. more analysis about the cross-entity comparison can be found in the impact of candidate number (section 4.6) and case study (section 4.7). ## conclusion in this work, we focus on the candidate ranking stage in zero-shot entity linking (el). to disambiguate better among entities which have high lexical similarity with the mention, we propose a readand-select (res) framework, which first produces mention-aware entity representations in the reading module, and then applies a sequence labeling paradigm on the fusion of all candidate representations in the selecting module, thus modeling both mention-entity and cross-entity interaction. experiments show that res achieves the state-of-the-art result on zeshel, no need of any laborious pretraining strategy, showing its effective generalization and disambiguation ability. future work may expand this framework to take longer contexts and descriptions into consideration.