Towards Better Entity Linking with Multi-View Enhanced Distillation (2023)

Abstract: Dense retrieval is widely used for entity linking to retrieve entities from large-scale knowledge bases. Mainstream techniques are based on a dual-encoder framework, which encodes mentions and entities independently and calculates their relevances via rough interaction metrics, resulting in difficulty in explicitly modeling multiple mention-relevant parts within entities to match divergent mentions. Aiming at learning entity representations that can match divergent mentions, this paper proposes a Multi-View Enhanced Distillation (MVD) framework, which can effectively transfer knowledge of multiple fine-grained and mention-relevant parts within entities from cross-encoders to dual-encoders. Each entity is split into multiple views to avoid irrelevant information being over-squashed into the mention-relevant view. We further design cross-alignment and self-alignment mechanisms for this framework to facilitate fine-grained knowledge distillation from the teacher model to the student model. Meanwhile, we reserve a global-view that embeds the entity as a whole to prevent dispersal of uniform information. Experiments show our method achieves state-of-the-art performance on several entity linking benchmarks.

## introduction entity linking (el) serves as a fundamental task in natural language processing (nlp), connecting mentions within unstructured contexts to their corresponding entities in a knowledge base (kb). el usually provides the entity-related data foundation for various tasks, such as kbqa @xcite , knowledge-based language models @xcite and information retrieval @xcite . most el systems consist of two stages: entity retrieval (candidate generation), which retrieves figure 1 : the illustration of two types of entities. mentions in contexts are in bold, key information in entities is highlighted in color. the information in the first type of entity is relatively consistent and can be matched with a corresponding mention. in contrast, the second type of entity contains diverse and sparsely distributed information, can match with divergent mentions. a small set of candidate entities corresponding to mentions from a large-scale kb with low latency, and entity ranking (entity disambiguation), which ranks those candidates using a more accurate model to select the best match as the target entity. this paper focuses on the entity retrieval task, which poses a significant challenge due to the need to retrieve targets from a large-scale kb. moreover, the performance of entity retrieval is crucial for el systems, as any recall errors in the initial stage can have a significant impact on the performance of the latter ranking stage @xcite . recent advancements in pre-trained language models (plms) @xcite have led to the widespread use of dense retrieval technology for large-scale entity retrieval @xcite . this approach typically adopts a dual-encoder architecture that embeds the textual content of mentions and entities independently into fixed-dimensional vectors @xcite to calculate their relevance scores using a lightweight interaction metric (e.g., dot-product). this allows for pre-computing the entity embeddings, enabling entities to be retrieved through various fast nearest neighbor search techniques @xcite . the primary challenge in modeling relevance between an entity and its corresponding mentions lies in explicitly capturing the mention-relevant parts within the entity. by analyzing the diversity of intra-information within the textual contents of entities, we identify two distinct types of entities, as illustrated in figure 1 . entities with uniform information can be effectively represented by the dual-encoder; however, due to its single-vector representation and coarse-grained interaction metric, this framework may struggle with entities containing divergent and sparsely distributed information. to alleviate the issue, existing methods construct multi-vector entity representations from different perspectives @xcite @xcite . despite these efforts, all these methods rely on coarse-grained entity-level labels for training and lack the necessary supervised signals to select the most relevant representation for a specific mention from multiple entity vectors. as a result, their capability to effectively capture multiple fine-grained aspects of an entity and accurately match mentions with varying contexts is significantly hampered, ultimately leading to suboptimal performance in dense entity retrieval. in order to obtain fine-grained entity representations capable of matching divergent mentions, we propose a novel multi-view enhanced distillation (mvd) framework. mvd effectively transfers knowledge of multiple fine-grained and mentionrelevant parts within entities from cross-encoders to dual-encoders. by jointly encoding the entity and its corresponding mentions, cross-encoders enable the explicit capture of mention-relevant components within the entity, thereby facilitating the learning of fine-grained elements of the entity through more accurate soft-labels. to achieve this, our framework constructs the same multi-view representation for both modules by splitting the textual information of entities into multiple fine-grained views. this approach prevents irrelevant information from being over-squashed into the mentionrelevant view, which is selected based on the results of cross-encoders. we further design cross-alignment and self-alignment mechanisms for our framework to separately align the original entity-level and finegrained view-level scoring distributions, thereby facilitating fine-grained knowledge transfer from the teacher model to the student model. motivated by prior works @xcite @xcite , mvd jointly optimizes both modules and employs an effective hard negative mining technique to facilitate transferring of hard-to-train knowledge in distillation. meanwhile, we reserve a global-view that embeds the entity as a whole to prevent dispersal of uniform information and better represent the first type of entities in figure 1 . through extensive experiments on several entity linking benchmarks, including zeshel, aida-b, msnbc, and wned-cweb, our method demonstrates superior performance over existing approaches. the results highlight the effectiveness of mvd in capturing fine-grained entity representations and matching divergent mentions, which significantly improves entity retrieval performance and facilitates overall el performance by retrieving high-quality candidates for the ranking stage. ## related work to accurately and efficiently acquire target entities from large-scale kbs, the majority of el systems are designed in two stages: entity retrieval and entity ranking. for entity retrieval, prior approaches typically rely on simple methods like frequency information @xcite , alias tables @xcite and sparse-based models @xcite to retrieve a small set of candidate entities with low latency. for the ranking stage, neural networks had been widely used for calculating the relevance score between mentions and entities @xcite @xcite . recently, with the development of plms (kenton and toutanova, 2019; @xcite , plmbased models have been widely used for both stages of el. @xcite and @xcite utilize the cross-encoder architecture that jointly encodes mentions and entities to rank candidates, @xcite employs the dualencoder architecture for separately encoding mentions and entities into high-dimensional vectors for entity retrieval. blink @xcite improves overall el performance by incorporating both architectures in its retrieve-then-rank pipeline, making it a strong baseline for the task. gerene @xcite directly generates entity names through an auto-regressive approach. to further improve the retrieval performance, various methods have been proposed. @xcite and @xcite demonstrate the effectiveness of hard negatives in enhancing retrieval performance. @xcite and ger @xcite construct mention/entity centralized graph to learn the fine-grained entity representations. however, being limited to the single vector representation, these methods may struggle with entities that have multiple and sparsely distributed information. although @xcite and muver @xcite construct multi-view entity representations and select the optimal view to calculate the relevance score with the mention, they still rely on the same entity-level supervised signal to optimize the scores of different views within the entity, which limit the capacity of matching with divergent mentions. in contrast to existing methods, mvd is primarily built upon the knowledge distillation technique @xcite , aiming to acquire finegrained entity representations from cross-encoders to handle diverse mentions. to facilitate finegrained knowledge transfer of multiple mentionrelevant parts, mvd splits the entity into multiple views to avoid irrelevant information being squashed into the mention-relevant view, which is selected by the more accurate teacher model. this framework further incorporates cross-alignment and self-alignment mechanisms to learn mentionrelevant view representation from both original entity-level and fine-grained view-level scoring distributions, these distributions are derived from the soft-labels generated by the cross-encoders. ## conclusion in this paper, we propose a novel multi-view enhanced distillation framework for dense entity retrieval. our framework enables better representation of entities through multi-grained views, and by using hard negatives as information car-