A Transformational Biencoder with In-Domain Negative Sampling for Zero-Shot Entity Linking (2022)

Abstract: Recent interest in entity linking has focused in the zero-shot scenario, where at test time the entity mention to be labelled is never seen during training, or may belong to a different domain from the source domain. Current work leverage pre-trained BERT with the implicit assumption that it bridges the gap between the source and target domain distributions. However, fine-tuned BERT has a considerable underperformance at zero-shot when applied in a different domain. We solve this problem by proposing a Transformational Biencoder that incorporates a transformation into BERT to perform a zero-shot transfer from the source domain during training. As like previous work, we rely on negative entities to encourage our model to discriminate the golden entities during training. To generate these negative entities, we propose a simple but effective strategy that takes the domain of the golden entity into perspective. Our experimental results on the benchmark dataset Zeshel show effectiveness of our approach and achieve new state-of-the-art.

## introduction entity linking (el) is an important task in natural language processing (nlp), which seeks to align entity mentions in a document to their referent entity in a knowledge base such as wikipedia. el has received widespread attention due to its application in a variety of tasks, including information extraction @xcite , knowledge base population @xcite , content analysis @xcite , etc. there has been great achievement in building el systems, however, majority of proposed works @xcite are built on the assumption that the entity set is shared among the train and test sets. in many practical cases, however, the train and test sets may come from different domain distributions. this potentially creates disjoint entity sets across * corresponding author the different domains, highlighting the importance of zero-shot el @xcite . zero-shot el aims to label mentions in the test set that have never been seen during training. a line of works have proposed zero-shot learning techniques for entity linking @xcite . the common paradigm in these works is to link labelled mentions in a document to entities in well structured knowledge bases. despite their promising successes, labelled data is typically expensive to produce or are not easily obtained for some specialized domains such as the legal domain. to enable research in this problem, @xcite developed the zeshel dataset which contains a diverse range of specialized domains, in which mentions and entities have rich textual content. without adopting resources (e.g., structured knowledge base) or assumptions (e.g., labelled mentions, a shared entity set), they expand the scope of zero-shot el to promote the generalizability of el system on unseen domains. so far, only few works have been proposed @xcite @xcite , where bert @xcite is found to be the notable encoder. majority of these works are devoted to retrieving candidate entities since this is essential to candidate entity ranking for el systems. @xcite achieved state-of-the-art for candidate entity generation by employing the biencoder @xcite to encode mentions and entities, an expressive sum-of-max (som) score function @xcite to compute their relevance scores, and by optimizing with hard negative entities. to generate hard negatives, @xcite use the score function to rank all entities across domains and select top-k entities. although the biencoder has been successfully applied, it faces a fundamental weakness which limits its ability to successfully achieve zero-shot transfer for the task. specifically, the fine-tuned bert as used in the biencoder has been shown to degrade substantially on zero-shot transfer if there is a shift between source and target domains @xcite . another problem is the high dimensionality of entities/mentions which poses a challenge for complex scorers (e.g., sum-of-max). som requires o(n 2 ) in running time and storage complexity, which makes it almost infeasible when sampling hard negatives. as el systems may have millions of entities, a more scalable solution is needed. in this paper, we overcome the aforementioned weakness of the biencoder by integrating it with a learnable transformation, developing an transformational bi-encoder (t-biencoder). as the name suggests, we focus on learning a transformation in the bert architecture to achieve zero-shot transfer from a source domain during training. with regard to the efficiency of the model and the optimization strategy, we recognize the performance advantages of sampling with hard negatives over sampling randomly. hard negative sampling works because generated entities are semantically different and close to the golden entity in the embedding space. this encourages discrimination between the golden entity and negative entities. we hypothesize that, the condition of lexical similarity between the entity and golden entity will lead to harder negatives and better optimization. that is, we propose to random (or hard) sample in-domain of the golden entity. by sampling in-domain, the entity set in which we sample from is relatively small, making it a more efficient alternative to sampling across domains. extensive experiments show the effectiveness of our approach as against prior works. our contributions can be summarized as follows: ## related works to enable progress on the zero-shot entity linking task, @xcite propose to use the naive baseline method bm25 @xcite to measure the relevance score of mention-entity pairs. following this work, a number of methods operating on zeshel have been proposed @xcite @xcite , where bert @xcite ) is found to be the notable encoder. this is not surprising as bert has shown to produce state-of-the-art results in several nlp tasks. among the existing works, @xcite propose a biencoder architecture where two independent bert encoders are employed to encode the textual descriptions of mentions and entities. a dot product is used as the scorer, referred to as dual by @xcite . the biencoder provides a strong baseline for the task due to the expressiveness of bert. @xcite adapts a bert architecture that repeats the position embedding to solve the long-range modeling problem in entity textual descriptions. @xcite propose a bidirectional multi-paragraph reading model that leverages more textual information to enhance text understanding capability. @xcite adopt the biencoder framework but employ a more expressive sum-of-max score function @xcite to measure the relevance between a mention and entity, achieving state-of-the-art results on the task. the majority of these works leverage negative entities during optimization. they also have an implicit assumption that bert is sufficient for zero-shot transfer. unlike previous work, we adapt the bert encoder with a transformation to improve zero-shot transfer. we also consider to sample negative entities in-domain of the golden entity rather than across all domains to improve optimization and efficiency. ## methodology in this section, we describe transformational biencoder (t-biencoder), our proposed method for zero-shot el. first, we formally present the task definition in section 3.1. next, in section 3.2 we introduce the biencoder @xcite . then, we describe our adaptation of the biencoder to develop t-biencoder in section 3.3. finally, we close by discussing our negative sampling strategy in section 3.4. ## conclusion we introduced a transformational biencoder (t-biencoder) that builds upon the recent proposed biencoder @xcite to solve the problem of zero-shot entity linking. our work shows how to explicitly improve the zero-shot transfer capability of the biencoder for el. we hypothesized that negative samples drawn in-domain of the goldenentity results in better optimization. our experimental analysis demonstrates that this assumption holds, where we see the benefits of our negative sampling strategy as well as the learned transformation. our results show strong improvements for the task, in both effectiveness and efficiency. we envision that the t-biencoder can further be improved by learning a transformation for each domain in the train set, since a single transformation may not effectively capture the relationship among different specialized domains. we leave this for future work.