Revisiting Sparse Retrieval for Few-shot Entity Linking (2023)

Abstract: Entity linking aims to link ambiguous mentions to their corresponding entities in a knowledge base. One of the key challenges comes from insufficient labeled data for specific domains. Although dense retrievers have achieved excellent performance on several benchmarks, their performance decreases significantly when only a limited amount of in-domain labeled data is available. In such few-shot setting, we revisit the sparse retrieval method, and propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains, showing the effectiveness of keyword-enhanced sparse retrieval.

## introduction entity linking (el) aligns entity mentions in documents with the corresponding entities in a knowledge base (kb), which is a crucial component of information extraction @xcite . typically, el systems follow a "retrieve and rerank" pipeline @xcite : candidate retrieval, where a small set of candidates are efficiently retrieved from a large number of entities, and candidate ranking, where candidates are ranked to find the most probable one. with the rapid development of pre-trained language models (plm), el has witnessed a radical paradigm shift towards dense retrieval (ma et al., 2021; sun et al., * both authors contributed equally to this work. although the bi-encoder has shown strong indomain performance, they typically require training on sufficient labeled data to perform well in a specific domain @xcite . however, such labeled data may be limited or expensive in new and specialized domains. as reported by @xcite , in the absence of sufficient training data, the recall of bi-encoder significantly decreases, performing even worse than unsupervised sparse retrievers (e.g., bm25 @xcite ). motivated by the above finding, in this work, instead of resorting to dense representations in the "semantic space", we go back to the "lexical space" and explore plm-augmented sparse representations for few-shot el. when applying bm25 to the first-stage retrieval, previous work only employs the mention string as query @xcite . however, this can be insufficient due to under-specified mentions (e.g., "his embodiments" in figure 1 ), requiring additional context to formulate a more comprehensive query. while an intuitive solution could be incorporating all context into the query, this approach is suboptimal for two reasons: (i) it does not create mentionspecific queries, i.e., for all mentions in a document, their queries and retrieved entities are the same; (ii) it may introduce noise into queries @xcite , since not all words in the context are necessary for understanding the mention. this paper introduces a keyword extractor to denoise the context for bm25 retrieval. the extraction is modeled as binary token classification, identifying which tokens in context should be added to the mention string. we employ a discriminative plm, specifically electra @xcite , as the basis of our keyword extractor, and leverage its discriminator head to predict a score for each token. the top-k tokens with the highest scores are then added into the final query. training the extractor requires binary labels to distinguish whether a token is a keyword. the keywords are expected to be related to the mention and necessary for understanding it. we propose a distant supervision method to automatically generate training data, based on overlapping words between the mention context and the description of the corresponding entity. these overlapping words are then ranked based on their bm25 score, which measures the relevance of the word to the entity description. we select the top-k words with the highest scores as keywords. following the few-shot experimental setting in @xcite , we evaluate our approach on the zeshel dataset @xcite . the results highlight our approach's superior performance across all test domains, with an average of 15.07% recall@64 improvement over the best dense retriever result. in addition to its superior performance, our method inherits the desirable properties of sparse representations such as efficiency of inverted indexes and interpretability. the contributions of this work are threefold: ## related work entity linking (el) bridges the gap between knowledge and downstream tasks @xcite @xcite . recent entity linking (el) methods follow a two-stage "retrieve and re-rank" approach (wu et al., 2020), and our work focuses on the first-stage retrieval. traditional retrieval systems based on lexical matching struggle with vocabulary mismatch issues @xcite . with the advent of pre-trained language models, the bi-encoder has largely mitigated this problem, but its effectiveness depends on extensive annotation data @xcite . without sufficient labeled data in specific domains, the recall of the bi-encoder drastically decreases @xcite , hence the need for further research about few-shot el. few-shot entity linking is recently proposed by @xcite . they split the data in a specific domain of zeshel @xcite and provide few examples to train. they address the data scarcity problem with synthetic mention-entity pairs, and design a meta-learning mechanism to assign different weights to synthetic data. with this training strategy, the performance of bi-encoder is better than simply training with insufficient data. however, it still cannot surpass the performance of the unsupervised sparse retrieval method (i.e., bm25 in ## methodology when applying bm25 to the first-stage retrieval of el, previous work directly uses mention string as query. we propose a keyword extractor to denoise the context, and add the extracted keywords into the query. then the expanded query is used for bm25 retrieval. in section 3.1, we introduce our distant supervision method to generate keyword labels. in section 3.2, we present the architecture, training and inference of our keyword extractor. preprocessing. bm25 implementation generally comes with stopword removal @xcite . in this work, we remove the words which occur in more than 20% of entities' description documents, resulting in removal of uninformative words (such as "we", "the", etc.). ## conclusion in this work, we focus on the first-stage retrieval in few-shot entity linking (el). given the significant performance drop in dense retrievers for few-shot scenarios, we revisit the sparse retrieval method and enhance its performance with a keyword extractor. we also propose a distant supervision ap-proach for automated training data generation. our extensive experiments demonstrate our method's superior performance, with case studies showing denoised context and mention-related keywords. future work may focus on more plm-augmented sparse retrieval solutions for el, expanding the query to words outside the context.