Zero-shot Entity Linking with Efficient Long Range Sequence Modeling (2020)

Abstract: This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to expand the long-range sequence modeling. Unlike many previous methods, our method does not require expensive pre-training of BERT with long position embeddings. Instead, we propose an efficient position embeddings initialization method called Embedding-repeat, which initializes larger position embeddings based on BERT-Base. On the zero-shot entity linking dataset, our method improves the STOA from 76.06% to 79.08%, and for its long data, the corresponding improvement is from 74.57% to 82.14%. Our experiments suggest the effectiveness of long-range sequence modeling without retraining the BERT model.

## introduction entity linking (el) is the task of grounding entity mentions by linking them to entries in a given database or dictionary of entities. traditional el approaches often assume that entities linked at the test time are present in the training set. nevertheless, many real-world applications prefer the zeroshot setting, where there is no external knowledge and a short text description provides the only information we have for each entity @xcite . for zero-shot entity linking @xcite , it is crucial to consider the context of entity description and mention, so that the system can generalize to unseen entities. however, most of the bert-based models are based on a context window with 512 tokens, limited to capturing the long-range of context. this paper defines a model's effective-reading-length (er-length) as the total length of the mention contexts and entity description that it can read. figure 1 demonstrates an example where long erlengths are more preferred than short ones. many existing methods that can be used to expand erlength @xcite , however, often need to completely re-do pre-training with the masked language modeling objective on the vast general corpus (like wikipedia), which is not only very expensive but also impossible in many scenarios. this paper proposes a practical way, embeddings-repeat, to expand bert's er-length by initializing larger position embeddings, allowing reading all information in the context. note our method differs from previous works since it can directly use the larger position embeddings initialized from bert-base to do fine-tuning on downstream tasks without any retraining. extensive experiments are conducted to compare different ways of expanding erlength, and the results show that embeddings-repeat can robustly improve performance. most importantly, we improve the accuracy from 76.06% to 79.08% in wikia's zero-shot el dataset, from 74.57% to 82.14% for its long data. since our method is effective and easy to implement, we expect our method will be useful for other downstream nlp tasks. ## conclusions and future work we propose an efficient position embeddings initialization method called embeddings-repeat, which initializes larger position embeddings based on bert models. for the zero-shot entity linking task, our method improves the sota from 76.06% to 79.08% on its dataset. our experiments suggest the effectiveness of increasing erlength as large as possible (e.g., the length of the longest data in the el experiments). our future work will be to extend our methods to other nlp tasks.