Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2020)

Abstract: This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model, along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions. Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbor search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation. Our code and models are available at https://github.com/facebookresearch/BLINK .

## introduction scale is a key challenge for entity linking; there are millions of possible entities to consider for each mention. to efficiently filter or rank the candidates, existing methods use different sources of external information, including manually curated mention tables @xcite , incoming wikipedia link popularity @xcite , and gold wikipedia entity categories @xcite . in this paper, we show that bert-based models set new state-of-the-art performance levels * work done during internship with facebook. for large scale entity linking when used in a zero shot setup, where there is no external knowledge and a short text description provides the only information we have for each entity. we also present an extensive evaluation of the accuracy-speed tradeoff inherent to large pre-trained models, and show is possible to achieve very efficient linking with modest loss of accuracy. more specifically, we introduce a two stage approach for zero-shot linking (see figure 1 for an overview), based on fine-tuned bert architectures @xcite . in the first stage, we do retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions @xcite . each retrieved candidate is then examined more carefully with a cross-encoder that concatenates the mention and entity text, following @xcite . this overall approach is conceptually simple but highly effective, as we show through detailed experiments. our two-stage approach achieves a new state-ofthe-art result on tackbp-2010, with an over 30% relative error reduction. by simply reading the provided text descriptions, we are able to outperform previous methods that included many extra cues such as entity name dictionaries and link popularity. we also improve the state of the art on existing zero-shot benchmarks, including a nearly 6 point absolute gain on the recently introduced wikia corpus @xcite and more than 7 point absolute gain on wikilinksned unseen-mentions @xcite . finally, we do an extensive evaluation of the accuracy-speed trade-off inherent in our bi-and cross-encoder models. we show that the two stage methods scales well in a full wikipedia setting, by linking against all the 5.9m wikipedia entities for tackbp-2010, while still outperforming existing model with much smaller candidate sets. we ## related work we follow most recent work in studying entity linking with gold mentions. two recent results are most closely related to our work. @xcite proposed the zero-shot entity linking task. they use cross-encoders for entity ranking, but rely on traditional ir-techniques for candidate generation and did not evaluate on large scale benchmarks such as tackbp. @xcite show that dense embeddings work well for candidate generation, but they did not do pre-training and included external category labels in their bi-encoder architectures, limiting their linking to entities in wikipedia. our approach can be seen as generalizing both of these lines of work, and showing for the first time that pre-trained zero-shot architectures are both highly accurate and computationally efficient at scale. @xcite studied different architectures to use deep pre-trained bidirectional transformers and performed detailed comparison of three different architectures, namely bi-encoder, poly-encoder, cross-encoder on tasks of sentence selection in dialogues. inspired by their work, we use similar architectures to the problem of entity linking, and in addition, demonstrate that biencoder can be a strong model for retrieval. instead of using the poly-encoder as a trade-off between cross-encoder and bi-encoder, we propose to train a bi-encoder model with knowledge distillation @xcite from a cross-encoder model to further improve the biencoder's performances. ## methodology figure 1 shows our overall approach. the biencoder uses two independent bert transformers to encode model context/mention and entity into dense vectors, and each entity candidate is scored as the dot product of these vectors. the candidates retrieved by the bi-encoder are then passed to the cross-encoder for ranking. the cross-encoder encodes context/mention and entity in one transformer, and applies an additional linear layer to compute the final score for each pair. ## experiments in this section, we perform an empirical study of our model on three challenging datasets. ## qualitative analysis in the first example, we see that the bi-encoder mistakenly links "ronaldo" to the brazilian football player, while the cross-encoder is able to use context word "juventus" to disambiguate. in the second example, the cross-encoder is able to identify from context that the sentence is describing art instead of fiction, where the bi-encoder failed. in the third example, the bi-encoder is able to find the correct entity "ancient greek,"; where the crossencoder mistakenly links it to the entity "ancient greek philosophy," likely because that the word "philosophers" is in context. we observe that crossencoder is often better at utilizing context information than bi-encoder, but can sometimes make mistakes because of misleading context cues. ## conclusion we proposed a conceptually simple, scalable, and highly effective two stage approach for entity linking. we show that our bert-based model outperforms ir methods for entity retrieval, and achieved new state-of-the-art results on recently introduced zero-shot entity linking dataset, wik-ilinksned unseen-mentions dataset, and the more established tackbp-2010 benchmark, without any task-specific heuristics or external entity knowledge. we present evaluations of the accuracy-speed trade-off inherent to large pre-trained models, and show that it is possible to achieve efficient linking with modest loss of accuracy. finally, we show that knowledge distillation can further improve biencoder model performance. future work includes: