R e F in ED : An Efficient Zero-shot-capable Approach to End-to-End Entity Linking (2022)

Abstract: We introduce ReFinED, an efficient end-to-end entity linking model which uses fine-grained entity types and entity descriptions to perform linking. The model performs mention detection, fine-grained entity typing, and entity disambiguation for all mentions within a document in a single forward pass, making it more than 60 times faster than competitive existing approaches. ReFinED also surpasses state-of-the-art performance on standard entity linking datasets by an average of 3.7 F1. The model is capable of generalising to large-scale knowledge bases such as Wikidata (which has 15 times more entities than Wikipedia) and of zero-shot entity linking. The combination of speed, accuracy and scale makes ReFinED an effective and cost-efficient system for extracting entities from web-scale datasets, for which the model has been successfully deployed.

## introduction entity linking (el) is the task of recognising mentions of entities in unstructured text documents and linking them to the corresponding entities in a knowledge base (kb), such as wikidata. el is commonly a first stage in systems for question answering @xcite , automated kb population @xcite , and relation extraction @xcite . currently, el systems use deep learning methods to learn representations for entities and mentions @xcite . initial techniques learned representations from text alone, which relied on entities appearing in similar contexts in the training data and meant models were only able to link mentions to entities that appeared in the training data. this is problematic both as kbs are continuously growing, and as it is infeasible to build an el dataset containing all entities in a large kb (such as wikidata with over 90 million entities). the largest public el dataset is wikipedia (using internal hyperlinks as labels), which covers just 3% of the entities in wikidata. recent models addressed this problem by producing entity representations from a subset of kb information, e.g., entity descriptions @xcite or fine-grained entity types @xcite , allowing linking to entities not present in the training data or added to the kb after training; termed "zero-shot" in the el literature. 1 however, existing zero-shot-capable el approaches are an order of magnitude more computationally expensive than non-zero-shot models @xcite as they either require numerous entity types @xcite , multiple forward passes of a large-scale model to encode mentions and descriptions @xcite , or regeneration of the input text autoregressively @xcite . this makes large-scale processing expensive and thus makes it difficult to benefit from many advantages of zero-shot el, e.g. the ability to keep up-to-date with new or updated kbs. in this paper, we propose an efficient end-toend zero-shot-capable el model, refined 2 , which uses fine-grained entity types and entity descriptions to perform entity linking or entity disambiguation (ed; where entity mentions are given). we show that combining information from entity types and descriptions in a simple transformer-based encoder yields performance which is stronger than more complex architectures, surpassing state-ofthe-art (sota) on 4 ed datasets and 5 el datasets, and improving overall el performance by 3.7 f1 points on average across 8 datasets. importantly, refined performs mention detection, fine-grained entity typing, and entity disambiguation for all men-tions within a document in a single forward pass, making it comparable in terms of inference speed to non-zero-shot models. it is 6 times faster than the most efficient zero-shot-capable baseline (which has 9 f1 points lower performance), and more than 60 times faster than more accurate systems (which come within 3 f1 points of refined's average ed performance). as opposed to previous el models which primarily use wikipedia as the target kb, refined targets wikidata, which enables it to link to 15 times more entities. this is because prior work uses information (e.g. titles, categories, first sentences) from wikipedia to perform linking. it is unclear whether prior work could be expanded to wikidata without a drop in performance because entity descriptions are less informative and there are fewer types per entity @xcite . the combination of high accuracy, scalability (with respect to kb size) and fast inference speed makes refined a strong choice for a "web-scale" foot_2 el system, in which cost scales approximately linearly with inference speed. we have successfully deployed refined to production in a real-world application and share the lessons learned in section 6. our contributions are as follows: ## related work single architecture for entity linking el consists of two main tasks, mention detection (md) and ed. md involves recognising mentions of entities in text, and ed assigns a kb entity to each mention. we follow @xcite in training a joint model for md and ed. ## deployment details we have successfully deployed the refined el model in a real-world application, the aim of which is to populate a kb by extracting facts from unstructured text found on web pages with high precision. the application requires running refined on a billion web pages (in which we link 25 billion mentions) multiple times per year. the scale of this deployment highlighted a number of learning points. firstly, the entity linking model must be computationally efficient. the inference speed of re-fined allows the processing of the billion web pages in 27k machine hours (2 days using 500 instances), on machines with a single t4 gpu. given availability of cloud compute, the cost of processing the same documents with the models evaluated in section 5.3 would scale approximately linearly with their inference speeds. that is, the blink bi-encoder would require 3000 instances for 2 days, or 500 instances for 12 days, implying a roughly 6-fold increase in cost. secondly, the scale of the number of pages also brings with it diversity of domains, meaning the model benefits from linking to a large catalogue of entities (over 90 million) -including zero-shot entities. thirdly, we observed that deploying an end-toend self-contained el model is easier to horizontally scale and has a lower operational cost than deploying multiple systems for each subcomponent (such as candidate generation). finally, in real-world data, unlike in ed datasets, there are a large number of cases where the correct entity does not exist in the kb. this meant that we had to train the model on examples where the correct entity was not in the candidate list to reduce overprediction. ## conclusion we propose a scalable end-to-end el model which uses entity types and entity descriptions to perform linking. our model achieves sota results for both ed (+0.7 f1 points on average across 6 datasets) and el (+3.7 f1 points on average across 8 datasets) while being 60 times faster than comparatively accurate baselines. we demonstrate our approach scales well to a kb (wikidata) 15 times larger than wikipedia while maintaining competitive performance. the combination of accuracy, speed and scale means the system is capable of being deployed to extract entities from web-scale datasets with higher accuracy and an order of magnitude lower cost than existing approaches.