Entity Embedding Completion for Wide-Coverage Entity Disambiguation (2022)

Abstract: Entity disambiguation (ED) is typically solved by learning to classify a given mention into one of the entities in the model’s entity vocabulary by referring to their embeddings. However, this approach cannot address mentions of entities that are not covered by the entity vocabulary. Aiming to enhance the applicability of ED models, we propose a method of extending a state-of-the-art ED model by dynamically computing embeddings of out-of-vocabulary entities. Specifically, our method computes embeddings from entity descriptions and mention contexts. Experiments with standard benchmark datasets show that the extended model performs comparable to or better than existing models whose entity embeddings are trained for all candidate entities as well as embedding-free models. We release our source code and model checkpoints at https://github.com/studio-ousia/steel.

## introduction entity disambiguation (ed) ( § 2) has been studied as a method to organize information about real-world entities in text. consequently, ed must be capable of handling diverse entities in various domains. existing state-of-the-art ed models are based on huge pretrained language models (plms) @xcite , and are trained to match a given mention with its candidate entities via embeddings @xcite @xcite . despite the advances in performance, these embeddingbased ed models do not address out-of-vocabulary entities that are absent from the training data. even worse, since maintaining millions of entity embeddings demands a huge memory space in training, the models are often trained only for candidate entities ( § 2) that appear in testing @xcite . 1 to address the above problems, researchers have explored ed models without entity embeddings. @xcite solves ed as autoregressive generation, which decodes the entity name token-bytoken. @xcite formulates ed as span extraction, which selects a correct entity name from input concatenating mention contexts and candidate entity names. although these approaches allow a model to handle millions of candidate entities, they still have difficulties in handling newlyemerged entities that are absent from the training data and evolved entities whose attributes have changed over time. @xcite and @xcite proposed zero-shot methods to handle any candidate entities by reading their descriptions at testing. the performance of these zero-shot methods, however, is inferior to the aforementioned models, probably because these methods represent each entity using short entityrelated text instead of massive training examples used in the ed models. in this paper, we propose an adaptation method that enables entity embedding-based ed models, which are trained for limited entities, to handle the other rare and emerging entities by dynamically predicting their embeddings. in particular, we actually extend the architecture of the state-of-the-art transformer-based ed model developed in @xcite . we first train this model with an entity vocabulary consisting of only common entities ( § 4.1). next, we extend the model ( § 4.3) by using our bert-based encoders that dynamically predict entity embeddings from descriptions and relevant sentences ( § 4.2), thus enabling the model to handle the out-of-vocabulary entities. through experiments ( § 5) with standard benchmark datasets for ed (i.e., @xcite , msnbc, aquaint, ace2004, wned-cweb, and wned-wiki @xcite )), we verify that our extended model performs competitively to the original state-of-theart model, whose trained entity embeddings encompass all the candidate entities of the evaluation datasets ( § 5). we then perform an empirical analysis on the nature of our proposed encoders, and on reducing memory costs in the inference time ( § 6). our key contributions are: ## entity disambiguation task given a knowledge base (kb) such as wikipedia, and a text with spans of mentions, the ed model assigns each mention span to an entity in the kb. the task is similar to word sense disambiguation. since the number of entities can be more than millions, during the evaluation, the ed models additionally assume a small list of candidate entities for each mention, using a mention-entity dictionary derived from hyperlinks in wikipedia. following existing embedding-based ed models (e.g., yamada et al. (2022) ), we use the standard entity candidate set, kb-yago @xcite . ## related work several studies have proposed knowledge-intensive transformer-based ed models that have entity embeddings as parameters, such as weights of the classifier @xcite @xcite . despite the stateof-the-art in the benchmark, they cannot handle out-of-vocabulary entities and entities of changed properties, since these embeddings are typically trained via ed task with labeled corpora. in addition, as argued by @xcite , it is hard to integrate embeddings of all ever-increasing entities into the huge transformer-based models, especially to maintain their gradient values during training. @xcite therefore demands expensive tpu clusters, and some employ transductive settings to train embeddings for a small subset of entities containing all candidate entities used in the evaluation @xcite . before transformer becomes the dominant architecture for the ed task, knowledge-intensive ed models were developed @xcite @xcite . since these models require relatively few parameters to be learned (e.g., several linear transformation matrices @xcite ), loading and training embeddings of entire entities may not be a problem compared to the transformer-based models. however, they still do not address entities absent in or updated from the training phase. zero-shot systems, which represent entities as descriptions or contexts rather than embeddings, can convey textual information pertaining to novel or changing entities. however, a single entity description, used by @xcite and @xcite , has less information compared to embeddings learned from many examples. fitzgerald et al. (2021) embeds each mention context as keys to retrieve associated entities from the target mention contexts (query). however, they also suffers from less informative contexts per key embedding. 2 generation-or extraction-based approaches can reduce computational costs by not integrating entity-level knowledge such as entity embeddings; @xcite generates tokens of entity titles and @xcite selects the correct title from the concatenated titles of candidate entities. however, while they can generate or extract any entities, they cannot properly model novel and evolving entities since they are trained with labeled data based on a specific kb. moreover, the performance still falls short of entity embedding-based models. along with the fact that transformer-based models performed well on the other entity-related tasks, such as question answering, by using entity embeddings @xcite @xcite , we can infer the importance of explicitly modeling knowledge as the parameters to optimize the task performance. this study focuses on this powerful entity embedding-based models. description: ben is force-sensitive and a son of han solo troopers are commanded by ren softmax kylo_ren transformer description encoder hyperlink encoder entity embedding inner product pooling sentence with hyperlinks: fell solo ben [cls] [sep] side dark the to dark_side [mask] linear transform ben solo fell to the datk side input: predicted entity: word tokens entity tokens figure 1: our method complements the entity embedding matrix by encoding entities' descriptions and relevant sentences. the underlines refer to mentions. ben solo, ben, and ren refer to the same entity kylo_ren. here, kylo_ren is newly added to the model's entity vocabulary by our method while it is originally not included in it. ## steel in this section, we present our adaptation method, steel, which complements static and encoderbased entity embeddings for entity linking. while our method is applicable to a variety of entity embedding-based methods, in this paper, we apply it to the architecture of the state-of-the-art ed model developed in @xcite . we begin by training the base ed model, whose entity embeddings comprise only a reasonable number of common entities v e ( § 4.1). we then estimate the embeddings for other entities / ∈ v e using our proposed encoders, and extend the model ( § 4.2). hereafter, v w and v e denote the word and entity vocabularies of the base model. h and d denote the size of the hidden states and of the embeddings. ## experiments we conduct experiments on entity disambiguation with the goal of verifying that the performance of yamada 500k base with steel is comparable to that of state-of-the-art models, whose entity vocabulary contains all candidate entities in the evaluation as well as to that of embedding-free models. ## analysis in this section, we provide a detailed analysis of our proposed adaptation method, steel. ## conclusions in this study, we proposed an adaptation method, steel, which makes trained entity embeddingbased ed models applicable to candidate entities that are not integrated into the model ( § 4). our methods extend trained models by predicting entity embeddings of other rare or emerging entities by encoding the corresponding descriptions or relevant sentences. in the experiments with standard benchmark datasets ( § 5), our model performed competitive or better than existing entity embedding-based models trained for all candidate entities used in the evaluation, and also outperformed embeddingfree models. we also conducted analyses concerning the predicted embeddings ( § 6.1 to 6.5), and evaluated the capability of reducing memory costs required during inference via quantization ( § 6.6). our methods, in addition to out-of-vocabulary entities, can also handle evolving entities whose properties have changed after training the model, even though they were included in the entity vocabulary set during training. this capability is vital in ed since a kb to be linked can be updated intermittently. for example, wikipedia articles before and after "joe biden" have become u.s. president are very different. his embedding trained using the former periods of articles may not be optimal for processing mentions in the newer articles. the performance of replacing trained embeddings with new ones should be evaluated in the future. as a base model to which our method is applied, we adopted @xcite since it is the state-of-the-art and its source code is publicly available. we release our implementation at https://github.com/studio-ousia/steel for the other models to enjoy our method in the future.