Exploring Neural Entity Representations for Semantic Information (2020)

Abstract: Neural methods for embedding entities are typically extrinsically evaluated on downstream tasks and, more recently, intrinsically using probing tasks. Downstream task-based comparisons are often difficult to interpret due to differences in task structure, while probing task evaluations often look at only a few attributes and models. We address both of these issues by evaluating a diverse set of eight neural entity embedding methods on a set of simple probing tasks, demonstrating which methods are able to remember words used to describe entities, learn type, relationship and factual information, and identify how frequently an entity is mentioned. We also compare these methods in a unified framework on two entity linking tasks and discuss how they generalize to different model architectures and datasets.

## introduction neural methods for generating entity embeddings have become the dominant approach to representing entities, with embeddings learned through methods such as pretraining, task-based training, and encoding knowledge graphs @xcite @xcite . these embeddings can be compared extrinsically by performance on a downstream task, such as entity linking (el). however, performance depends on several factors, such as the architecture of the model they are used in and how the data is preprocessed, making direct comparison of the embeddings hard. another way to compare these embeddings is intrinsically using probing tasks @xcite , which have been used to examine entity embeddings for information such as an entity's type, relation to other entities, and factual information @xcite @xcite . these prior examinations have often examined only a few methods, and some propose tasks that can only be applied to certain classes of embeddings, such as those produced from a mention of an entity in context. we address these gaps by comparing a wide range of entity embedding methods for semantic information using both probing tasks as well as downstream task performance. we propose a set of probing tasks derived simply from wikipedia and db-pedia, which can be applied to any method that produces a single embedding per entity. we use these to compare eight entity embedding methods based on a diverse set of model architectures, learning objectives, and knowledge sources. we evaluate how these differences are reflected in performance on predicting information like entity types, relationships, and context words. we find that type information is extremely well encoded by most methods and that this can lead to inflated performance on other probing tasks. we propose a method to counteract this and show that it allows a more reliable estimate of the encoded information. finally, we evaluate the embeddings on two el tasks to directly compare their performance when used in different model architectures, identifying some that generalize well across multiple architectures and others that perform particularly well on one task. we aim to provide a clear comparison of the strengths and weaknesses of various entity embedding methods and the information they encode to guide future work. our probing task datasets, embeddings, and code are available online. 1 is task-learned embeddings, which are learned as part of a downstream task, such as el. pretrained embeddings are learned through a dedicated pretraining phase designed to produce entity embeddings. finally, derived embeddings are produced by models capable of embedding any generic text, but that had no specific entity-based training. along the content dimension, the first type is description-based embeddings, which are learned or generated from a text description of the entity. context-based embeddings are learned from words surrounding mentions of the entity. lastly, graph-based embeddings are learned entirely from a knowledge graph, linking entities to types and to each other. models may leverage multiple types of information to learn embeddings. we use the march 5, 2016 dump of wikipedia to train our task-learned and pretrained embedding models, while the derived embedding models are publicly available pre-trained language models. 234 ## entity embedding probing tasks we next introduce a set of 22 probing tasks which can be applied to all of the embedding methods described above, divided into 5 categories based on the information they probe: context words used to describe a given entity, entity type information, relationships between entities, how frequently an entity is referenced, and factual knowledge. ## probing experiment analysis we present the results of our probing tasks in tables 1 and 2 and analyze them in the following sections. context word prediction. ganea performs best on the two word context tasks, beating even bert and bert-large, and demonstrating one of the advantages of their shared word and entity embedding space. cnn performs on par with the bow model, indicating that the task-learned embeddings store a fair amount of lexical information to complete the el task. the rnn embeddings perform at almost chance level, which could mean the lexical information is stored in the rnn layers and not transferred to the entity embeddings. examining the highest performing words across the models, the most common high-performing categories are domain-specific terms like nhl, genus, and manga, place names and demonyms like china and australian, and entity type descriptors like rapper and pitcher. the higher performance on domain-specific words can also be seen in the general increased performance on mid-frequency words, which are often more domain-specific. this helps explain the surprisingly decent performance of biggraph, which was not trained with text data, but may be able to use the fine-grained entity type data it was trained on to identify specific domains. ## conclusion in this work, we propose a new set of probing tasks for evaluating entity embeddings which can be applied to any method that creates one embedding per entity. using these tasks, we find that entity type information is one of the strongest signals present in all but one of the embedding models, followed by coarse information about how likely an entity is to be mentioned. we show that the embeddings are particularly able to use entity type information to bootstrap their way to improved performance on entity relationship and factual information prediction tasks and propose methods to counteract this to more accurately estimate how well they encode relationships and facts. overall, we find that while bert-based entity embeddings perform well on many of these tasks, their high performance can often be attributed to strong entity type information encoding. more specialized models such as wikipedia2vec are better able to detect and identify relationships, while the embeddings of @xcite better capture the lexical and distributional semantics of entities. additionally, we provide a direct comparison of the embeddings on two downstream el tasks, where the models that performed well on the probing tasks such as ganea, wiki2v, and bert performed best on the downstream tasks. we find that the best performing embedding model depends greatly on the surrounding architecture and encourage future practitioners to directly compare newly proposed methods with prior models in a consistent architecture, rather than only compare results. our work provides insight into the information encoded by static entity embeddings, but entities can change over time, sometimes quite significantly. one future line of work we would like to pursue using our tests is to investigate how changes in entities over time can be reflected in the embeddings, and how those changes could be modeled as transformations in the embedding space. context-based embeddings in particular could then be dynamically updated with new information, instead of being retrained from scratch. zhengyan zhang, xu han, zhiyuan liu, xin jiang, maosong sun, and qun liu. 2019. ernie: enhanced language representation with informative entities. in proceedings of the 57th conference of the association for computational linguistics, acl 2019, florence, italy, july 28-august 2, 2019, volume 1: long papers, pages 1441-1451. association for computational linguistics.