Connecting Embeddings for Knowledge Graph Entity Typing (2020)

Abstract: Knowledge graph (KG) entity typing aims at inferring possible missing entity type instances in KG, which is a very significant but still under-explored subtask of knowledge graph completion. In this paper, we propose a novel approach for KG entity typing which is trained by jointly utilizing local typing knowledge from existing entity type assertions and global triple knowledge in KGs. Specifically, we present two distinct knowledge-driven effective mechanisms of entity type inference. Accordingly, we build two novel embedding models to realize the mechanisms. Afterward, a joint model via connecting them is used to infer missing entity type instances, which favors inferences that agree with both entity type instances and triple knowledge in KGs. Experimental results on two real-world datasets (Freebase and YAGO) demonstrate the effectiveness of our proposed mechanisms and models for improving KG entity typing. The source code and data of this paper can be obtained from: https://github.com/Adam1679/ConnectE .

## introduction the past decade has witnessed great thrive in building web-scale knowledge graphs (kgs), such as freebase @xcite , yago @xcite , google knowledge graph @xcite , which usually consists of a huge amount of triples in the form of (head entity, relation, tail entity) (denoted (e, r, ẽ)). kgs usually suffer from incompleteness and miss important facts, jeopardizing their usefulness in downstream tasks such as question answering @xcite , semantic parsing @xcite , relation classification @xcite . hence, the task of knowledge graph completion (kgc, i.e. completing knowledge graph entries) is extremely significant and attracts wide attention. this paper concentrates on kg entity typing, i.e. inferring missing entity type instances in kgs, which is an important sub-problem of kgc. entity type instances, each of which is in the formed of (entity, entity type) (denoted (e, t)), are essential entries of kgs and widely used in many nlp tasks such as relation extraction @xcite , coreference resolution @xcite , entity linking @xcite . most previous works of kgc focus on inferring missing entities and relationships @xcite @xcite @xcite , paying less attention to entity type prediction. however, kgs also usually suffer from entity types incompleteness. for instance, 10% of entities in fb15k @xcite , which have the /music/artist type, miss the /people/person type @xcite . kg entity type incompleteness leads to some type-involved algorithms in kg-driven tasks grossly inefficient or even unavailable. to solve kg entity type incompleteness issue, in this paper we propose a novel embedding methodology to infer missing entity type instances that employs not only local typing knowledge from entity type assertions, as most conventional mod-els do, but also leverages global triple knowledge from kgs. accordingly, we build two distinct knowledge-driven type inference mechanisms with these two kinds of structural knowledge. mechanism 1. missing entity types of an entity can be found from other entities that are close to the entity in the embedding space, using local typing knowledge as in fig. 1 (mech.1). mechanism 2. missing entity types of an (head or tail) entity can be inferred from the types of other (tail or head) entities through their relationships, using global triple knowledge as in fig. 1 (mech.2). the main idea behind mech.1 is based on the observation that the learned entities' embeddings by conventional kg embedding methods @xcite cluster well according to their types in vector space. for instance, in fig. 1 (mech.1), given an entity barack obama, it's missing hierarchical type /people/person can be induced by the given hierarchical type of similar entity donald trump. in addition, the key motivation behind mech.2 is that the relationship shall remain unchanged if the entities in a triple fact are replaced with their corresponding hierarchical types. for instance, given a global triple fact (barack obama, born in, honolulu), under this assumption, we can induce a new type triple (/people/person, born in, /location/location) foot_0 . formally, honolulu -barack @xmath0, which can be used to infer missing entity types, e.g. (barack obama, @xmath1, as mech.2 does. fig. 1 demonstrates a simple illustration of effective mechanisms of entity type inference. both mechanisms are utilized to build our final composite model. specifically, we build two embedding models to realize the two mechanisms respectively. first, considering entities and entity types are completely distinct objects, we build two distinct embedding spaces for them, i.e., entity space and entity type space. accordingly, we encode (e, t) entity type instance by projecting the entity from entity space to entity type space with mapping matrix m, hence we have (1): m • e t , called e2t. moreover, we learn the plausibility of (t e , r, t ẽ) global type triple by newly generalizing from (e, r, ẽ) global triple fact, even though this type triple is not present originally. following translating assumption @xcite , we have (2): t ẽ -r • t e , called trt. e2t and trt are the implementation models of the two mechanisms. fig. 2 demonstrates a brief illustration of our models. a ranking-based embedding framework is used to train our models. thereby, entities, entity hierarchical types, and relationships are all embedded into low-dimensional vector spaces, where the composite energy score of both e2t and trt are computed and utilized to determine the optimal types for (entity, entity @xmath2. the experimental results on real-world datasets show that our composite model achieves significant and consistent improvement compared to all baselines in entity type prediction and achieves comparable performance in entity type classification. our contributions are as follows: ## related works entity typing is valuable for many nlp tasks @xcite , such as knowledge base population @xcite , question answering @xcite , etc. in recent years, researchers attempt to mine fine-grained entity types @xcite @xcite with external text information, such as web search query logs @xcite , the textual surface patterns @xcite , context representation @xcite , wikipedia @xcite , @xmath3./ triple know. syn. to be more universal, @xcite propose two embedding models, i.e. linear model (lm) and projection embedding model (pem), which can infer missing entity types only with kg itself. although pem has more expressive power than lm, however, both of them ignore global triple knowledge, which could also be helpful for encoding entity type assertions via shared entities' embeddings. to address this issue, @xcite propose a state-of-the-art model (ete) to combine triple knowledge and entity type instances for entity type prediction, and build two entity type embedding methodologies: (1) synchronous training: treat (entity, entity type) assertions as special triple facts that have a unique relationship "rdf:type", e.g. (barack obama, "rdf:type", person), and encode all mixed triple facts (original triple data fused with all generated special ones) by conventional entity relation embedding models, such as rescal @xcite , hole @xcite and transe @xcite . (2) asynchronous training: first learn the entities' embeddings e by conventional entity relation embedding models mentioned above, and then only update entity types' embeddings t for min et ℓ1 while keeping e fixed, called rescal-et, hole-et, transe-et and ete. although these approaches expect to explore global triple knowledge for entity type prediction, they still lack of expressive ability due to its simplicity of embeddings. in addition, they irrationally assume both the embeddings of entities and entity types being in the same latent space (∈ r κ ). since entities and entity types are completely distinct objects, it may not be reasonable to represent them in a common semantic space. in this paper, we introduce an enhanced kg entity type embedding model with better expressing and reasoning capability considering both local entity typing information and global triple knowledge in kgs. note that incorporating more external information @xcite is not the main focus in this paper, as we only consider the internal structural information in kgs instead, which correspondingly makes our work much more challenging but also more universal and flexible due to the limited information. recently, @xcite also attempt to embedding structural information in kg. however, the goals and models are very different from ours. they encodes the concepts, not hierarchical types. on the contrary, we focus on the latter not the former. ## embedding-based framework we consider a kg containing entity type instances of the form (e, t) ∈ h (h is the training set consists of lots of (entity, entity type) assertions), where @xmath4. for example, e could be barack obama and t could be /people/person. as a single entity can have multiple types, entities in kg often miss some of their types. the aim of this work is to infer missing entity type instances in kgs. our work concerns energy-based methods, which learn low-dimensional vector representations (embeddings) of atomic symbols (i.e. entities, entity hierarchical types, relationships). in this framework, we learn two submodels: (1) one for predicting entity types given entities, and (2) another one to encode the interactions among entity types and relationships from kgs. the joint action of both models in prediction allows us to use the connection between triple knowledge and entity type instances to perform kg entity typing. ## conclusion and future work in this paper, we described a framework for leveraging global triple knowledge to improve kg entity typing by training not only on (entity, entity type) assertions but also using newly generated (head type, relationship, tail type) type triples. specifically, we propose two novel embedding-based models to encode entity type instances and entity type triples respectively. the connection of both models is utilized to infer missing entity type instances. the empirical experiments demonstrate the effectiveness of our proposed model. our modeling method is general and should apply to other typeoriented tasks. next, we are considering to use this framework to conduct kg entity type noise detection.