Context-aware Entity Typing in Knowledge Graphs (2021)

Abstract: Knowledge graph entity typing aims to infer entities’ missing types in knowledge graphs which is an important but under-explored issue. This paper proposes a novel method for this task by utilizing entities’ contextual information. Specifically, we design two inference mechanisms: i) N2T: independently use each neighbor of an entity to infer its type; ii) Agg2T: aggregate the neighbors of an entity to infer its type. Those mechanisms will produce multiple inference results, and an exponentially weighted pooling method is used to generate the final inference result. Furthermore, we propose a novel loss function to alleviate the false-negative problem during training. Experiments on two real-world KGs demonstrate the effectiveness of our method. The source code and data of this paper can be obtained from https://github.com/CCIIPLab/CET .

## introduction knowledge graphs (kgs) store world knowledge in a structured way. they consist of collections of triples in the form of (head entity, relation, tail entity), and entities are labeled with types (see figure 1 ). the entity type information on knowledge graph has applications in many nlp tasks including entity linking @xcite , question answering @xcite and fine-grained entity typing in text @xcite @xcite . an entity can have multiple types, and the entity type information on the knowledge graph is usually incomplete. in this paper, we focus on knowledge graph entity typing (kget), which aims to infer entities' missing types in knowledge graphs. existing methods for the kget task can be divided into embedding-based methods and graph convolutional networks (gcns) for the multirelational graph. knowledge graph embedding (kge) is representative of embedding-based methods. treating entities' known types as special triples with a unique relation "has type", e.g., (barack obama, has type, person), the kget task can be understood as a subtask of knowledge graph completion. consequently, kge methods can infer entities' missing types by completing (entity, has type, ?). recently, two embedding-based kget models based on kge have been proposed: ete @xcite and connecte @xcite . they first obtain entity embeddings from kge methods, then use them to infer entities' missing types. gcns for the multi-relational graph can aggregate the rich information in entities' neighbors to infer entities' missing types. existing methods usually encode all attributes of an entity into one embedding, then use this representation to conduct inference. however, when judging whether an entity has a particular type, only some attributes of this entity may be helpful while the others remain useless. for example, in figure 1 , only the neighbor (graduate from, cornell university) can indicate the central entity steven weinberg have type cornell university alumni. we argue that always considering all attributes of an entity during inference may introduce irrelevant information as noise and ultimately reduce the accuracy of entity typing. besides the above-mentioned shortcoming, ete and connecte also ignore entities' known type information when training entity embeddings, which is important for entity typing. for instance, in figure 1 there are no known triples which can indicate entity steven weinberg has type jewish physicists. in this case, the model needs to utilize the known type information. steven weinberg has type jewish scientists, and in the known triples exists (steven weinberg, has won prize, nobel prize in physics). combining the two, we can infer steven weinberg has type jewish physicists. in short, these two kget models have difficult using the entities' known types to infer the missing ones. in the experiment, we found this seriously affect their performance. to overcome those shortcomings in existing methods, we propose a novel method for the kget task, called cet (context-aware entity typing). specifically, cet contains two inference mechanisms: i) n2t: independently use each neighbor of an entity to infer its type; ii) agg2t: aggregate the neighbors of an entity to infer its type. according to our observation, one neighbor usually represents a specific attribute of the central entity. thus, the n2t mechanism allows cet to consider each attribute of an entity during inference individually. in contrast, previous works mix various attributes of an entity into one embedding for inference. therefore, we believe cet can produce more accurate entity typing results than existing methods. moreover, some complex types like 21st-century american novelists involve multiple semantic aspects of an entity. it's difficult to infer those types only using a single neighbor. therefore, we further propose the agg2t mechanism, which simultaneously considers multiple attributes of the central entity during inference by aggregating neighbors. we also treat the known types of the central entity as its neighbors to use them to infer the missing types. to aggregate the inference results generated by n2t and agg2t mechanism, we adopt a carefully designed pooling method similar to softpool @xcite . experiments show that this pooling method can produce stable and interpretable inference results. in addition, we face serious false-negative problem during training. some (entity, type) pairs are valid but happen to be missing in current knowledge graphs. treating them as negative samples will seriously affect model performance. we propose a novel loss function to alleviate this. to sum up, our contributions are three-fold: embedding-based methods. @xcite propose to learn type embedding for knowledge graph entity typing and build two methodologies: i) synchronous training: adding entities' known types to knowledge graphs in the form of triples with a unique relation "has type", e.g., (barack obama, has type, person), knowledge graph embedding (kge) methods @xcite @xcite can learn the embeddings of entities and types simultaneously. kge methods can infer entities' missing types by completing (entity, has type, ?). ii) asynchronous training: the model first obtains entities' embeddings from kge methods, then minimizes the l1 distance between the entities' and their corresponding types' embeddings while keeping the entities' embeddings fixed. during inference, the smaller l1 distance between an entity and a specific type means the entity is more likely to have this type. @xcite observe that there will be only one type of relation associate with types in synchronous training. they claim this lack of diversity of relations means that synchronous training methods have difficulty solving the kget task. so they proposed a model called ete, which follows the asynchronous training strategy and uses conte @xcite to obtain entity embeddings. @xcite propose connecte, a more advanced kget model which contains two inference mechanisms. one is called e2t, which uses a linear transformation to project the entities' embeddings into type embedding space. another is called trt, which uses the neighbors' types to infer the central entities' missing types. trt is based on the assumption that the relationship can remain unchanged when replacing the entities in the triple to their corresponding types. for instance, if triple (barack obama, born in, honolulu) holds, a new triple (person, born in, location) should also hold. connecte also follows the asynchronous training strategy, which first uses transe to obtain entities' embedding then fixes them to train e2t and trt. ete and connecte do not consider entities' known types when training entities' embeddings, which means they do not encode the known type information into entities' embeddings. therefore, both of them have difficulty using entities' known types to infer the missing ones, which seriously affects their performance. our experiments support this claim. gcns for multi-relational graph. the trt mechanism in connecte attempts to use entities' neighbors to infer entities' missing types. however, trt only utilizes the neighbors' types. to fully utilize the information in entities' neighbors, gcns for multi-relational graphs can be used to encode entities' neighbors. @xcite proposed r-gcn, an extension of gcns for relational graphs. r-gcn aggregate the information in neighbors using the relation-specific filter. weighted graph convolutional network @xcite utilizes learnable relational specific scalar weights to aggregate neighbors. @xcite proposed a more generalized framework by leverage composition operators from kge techniques during gcn aggregation. in the kget task, the entities' missing types can be inferred by performing multi-label classification on entities' embeddings obtained by gcns. existing methods usually encode all attributes of an entity into one embedding during inference. we argue this will introduce noise as sometimes only part of attributes of an entity is helpful for the kget task while the others may be useless. to overcome this shortcoming, we propose the n2t mechanism. by independently uses each neighbor of an entity to infer its missing types, the n2t mechanism allows our model to consider each attribute of an entity during inference individually. this can reduce the impact of irrelevant information on entity typing. also, we treat entities' known types as neighbors which means our model can use them to infer entities' missing types. others. note that embedding knowledge graphs containing concepts (ontologies) and modeling the relationship between concepts @xcite are not the goal of this paper. we concentrate on inferring entities' missing types. some other works on kget @xcite mainly focus on how to combine additional information, such as the text description of the entities, to infer the missing types. our work only uses the information on the knowledge graphs to infer the missing types of entities, which is more universal. ## method in this section, we introduce our proposed method in detail. we first introduce the notations used in this paper. afterward, we introduce two inference mechanisms used in our method. finally, we introduce a novel loss function that can alleviate the false negative problem during training. ## conclusion this paper describes a novel knowledge graph entity typing method called cet, which utilizes the entities' contextual information to infer entities' missing types. we design two inference mechanisms, one is to independently use each neighbor of an entity to infer its types, another is to aggre-gate entities' neighbors than conduct inference. in addition, we propose a novel loss function to alleviate the false negative problem during training. our method is highly flexible, and we are considering introducing advanced graph convolutional network technology into our method. a hyper-parameter settings r-gcn. we use one layer r-gcn with 100dimension embeddings in our experiment. the hyper-parameters are tuned according to the mrr on the validation set. the search space for the grid search are set as follows: learning rate @xmath0.001, 0.005, 0.01}, activation function ϕ ∈ none, relu, tanh, and the weight of negative samples in fna loss β ∈ {1, 2, 3, 4}. the input embedding is randomly initialized with a uniform distribution [-0.1, 0.1], and the training batch size is fixed to 128. using basis-or block-diagonal-decomposition do not improve results but removing the self-loop improve performance.