Transformer-based Entity Typing in Knowledge Graphs (2022)

Abstract: We investigate the knowledge graph entity typing task which aims at inferring plausible entity types. In this paper, we propose a novel Transformer-based Entity Typing (TET) approach, effectively encoding the content of neighbours of an entity by means of a transformer mechanism. More precisely, TET is composed of three different mechanisms: a local transformer allowing to infer missing entity types by independently encoding the information provided by each of its neighbours; a global transformer aggregating the information of all neighbours of an entity into a single long sequence to reason about more complex entity types; and a context transformer integrating neighbours content in a differentiated way through information exchange between neighbour pairs, while preserving the graph structure. Furthermore, TET uses information about class membership of types to semantically strengthen the representation of an entity. Experiments on two real-world datasets demonstrate the superior performance of TET compared to the state-of-the-art.

## introduction a knowledge graph (kg) @xcite is a multi-relational graph encoding factual knowledge, with the form (h, r, t) where h, t are the head and tail entities connected via the relation r. in this paper, we consider kgs with minimal schema information, i.e., those containing entity type assertions, as the only schema information, of the form (e, has_type, c) stating that the entity e has type c; e.g., to capture that barack obama has type president. entity type knowledge is widely used in nlp tasks, e.g., in relation extraction @xcite , entity and relation linking @xcite , question answering @xcite , and finegrained entity typing on text @xcite @xcite . however, entity types are far from complete, since in real-world applications they are continuously emerging. for example, about 10% of entities in fb15k @xcite have the type /music/artist, but do not have /people/person @xcite . in light of this, it has been recently investigated the knowledge graph entity typing (kget) task, aiming at inferring missing entity types in a kg. most existing approaches to kget use methods based on either embeddings or graph convolutional networks (gcn). despite the huge progress these methods have made, there are still some important challenges to be solved. on the one hand, most embedding-based models @xcite @xcite encode all neighbors of a target entity into a single vector, but in many cases only some neighbors are necessary to infer the correct types. for example, as shown in figure 1 , to predict that the entity barack obama has type president, only the neighbor is_leader_of ------→ u.s is needed. indeed, using too many neighbors, such as graduate_from --------→ columbia university, will introduce noise. the cet model @xcite overcomes this problem by encoding each neighbor independently. however, since entities and relations are repre-sented by transe @xcite , there is a restriction on the direction of the representation of entities and relations direction, fixing it from entity to relation or vice versa. as a consequence, certain interactions between neighbor entities and relations are ignored. also, to predict more complex types, cet directly adds and averages the neighbor representations, weakening the contribution of different neighbors, since it ignores that the contribution of different neighbors to different types might not be the same. for example, as shown in figure 1 , the inference of the type 20th-century american writer involves multiple semantic aspects of barack obama, it requires to jointly consider the neighbors write --→ a promised land, was_born_in -------→ 1961, and is_leader_of ------→ u.s, but the neighbor degree_award -------→ juris doctor should get less attention. on the other hand, gcn frameworks for kget use expressive representations for entities and relations based on their neighbor entities and relations @xcite @xcite @xcite . however, a common problem of gcn-based models is that they aggregate information only along the paths starting from neighbors of the target entity, limiting the representation of interdependence between neighbors that are not directly connected. for example, in figure 1 the entities juris doctor and u.s are not connected, but combining their information could help to infer that american legal scholars is a type of barack obama. this could be fixed by increasing the number of layers, but with an additional computational cost. the main objective of this paper is to introduce a transformer-based approach to kget that addresses the highlighted challenges. the transformer architecture @xcite has been essential for nlp, e.g., in pre-trained language models @xcite @xcite , document modeling @xcite , and link prediction @xcite . transformers are well-suited for kget as entities and relations in a kg can be regarded as tokens, and using the transformer as encoder, one can thus achieve bidirectional deep interaction between entities and relations. specifically, we propose tet, a transformer-based entity typing model for kget, composed of the following three inference modules. a local transformer that independently encodes the relational and type neighbors of an entity into a sequence, facilitating bidirectional interaction between elements within the sequence, addressing the first problem. a global transformer that aggregates all neighbors of an entity into a single long sequence to simultaneously consider multiple attributes of an entity, allowing to infer more 'complex' types, thus addressing the third problem. a context transformer that aggregates neighbors of an entity in a differentiated manner according to their contribution while preserving the graph structure, thus addressing the second problem. furthermore, we use semantic knowledge about the known types in a kg. in particular, we find out that types are normally clustered in classes. for example, the types medicine/disease, medicine/symptom, and medicine/drug belong to the class medicine. we use this class membership information for replacing the 'generic' relation has_type with a more fine-grained relation that captures to which class a type belongs to, enriching the semantic content of connections between entities and types. to sum up, our contributions are: data, code, and an extended version with appendix are available at https://github. com/zhiweihu1103/et-tet . ## related work the knowledge graph completion (kgc) task is usually concerned with predicting the missing head or tail entities of a triple. kget can thus be seen as a specialization of kgc. existing kget methods can be classified in embedding-and gnc-based. embedding-based methods. ete @xcite learns entity embeddings for kgs by a standard representation learning method @xcite , and further builds a mechanism for information exchange between entities and their types. connecte @xcite jointly embeds entities and types into two different spaces and learns a mapping from the entity space to the type space. core @xcite utilizes the models ro-tate @xcite and complex @xcite to embed entities and types into two different complex spaces, and develops a regression model to link them. however, the above methods do not fully consider the known types of entities while training the entity embedding representation, which seriously affects the prediction performance of missing types. also, the representation of types in these methods is such that they cannot be semantically differentiated. cet @xcite jointly utilizes information about existing type assertions in a kg and about the neighborhood of entities by respectively employing an independentbased mechanism and an aggregated-based one. it also utilizes a pooling method to aggregate their inference results. attet @xcite designs an attention mechanism to aggregate the neighborhood knowledge of an entity using type-specific weights, which are beneficial to capture specific characteristics of different types. a shortcoming of these two methods is that, unlike our tet model, they are not able to cluster types in classes, and are thus not able to semantically differentiate them in a fine-grained way. gcn-based methods. graph convolutional networks (gcns) have proven effective on modeling graph structures @xcite @xcite . however, directly using gcns on kgs usually leads to poor performance since kgs have different kinds of entities and relations. to address this problem, @xcite proposes to apply relation-specific transformations in gcn's aggregation. hmgcn @xcite proposes a hierarchical multi-graph convolutional network to embed multiple kinds of semantic correlations between entities. compgcn @xcite uses composition operators from kg-embedding methods by jointly embedding both entities and relations in a relational graph. connecte-mrgat @xcite proposes a multiplex relational graph attention network to learn on heterogeneous relational graphs, and then utilizes the connecte method for infering entity types. race2t @xcite introduces a relational graph attention network method, utilizing the neighborhood and relation information of an entity for type inference. a common problem with these methods is that they follow a simple single-layer attention formulation, restricting the information transfer between unconnected neighbors of an entity. transformer-based methods. to the best of our knowledge, there are no transformer-based approaches to kget. however, two transformerbased frameworks for the kgc task have been already proposed: coke @xcite and hitter @xcite . our experiments show that they are not suitable for kget. ## method in this section, we describe the architecture of our tet model (cf. figure 2 ). we start by introducing necessary background (sec. 3.1), then present in detail the architecture of tet (sec. 3.2). finally, we describe pooling and optimization strategies (sec. 3.3 and 3.4). ## experiments in this section, we discuss the evaluation of tet relative to twelve baselines on a wide array of entity typing benchmarks. we first describe datasets and baseline models (sec. 4.1). then we discuss the experimental results (sec. 4.2). finally, we present ablation study experiments (sec. 4.3). ## conclusions in this paper, we propose a novel transformer-based model for kget which utilizes contextual information of entities to infer missing types for kgs with minimal schema information. tet has three modules allowing to encode local and global neighborhood information from different perspectives. we also enhance the representation of entities by using class membership knowledge of types. we experimentally showed the benefits of our model.