R e I nception E : Relation-Aware Inception Network with Joint Local-Global Structural Information for Knowledge Graph Embedding (2020)

Abstract: The goal of Knowledge graph embedding (KGE) is to learn how to represent the low dimensional vectors for entities and relations based on the observed triples. The conventional shallow models are limited to their expressiveness. ConvE (Dettmers et al., 2018) takes advantage of CNN and improves the expressive power with parameter efficient operators by increasing the interactions between head and relation embeddings. However, there is no structural information in the embedding space of ConvE, and the performance is still limited by the number of interactions. The recent KBGAT (Nathani et al., 2019) provides another way to learn embeddings by adaptively utilizing structural information. In this paper, we take the benefits of ConvE and KBGAT together and propose a Relation-aware Inception network with joint local-global structural information for knowledge graph Embedding (ReInceptionE). Specifically, we first explore the Inception network to learn query embedding, which aims to further increase the interactions between head and relation embeddings. Then, we propose to use a relation-aware attention mechanism to enrich the query embedding with the local neighborhood and global entity information. Experimental results on both WN18RR and FB15k-237 datasets demonstrate that ReInceptionE achieves competitive performance compared with state-of-the-art methods.

## introduction knowledge graphs (kgs) are at the core of most state-of-the-art natural language processing solutions and have been spotlighted in many real-world applications, including question answering @xcite , dialogue generation @xcite and machine reading comprehension @xcite . typically, kgs are directed graphs whose nodes denote the entities and edges represent the different relations between entities. the structured knowledge in kgs is organized in the form of triples (h, r, t), where h and t stand for the head and tail entities respectively, and r represents the relation from h to t. although large-scale kgs (e.g., freebase @xcite , dbpedia @xcite ) have already contained millions or even billions of triples, they are still far from complete since the emerging new knowledge appears. knowledge graph embedding (kge) is an effective solution to solve the incompletion problem. kge aims to learn the low-dimensional vectors (embeddings) for entities and relations based on the observed triples in kgs. conventional models including transe @xcite and its numerous extensions (e.g., transd @xcite , transr @xcite , distmul @xcite , complex @xcite , etc.) have been proposed. these shallow models are limited to their expressiveness @xcite . recently, cnn-based methods have been proposed to capture the expressive features with parameter efficient operators. conve @xcite takes advantage of cnn and uses convolution filters on 2d reshapings of the head entity and relation embeddings. through this, conve can increase the interactions between head and relation embeddings. empirical results have proved that increasing the number of interactions is beneficial to the kge task, but conve is still limited by the number of interactions @xcite . furthermore, conve does not consider the structural information. in contrast, graph-based methods are effective to aggregate neighborhood information to enrich the entity/relation representation @xcite @xcite . among them, kbgat @xcite achieves state-of-the-art performance on various benchmark datasets via using graph attention networks (gat) @xcite . kbgat learns embeddings for every entity by taking all possible relations into account, which requires multiple hops of reasoning. in contrast, it can be beneficial to learn embeddings from a query-relevant subgraph of the local neighborhood and global entities. as an example shown in figure 1 , given a query (jack london, nationality, ?) for jack london, we can gather the relationaware local neighbor (place lived, okaland). the local neighbor allows us to project jack london into the okaland region of the embedding space, which can lead to a high score for predicting the target america, as okaland and america are close in embedding space. besides, we also note that a specific relation can be acted as the "bridge" to link the related entities. considering the relation nationality, the related head entities { kaneto shiozawa, shammi kapoor, will smith, • • • } and tail entities { america, canada, japan, • • • } tend to be a set of person names and countries. these related entities act as a strong signal to judge whether a triple is valid or not. based on the above observations, we take the benefits of conve and kbgat together and propose a relation-aware inception network with joint local-global structural information for knowledge graph embedding, and we name it reincep-tione. in reinceptione, we first adapt inception network @xcite ) -a high performing convolutional neural network with carefully designed filters, to increase the interactions using multiple convolution filters with different scales, while at the same time to keep parameter efficient. then, we construct a local neighborhood graph and a global entity graph by sharing the head and relation respectively for a given query. with the constructed graphs, we apply a relation-aware attention mechanism to aggregate the local neighborhood features and gather the global entity information to enrich the head/relation representation. finally, we aggregate the joint local-global structural information using a fully connected layer to predict the missing links. in summary, we make the following three contributions: (1) it is the first to explore inception network to learn query embedding which aims to further increase the interactions between head and relation embeddings; section 2 describes our proposed method for kge. in section 3, the experimental results are presented. we make a conclusion in section 4. ## our approach in this section, we first describe the background and definition in subsection 2.1, and inception-based query encoder in subsection 2.2. then, we introduce the relation-aware local attention and global attention in subsection 2.3 and 2.4, respectively. finally, we describe the joint using of them in subsection 2.5. ## conclusions in this paper, we propose a novel relation-aware inception network for knowledge graph embedding, called reinceptione. reinceptione takes the benefits of conve and kbgat together. the proposed method first employs inception network to learn the query embedding, with the aim of increasing the interaction between head and relation embeddings, while at the same time to keep the parameter efficient. then, we gather the relation-aware local neighborhood and global entity information with an attention mechanism and enrich the query embedding with the joint local-global structural information. empirical studies demonstrate that our proposed method obtains comparative performance compared with the state-of-the-art performance on two widely used benchmark datasets wn18rr and fb15k-237.