Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding (2020)

Abstract: Distance-based knowledge graph embeddings have shown substantial improvement on the knowledge graph link prediction task, from TransE to the latest state-of-the-art RotatE. However, complex relations such as N-to-1, 1-to-N and N-to-N still remain challenging to predict. In this work, we propose a novel distance-based approach for knowledge graph link prediction. First, we extend the RotatE from 2D complex domain to high dimensional space with orthogonal transforms to model relations. The orthogonal transform embedding for relations keeps the capability for modeling symmetric/anti-symmetric, inverse and compositional relations while achieves better modeling capacity. Second, the graph context is integrated into distance scoring functions directly. Specifically, graph context is explicitly modeled via two directed context representations. Each node embedding in knowledge graph is augmented with two context representations, which are computed from the neighboring outgoing and incoming nodes/edges respectively. The proposed approach improves prediction accuracy on the difficult N-to-1, 1-to-N and N-to-N cases. Our experimental results show that it achieves state-of-the-art results on two common benchmarks FB15k-237 and WNRR-18, especially on FB15k-237 which has many high in-degree nodes.

## introduction knowledge graph is a multi-relational graph whose nodes represent entities and edges denote relationships between entities. knowledge graphs store facts about people, places and world from various sources. those facts are kept as triples (head entity, relation, tail entity) and denoted as (h, r, t). a large number of knowledge graphs, such as freebase @xcite , dbpedia @xcite , nell @xcite and yago3 @xcite , have been built over the years and successfully applied to many domains such as recommendation and question answering @xcite . however, these knowledge graphs need to be updated with new facts periodically. therefore many knowledge graph embedding methods have been proposed for link prediction that is used for knowledge graph completion. knowledge graph embedding represents entities and relations in continuous vector spaces. started from a simple and effective approach called transe @xcite , many knowledge graph embedding methods have been proposed, such as transh @xcite , distmult @xcite , conve @xcite to the latest rotate @xcite and quate @xcite . though much progress has been made, 1-to-n, @xcite still remain challenging. in figure 1 , relation "profession" demonstrates an n-to-n example and the corresponding edges are highlighted as green. assuming the triple (sergeirachmaninoff, profession, pianist) is unknown. the link prediction model takes "sergeirachmaninoff" and relation "profession" and rank all entities in the knowledge graph to predict "pianist". entity "sergeirachmaninoff" connected to multiple entities as head entity via relation "profession", while "pianist" as a tail entity also reaches to multiple entities through relation "profession". it makes the n-to-n prediction hard because the mapping from certain entity-relation pair could lead to multiple different entities. same issue happens with the case of 1-to-n and n-to-1 predictions. the recently proposed rotate @xcite figure 1 : snapshot of knowledge graph in fb15k-237. entities are represented as golden blocks. in this work, a novel distance-based knowledge graph embedding called orthogonal transform embedding (ote) with graph context is proposed to alleviate the 1-to-n, n-to-1 and n-to-n issues, while keeps the desired relation patterns as rotate. first, we employ orthogonal transforms to represent relations in high dimensional space for better modeling capability. the orthogonal transform embedding also models the symmetry/antisymmery, inversion and compositional relation patterns just as rotate does. rotate can be viewed as an orthogonal transform in 2d complex space. second, we integrate graph context directly into the distance scoring, which is helpful to predict 1-to-n, n-to-1 and n-to-n relations. for example, from the incomplete knowledge graph, people find useful context information, such as (sergeirachmaninoff, role, piano) and (sergeirachmaninoff, profession, composer) in figure 1 . in this work, each node embedding in knowledge graph is aug-mented with two graph context representations, computed from the neighboring outgoing and incoming nodes respectively. each context representation is computed based on the embeddings of the neighbouring nodes and the corresponding relations connecting to these neighbouring nodes. these context representations are used as part of the distance scoring function to measure the plausibility of the triples during training and inference. we show that ote together with graph context modeling performs consistently better than rotate on the standard benchmark fb15k-237 and wn18rr datasets. in summary, our main contributions include: ## our proposed method we consider knowledge graph as a collection of triples @xmath0, r, t)} with v as the graph node set, and r as the graph edge set. each triple has a head entity h and tail entity t, where h, @xmath1. relation @xmath2. as discussed in the introduction section, 1-to-n, n-to-1 and n-to-n relation prediction @xcite are difficult to deal with. they are addressed in our proposed approach by: 1) orthogonal relation transforms that operate on groups of embedding space. each group is modeled and scored independently, and the final score is the sum of all group scores. hence, each group could address different aspects of entity-relation pair and alleviate the 1-to-n and n-to-n relation mapping issues; and 2) directed graph context to integrate knowledge graph structure information to reduce the ambiguity. next, we first briefly review rotate that motivates our orthogonal transform embedding (ote), and then describe the proposed method in details. ## conclusions in this paper we propose a new distance-based knowledge graph embedding for link prediction. it includes two-folds. first, ote extends the modeling of rotate from 2d complex domain to high dimensional space with orthogonal relation transforms. second, graph context is proposed to integrate graph structure information into the distance scoring function to measure the plausibility of the triples during training and inference. the proposed approach effectively improves prediction accuracy on the difficult n-to-1, 1-to-n and n-to-n link predictions. experimental results on standard benchmark fb15k-237 and wn18rr show that ote improves consistently over ro-tate, the state-of-the-art distance-based embedding model, especially on fb15k-237 with many high in-degree nodes. on wn18rr our model achieves the new state-of-the-art results. this work is partially supported by beijing academy of artificial intelligence (baai).