P air RE : Knowledge Graph Embeddings via Paired Relation Vectors (2021)

Abstract: Distance based knowledge graph embedding methods show promising results on link prediction task, on which two topics have been widely studied: one is the ability to handle complex relations, such as N-to-1, 1-to-N and N-to-N, the other is to encode various relation patterns, such as symmetry/antisymmetry. However, the existing methods fail to solve these two problems at the same time, which leads to unsatisfactory results. To mitigate this problem, we propose PairRE, a model with paired vectors for each relation representation. The paired vectors enable an adaptive adjustment of the margin in loss function to fit for different complex relations. Besides, PairRE is capable of encoding three important relation patterns, symmetry/antisymmetry, inverse and composition. Given simple constraints on relation representations, PairRE can encode subrelation further. Experiments on link prediction benchmarks demonstrate the proposed key capabilities of PairRE. Moreover, We set a new state-of-the-art on two knowledge graph datasets of the challenging Open Graph Benchmark.

## introduction knowledge graphs store huge amounts of structured data in the form of triples, with projects such as wordnet @xcite , freebase @xcite , @xcite and dbpedia @xcite . they have gained widespread attraction from their successful use in tasks such as question answering @xcite , semantic parsing @xcite , and named entity disambiguation @xcite and so on. since most knowledge graphs suffer from incompleteness, predicting missing links between entities has been a fundamental problem. this problem is named as link prediction or knowledge graph completion. knowledge graph embedding methods, which embed all entities and relations into a low dimensional space, have been proposed for this problem. distance based embedding methods from transe @xcite to the recent state-of-the-art rotate @xcite have shown substantial improvements on knowledge graph completion task. two major problems have been widely studied. the first one refers to handling of 1-to-n, n-to-1, and n-to-n complex relations @xcite . in case of the 1-to-n relations, given triples like (stevenspielberg, directorof , ?), distance based models should make all the corresponding entities about film name like jaws and jurassicp ark have closer distance to entity stevenspielberg after transformation via relation directorof . the difficulty is that all these entities should have different representations. same issue happens in cases of n-to-n and n-to-1 relations. the latter is learning and inferring relation patterns according to observed triples, as the success of knowledge graph completion heavily relies on this ability @xcite . there are various types of relation patterns: symmetry (e.g., issimilart o), antisymmetry (e.g., f atherof ), inverse (e.g., p eoplebornhere and p laceof birth), composition (e.g., my mother's father is my grandpa) and so on. previous methods solve these two problems separately. transh @xcite , transr @xcite , transd @xcite all focus on ways to solve complex relations. however, these methods can only encode symmetry/antisymmetry relations. the recent state-ofthe-art rotate shows promising results to encode symmetry/antisymmetry, inverse and composition relations. however, complex relations remain challenging to predict. here we present pairre, an embedding method that is capable of encoding complex relations and multiple relation patterns simultaneously. the proposed model uses two vectors for relation representation. these vectors project the corresponding head and tail entities to euclidean space, where the distance between the projected vectors is minimized. this provides three important benefits: ## background and notation given a knowledge graph that is represented as a list of fact triples, knowledge graph embedding methods define scoring function to measure the plausibility of these triples. we denote a triple by (h, r, t), where h represents head entity, r represents relation and t represents tail entity. the column vectors of entities and relations are represented by bold lower case letters, which belong to set e and r respectively. we denote the set of all triples that are true in a world as t . f r (h, t) represents the scoring function. we take the definition of complex relations from @xcite . for each relation r, we compute average number of tails per head (tphr) and average number of heads per tail (hptr). if @xmath0.5 and @xmath1.5, r is treated as 1-to-1; if @xmath2.5 and @xmath3.5, r is treated as a n-to-n; if @xmath4.5 and @xmath5.5, r is treated as 1-to-n. we focus on four important relation patterns, which includes: (1) symmetry/antisymmetry. a relation r is symmetric if ∀e 1 , e 2 ∈ e, (e 1 , r, e 2 ) ∈ t ⇐⇒ (e 2 , r, e 1 ) ∈ t and is antisymmetric if (e 1 , r, e 2 ) ∈ t ⇒ (e 2 , r, e 1 ) / ∈ t ; (2) inverse. if ∀e 1 , e 2 ∈ e, (e 1 , r 1 , e 2 ) ∈ t ⇐⇒ (e 2 , r 2 , e 1 ) ∈ t , then r 1 and r 2 are inverse relations; (3) composition. if ∀e 1 , e 2 , e 3 ∈ e, (e 1 , r 1 , e 2 ) ∈ t ∧ (e 2 , r 2 , e 3 ) ∈ t ⇒ (e 1 , r 3 , e 3 ) ∈ t , then r 3 can be seen as the composition of r 1 and r 2 ; (4) subrelation @xcite . if ∀e 1 , e 2 ∈ e, (e 1 , r 1 , e 2 ) ∈ t ⇒ (e 1 , r 2 , e 2 ) ∈ t , then r 2 can be seen as a subrelation of r 1 . the recent state-of-the-art, rotate, which can encode symmetry/antisymmetry, inverse and composition relation patterns, utilizes rotation based translational method in a complex space. although expressiveness for different relation patterns, complex relations remain challenging. gc-ote @xcite proposes to improve complex relation modeling ability of rotate by introducing graph context to entity embedding. however, the calculation of graph contexts for head and tail entities is time consuming, which is inefficient for large scale knowledge graphs, e.g. ogbl-wikikg @xcite . another related work is se @xcite , which utilizes two separate relation matrices to project head and tail entities. as pointed out by @xcite , this model is not able to encode symmetry/antisymmetry, inverse and composition method score function performance of complex relations semantic matching models. semantic matching models exploit similarity-based scoring functions, which can be divided into bilinear models and neural network based models. as the models have been developed, such as rescal @xcite , distmult @xcite , hole @xcite , complex @xcite and quate @xcite , the key relation encoding abilities are enriched. however, all these models have the flaw in encoding composition relations @xcite . rescal, complex and simple (kazemi and poole, 2018) are all proved to be fully expressive when embedding dimensions fulfill some requirements @xcite @xcite . the fully expressiveness means these models can express all the ground truth which exists in the data, including complex relations. however, these requirements are hardly fulfilled in practical use. it is proved by @xcite ) that, to achieve complete expressiveness, the embedding dimension should be greater than n /32, where n is the number of entities in dataset. neural networks based methods, e.g., convolution neural networks @xcite , graph convolutional networks @xcite show promising performances. however, they are difficult to analyze as they work as a black box. encoding subrelation. existing methods encode subrelation by utilizing first order logic rules. one way is to augment knowledge graphs with grounding of rules, including subrelation rules @xcite . the other way is adding constraints on entity and relation representations, e.g., complex-nne-aer and simple + . the second way enriches the models' expressiveness with relatively low cost. in this paper, we show that pairre can encode subrelation with constraints on relation representations while keeping the ability to encode symmetry/antisymmetry, inverse and composition relations. ## methodology to overcome the problem of modeling 1-to-n/nto-1/n-to-n complex relations and enrich the capabilities for different relation patterns, we propose a model with paired vectors for each relation. given a training triple (h, r, t), our model learns vector embeddings of entities and relation in real space. specially, pairre takes relation embedding as paired vectors, which is represented as [r h , r t ]. r h and r t project head entity h and tail entity t to euclidean space respectively. the projection operation is the hadamard product foot_0 between these two vectors. pairre then computes distance of the two projected vectors as plausibility of the triple . we want that h • r h ≈ t • r t when (h, r, t) holds, while h • r h should be far away from t • r t otherwise. in this paper, we take the l 1 -norm to measure the distance. in order to remove scaling freedoms, we also add constraint on embeddings similar to previous distance based models @xcite @xcite . and the constraint is only added on entity embeddings. we want relation embeddings to capture semantic connection among relation vectors (e.g., p eoplebornhere and p laceof birth) and complex characteristic (e.g., 1-n) easily and sufficiently. for entity embedding, the l 2 -norm is set to be 1. the scoring function is defined as follows: the model parameters are, all the entities' embeddings, {e j } e @xmath6, {r j } r @xmath7. illustration of the proposed pairre is shown in figure 1 . compared to transe/rotate, pairre enables an entity to have distributed representations when involved in different relations. we also find the paired relation vectors enable an adaptive adjustment of the margin in loss function, which alleviates the modeling problem for complex relations. let's take a 1-to-n relation as an example. we set the embedding dimension to one and remove the constraint on entity embeddings for better illustration. given triples (h, r, ?), where the correct tail entities belong to set @xmath8, t 2 , ..., t n }, pairre predicts tail entities by letting the above analysis shows pairre can adjust the value of r t to fit the entities in s. the larger the size of s, the smaller the absolute value r t . while models like transe or rotate have a fixed margin for all complex relation types. when the size of s is large enough, these models will be difficult to fit the data. for n-to-1 relations, pairre can also adjust the value of r h adaptively to fit the data. meanwhile, not adding a relation specific translational vector enables the model to encode several key relation patterns. we show these capabilities below.