R EL W ALK – A L ATENT V ARIABLE M ODEL A PPROACH TO K NOWLEDGE G RAPH E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge Graph Embedding (KGE) is the task of jointly learning entity and relation embeddings for a given knowledge graph.
Existing methods for learning KGEs can be seen as a two-stage process where (a) entities and relations in the knowledge graph are represented using some linear algebraic structures (embeddings), and (b) a scoring function is deﬁned that evaluates the strength of a relation that holds between two entities using the corresponding relation and entity embeddings.
Unfortunately, prior proposals for the scoring functions in the ﬁrst step have been heuristically motivated, and it is unclear as to how the scoring functions in KGEs relate to the generation process of the underlying knowledge graph.
To address this issue, we propose a generative account of the KGE learning task.
Speciﬁcally, given a knowledge graph represented by a set of relational triples ( h, R, t ) , where the semantic relation R holds between the two entities h (head) and t (tail), we extend the random walk model (Arora et al., 2016a) of word embeddings to KGE.
We derive a theoretical relationship between the joint probability p ( h, R, t ) and the embeddings of h , R and t .
Moreover, we show that marginal loss minimisation, a popular objective used by much prior work in KGE, follows naturally from the log-likelihood ratio maximisation under the probabilities estimated from the KGEs according to our theoretical relationship.
We propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph.
The KGEs learnt by our proposed method obtain state-of-the-art performance on FB15K237 and WN18RR benchmark datasets, providing empirical evidence in support of the theory.
I NTRODUCTION
Knowledge graphs such as Freebase (Bollacker et al., 2008) organise information in the form of graphs, where entities are represented by vertices in the graph and the relation between two entities is represented by the edge that connects the corresponding two vertices.
By embedding entities and relations that exist in a knowledge graph in some (possibly lower-dimensional and latent) space we can infer previously unseen relations between entities, thereby expanding a given knowledge graph (Nickel et al., 2016; Yang et al., 2015; Lin et al., 2015; Nickel et al., 2011; Trouillon et al., 2016; Wang et al., 2017; Bordes et al., 2011).
Existing KGE methods can be seen as involving two main steps.
First, given a knowledge graph represented by a set of relational triples ( h, R, t ) , where a semantic relation R holds between a head entity h and a tail entity t , entities and relations are represented using some mathematical structures such as vectors, matrices or tensors.
Second, a scoring function is proposed that evaluates the relational strength of a triple ( h, R, t ) and entity and relation embeddings that optimise the deﬁned scoring function are learnt using some optimisation method.
Table 1 shows some of the scoring functions proposed in prior work in KGE learning.
Despite the wide applications of entity and relation embeddings created via KGE methods, the existing scoring functions are motivated heuristically to capture some geometric requirements of the embedding space.
For example, TransE (Bordes et al., 2011) assumes that the entity and relation embeddings co-exist in the same (possibly lower dimensional) vector space and translating (shifting) the head entity embedding by the relation embedding must make it closer to the tail entity embedding, whereas ComplEx (Trouillon et al., 2016) models the asymmetry in relations using
[Table]
the component-wise multi-linear inner-product among entity and relation embeddings.
Relational triples extracted from a given knowledge graph are used as positive training instances, whereas pseudo-negative (Bordes et al., 2011) instances are automatically generated by randomly corrupting positive instances.
Finally, KGE are learnt such that the prediction loss computed over the positive and negative instances is minimised.
Despite the good empirical performances of the existing KGE methods, theoretical understanding of
KGE methods is comparatively under developed.
For example, it is not clear how the heuristically deﬁned KGE objectives relate to the generative process of a knowledge graph.
In this paper, we attempt to ﬁll this void by providing a theoretical analysis of KGE.
Speciﬁcally, in section 2, we propose a generative process where we explain the formation of a relation R between two entities h and t using the corresponding relation and entity embeddings.
Following this generative story, we derive a relationship between the probability of R holding between h and t , p ( h, t | R ) , and the embeddings of R , h and t .
Interestingly, the derived relationship is not covered by any of the previously proposed heuristically-motivated scoring functions, providing the ﬁrst-ever KGE method with a provable generative explanation.
Next, in section 3, we show that the margin loss , which has been popularly used as a training objective in prior work on KGE, naturally arises as the log-likelihood ratio computed from p ( h, t | R ) .
Based on this result, we derive a training objective that we subsequently optimise for learning KGEs that satisfy our theoretical relationship.
Using standard benchmark datasets proposed in prior work on
KGE learning, we evaluate the learnt KGEs on a link prediction task and a triple classiﬁcation task.
Experimental results show that the learnt KGEs obtain state-of-the-art performance on FB15K and WN18RR benchmarks, thereby providing empirical evidence to support the theoretical analysis.
R ELATIONAL W ALK
Let us consider a knowledge graph D where the knowledge is represented by relational triples ( h, R, t ) ∈ D .
Here, R is a relational predicate of two arguments, where h ( head ) and t ( tail ) entities respectively ﬁlling the ﬁrst and second arguments.
We assume relations to be asymmetric in general.
In other words, if ( h, R, t ) ∈ D then it does not necessarily follow that ( t, R, h ) ∈ D .
The goal of KGE is to learn embeddings (representations) for the relations and entities in the knowledge graph such that the entities that participate in similar relations are embedded closely to each other in the entity embedding space, while at the same time relations that hold between similar entities are embedded closely to each other in the relational embedding space.
We call the learnt entity and relation embeddings collectively as KGEs.
Following prior work on KGE (Bordes et al., 2011;
Trouillon et al., 2016; Yang et al., 2015), we assume that entities and relations are embedded in the same vector space, allowing us to perform linear algebraic operations using the embeddings in the same vector space.
Let us consider a random walk characterised by a time-dependent knowledge vector c k , where k is the current time step.
The knowledge vector represents the knowledge we have about a particular group of entities and relations that express some facts about the world.
For example, the knowledge that we have about people that are employed by companies can be expressed using entities of classes such as people and organisation, using relations such as CEO-of, employed-at, works-for , etc.
We assume that entities h and t are represented by time-independent d -dimensional vectors, respectively h , t ∈ R d .
We assume the task of generating a relational triple ( h, R, t ) in a given knowledge graph to be a two-step process as described next.
First, given the current knowledge vector at time k , c
=
c k and the relation R , we assume that the probability of an entity h satisfying the ﬁrst argument of R to be given by (1). [Equation]
[Equation]
Here, R 1 ∈ R d × d is a relation-speciﬁc orthogonal matrix that evaluates the appropriateness of h for the ﬁrst argument of R .
For example, if R is the CEO-of relation, we would require a person as the ﬁrst argument and a company as the second argument of R .
However, note that the role of Rextends beyond simply checking the types of the entities that can ﬁll the ﬁrst argument of a relation.
For our example above, not all people are CEOs and R 1 evaluates the likelihood of a person to be selected as the ﬁrst argument of the CEO-of relation.
Z c is a normalisation coefﬁcient such that (cid:80) h ∈V p ( h
| R, c ) = 1 , where the vocabulary V is the set of all entities in the knowledge graph.
After generating h , the state of our random walker changes to c (cid:48)
=
c k +1 , and we next generate the second argument of R with the probability given by (2).
[Equation]
Here, R 2 ∈ R d × d is a relation-speciﬁc orthogonal matrix that evaluates the appropriateness of t as the second argument of R .
Z c (cid:48) is a normalisation coefﬁcient such that (cid:80) t ∈V p ( t | R, c )
= 1 .
Following our previous example of the CEO-of relation, R 2 evaluates the likelihood of an organisation to be a company with a CEO position.
Importantly, R 1 and R 2 are representations of the relation R and independent of the entities.
Therefore, we consider ( R 1 and R 2 ) to collectively represent the embedding of R .
Orthogonality of R 1 , R 2 is a requirement for the mathematical proof and also act as a regularisation constraint to prevent overﬁtting by restricting the relational embedding space.
We ﬁrst perform our mathematical analysis for relational embeddings represented by orthogonal matrices and discuss later how this requirement can be relaxed.
We assume a slow random walk where the knowledge vectors do not change signiﬁcantly between consecutive time steps ( c k ≈ c k +1 ).
More speciﬁcally, we assume that (cid:107) c
k − c k +1 (cid:107) ≤ (cid:15) 2 for some small (cid:15) 2 > 0 .
This is a realistic assumption for generating the two entity arguments in the same relational triple because, if the knowledge vectors were signiﬁcantly different in the two generation steps, then it is likely that the corresponding relations are also different, which would not be coherent with the above-described generative process.
Moreover, we assume that the knowledge vectors are distributed uniformly in the unit sphere and denote the distribution of knowledge vectors by C .
To learn KGEs, we must estimate the probability that h and t satisfy the relation R , p ( h, t | R ) , which can be obtained by taking the expectation of p ( h, t | R, c , c (cid:48) )
w.r.t.
c , c (cid:48) ∼ C given by (3).
[Equation]
Here, partition functions are given by Z c
[Equation]
and
[Equation]
c [Equation] (cid:1) . (4) follows from our two-step generative process where the generation of h and t in each step is independent given the relation and the corresponding knowledge vectors.
Computing the expectation in (5) is generally difﬁcult because of the two partition functions Z c and
Z c (cid:48) .
However, Lemma 1 shows that the partition functions are narrowly distributed around a constant value for all c (or c (cid:48) )
values with high probability.
Lemma 1 ( Concentration Lemma ) .
If the entity embedding vectors satisfy the Bayesian prior v = s ˆ v , where ˆ v is from the spherical Gaussian distribution, and s is a scalar random variable, which is always bounded by a constant κ , then the entire ensemble of entity embeddings satisﬁes that
[Equation]
for (cid:15) z
= O (1 / √ n ) , and δ = exp( − Ω(log 2 n )) , where n ≥ d is the number of words and Z c is the partition function for c given by (cid:80) c
∈V exp (cid:0) h (cid:62) R 1 c (cid:1) .
proof: To prove the concentration lemma, we show that the mean E h [ Z c ] of Z c is concentrated around a constant for all knowledge vectors c and its variance is bounded.
Recall that
[Equation]
If P is an orthogonal matrix and x is a vector, then (cid:107) P (cid:62) x
(cid:107) {+
+} = ( P (cid:62) x [Equation] (cid:62)
[Equation]
P
[Equation]
PP
[Equation] x = [Equation] x
[Equation]
22 , because P (cid:62) P = I .
Therefore, from (7) and the orthogonality of the relational embeddings, we see that R 1 c is a simple rotation of c and does not alter the length of c .
We represent h
= s h
ˆ h , where s h
= (cid:107) h (cid:107) and ˆ h is a unit vector (i.e. (cid:107) ˆ h (cid:107) 2 = 1 ) distributed on the spherical Gaussian with zero mean and unit covariance matrix I d ∈ R d × d . Let s be a random variable that has the same distribution as s h .
Moreover, let us assume that s is upper bounded by a constant κ such that s ≤ κ .
From the assumption of the knowledge vector c , it is on the unit sphere as well, which is then rotated by R 1 .
We can write the partition function using the inner-product between two vectors h and R 1 c , Z c = (cid:80) h ∈V exp (cid:0) h (cid:62) ( R 1 c ) (cid:1) .
Arora et al.
(2016a) showed that (Lemma 2.1 in their paper) the expectation of a partition function of this form can be approximated as follows:
[Equation]
where n = |V| is the number of entities in the vocabulary.
(8) follows from the expectation of a sum and the independence of h and R 1 from c .
The inequality of (9) is obtained by applying the Taylor expansion of the exponential series and the ﬁnal equality is due to the symmetry of the spherical
Gaussian.
From the law of total expectation, we can write
[Equation]
(10) where, x = h (cid:62) R 1 c .
Note that conditioned on s h , h is a Gaussian random variable with variance σ 2 = sh .
Therefore, conditioned on s h , x is a random variable with variance σ 2 = σ 2 h .
Using this distribution, we can evaluate E x | s h (cid:2) exp (cid:0) h (cid:62) R 1 c
(cid:1)(cid:3) as follows:
[Equation]
Therefore
[Equation]
it follows that
[Equation]
where s is the variance of the (cid:96) 2 norms of the entity embeddings.
Because the set of entities is given and ﬁxed, both n and σ are constants, proving that E [ Z c ] does not depend on c .
Next
[Equation]
we calculate the variance V c
[ Z c ] as follows:
[Equation]
Because 2 h (cid:62) R 1 t is a Gaussian random variable with variance 4 σ 2 = 4 s 2 h from a similar calculation as in (11) we obtain,
[Equation]
By substituting (16) in (15) we have that
[Equation]
for Λ = exp(8 κ 2 ) a constant bounding s ≤ κ as stated.
From above, we have bounded both the mean and variance of the partition function by constants that are independent of the knowledge vector.
Note that neither exp (cid:0) h (cid:62) R 1 c (cid:1) nor exp (cid:0) t (cid:62) R 2 c (cid:48) (cid:1) are subGaussian nor sub-exponential.
Therefore, standard concentration bounds derived for sub-Gaussian or sub-exponential random variables cannot be used in our analysis.
However, the argument given in
Appendix A.1 in Arora et al.
(2016b)
for a partition function with bounded mean and variance can be directly applied to Z c in our case, which completes the proof of the concentration lemma.
From the symmetry between h and t , Lemma 1 also applies for the partition function (cid:80)
[Equation]
R [Equation] c [Equation]
Under the conditions required to satisfy Lemma 1, the following main theorem of this paper holds:
Theorem 1.
Suppose that the entity embeddings satisfy (1) .
Then, we have
[Equation]
[Equation]
for (cid:15) =
O (1 / √ n ) + (cid:101) O
(1 /d ) , where
[Equation]
The complete proof of Theorem 1 is given in Appendix A. Below we brieﬂy sketch the main steps.
Proof sketch: Let F be the event that both c and c (cid:48) are within (1 ± (cid:15)
z ) Z .
Then, from Lemma 1 and the union bound, event F happens with probability at least 1 − 2 exp( − Ω(log 2 n )) .
The R.H.S. of (5) can be split into two parts T 1 and T 2 according to whether F happens or not.
[Equation]
T 1 can be approximated as given by (21).
[Equation] [Equation]
On the other hand, T 2 can be shown to be a constant, independent of d , given by (22).
[Equation]
The vocabulary size n of real-world knowledge graphs is typically over 10 5 , for which T 2 becomes negligibly small.
Therefore, it sufﬁces to consider only T 1 .
Because of the slowness of the random walk we have c ≈ c (cid:48)
Using the law of total expectation we can write T 1 as follows:
[Equation]
where
A ( c ) :
= E c (cid:48) | c (cid:2) exp (cid:0) t (cid:62) R 2 c [Equation] (cid:1)(cid:3) .
Doing some further evaluations we show that
[Equation]
Plugging (50) back in (23) provides the claim of the theorem.
The relationship given by (18) indicates that head and tail entity embeddings are ﬁrst transformed respectively by R 1 (cid:62) and R 2 (cid:62) , and the squared (cid:96) 2 norm of the sum of the transformed vectors is proportional to the probability p ( h, t | R ) .
L EARNING K NOWLEDGE G RAPH E MBEDDINGS
In this section, we derive a training objective from Theorem 1 that we can then optimise to learn KGE.
The goal is to empirically validate the theoretical result by evaluating the learnt KGEs.
Knowledge graphs represent information about relations between two entities in the form of relational triples .
The joint probability p ( h, R, t ) given by Theorem 1 is useful for determining whether a relation R exists between two given entities h and t .
For example, if we know that with a high probability that R holds between h and t , then we can append ( h, R, t ) to the knowledge graph.
The task of expanding knowledge graphs by predicting missing links between entities or relations is known as the link prediction problem (Trouillon et al., 2016).
In particular, if we can automatically append such previously unknown knowledge to the knowledge graph, we can expand the knowledge graph and address the knowledge acquisition bottleneck.
To derive a criteria for determining whether a link must be predicted among entities and relations, let us consider a relational triple ( h, R, t ) ∈ D that exists in a given knowledge graph D .
We call such relational triples as positive triples because from the assumption it is known that R holds between h and t .
On the other hand, consider a negative relational triple ( h (cid:48) , R, t (cid:48) )
∈ D formed by, for example, randomly perturbing a positive triple.
A popular technique for generating such (pseudo) negative triples is to replace h or t with a randomly selected different instance of the same entity type.
As an alternative for random perturbation, Cai and Wang (2018) proposed a method for generating negative instances using adversarial learning.
Here, we are not concerned about the actual method used for generating the negative triples but assume a set of negative triples, ¯ D , generated using some method, to be given.
Given a positive triple ( h, R, t ) ∈ D and a negative triple ( h (cid:48) , R, t (cid:48) )
∈ ¯ D , we would like to learn
KGEs such that a higher probability is assigned to ( h, R, t ) than that assigned to ( h (cid:48) , R, t (cid:48) ) .
We can formalise this requirement using the likelihood ratio given by (25).
[Equation]
Here, η > 1 is a threshold that determines how higher we would like to set the probabilities for the positive triples compares to that of the negative triples.
By taking the logarithm of both sides in (25) we obtain
[Equation]
If a positive triple ( h, R, t ) is correctly assigned a higher probability than a negative triple p ( h (cid:48) , R, t (cid:48) ) , then the left hand side of (26) will be negative, indicating that there is no loss incurred during this classiﬁcation task.
Therefore, we can re-write (26) to obtain the marginal loss Bordes et al.
(2013; 2011), L ( D , ¯ D ) , a popular choice as a learning objective in prior work in KGE, as shown in (27). [Equation]
[Equation]
We can assume 2 d log η to be the margin for the constraint violation.
Theorem 1 requires R 1 and R 2 to be orthogonal.
To reﬂect this requirement, we add two (cid:96)regularisation terms (cid:107) R 1 (cid:62) R 1 − I (cid:107) 22 and (cid:107) R 2 (cid:62) R 2 − I (cid:107) 22 respectively with regularisation coefﬁcients λ 1 and λ 2 to the objective function given by (27).
In our experiments, we compute the gradients (27) w.r.t.
each of the parameters h , t , R 1 and R 2 and use stochastic gradient descent (SGD) for optimisation.
This approach can be easily extended to learn from multiple negative triples as shown in Appendix B.
R ELATED W ORK
At a high-level of abstraction, KGE methods can be seen as differing in their design choices for the following two main problems: (a) how to represent entities and relations, and (b) how to model the interaction between two entities and a relation that holds between them.
Next, we brieﬂy discuss prior proposals to those two problems (refer (Wang et al., 2017; Nickel et al., 2015; Nguyen, 2017) for an extended survey on KGE).
A popular choice for representing entities is to use vectors, whereas relations have been represented by vectors, matrices or tensors.
For example, TransE (Bordes et al., 2011), TransH (Wang et al., 2014),
TransD (Ji et al., 2015), TransG (Xiao et al., 2016), TransR (Lin et al., 2015), lppTransD (Yoon et al., 2016), DistMult (Yang et al., 2015), HolE (Nickel et al., 2016) and ComplEx (Trouillon et al., 2016) represent relations by vectors, whereas Structured Embeddings (Bordes et al., 2011), TranSparse (Ji et al., 2016),
STransE (Nguyen et al., 2016), RESCAL (Nickel et al., 2011) use matrices and Neural Tensor Network (NTN) (Socher et al., 2013) uses 3D tensors.
ComplEx (Trouillon et al., 2016) introduced complex vectors for KGEs to capture the asymmetry in semantic relations.
(Ding et al., 2018) obtained state-of-the-art performance for KGE by imposing non-negativity and entailment constraints to ComplEx.
Given entity and relation embeddings, a scoring function is deﬁned that evaluates the strength of a relation R between two entities h and t in a triple ( h, R, t ) .
The scoring functions that encode various intuitions have been proposed such as the (cid:96) 1 or (cid:96) 2 norms of the vector formed by a translation of the head entity embedding by the relation embedding over the target embedding, or by ﬁrst performing a projection from the entity embedding space to the relation embedding space (Yoon et al., 2016)
As an alternative to using vector norms as scoring functions, DistMult and ComplEx use the component-wise multi-linear dot product.
Once a scoring function is deﬁned, KGEs are learnt that assign better scores to relational triples in existing knowledge graphs (positive triples) over triples where the relation does not hold (negative triples) by minimising a loss function such as the logistic loss (RESCAL, DistMult, ComplEx) or marginal loss (TransE, TransH, TransD, TransD).
Because knowledge graphs record only positive triples, a popular method to generate pseudo negative triples is to perturb a positive instance by replacing its head or tail entity by an entity selected uniformly at random from the vocabulary of the entities.
However, uniformly sampled negative triples are likely to be obvious examples that do not provide much information to the learning process and can be detected by simply checking for the type of the entities in a triple.
Cai and Wang (2018) proposed an adversarial learning approach where a generator assigns a probability to each relation triple and negative instances are sampled according to this probability distribution to train a discriminator that discriminates between positive and negative instances.
(Xiao et al., 2016) proposed TransG, a generative model based on the Chinese restaurant process, to model multiple relations that exist between a pair of entities.
However, their relation embeddings are designed to satisfy vector translation similar to TransE.
As an alternative to directly learning embeddings from a graph, several methods (Grover and Leskovec, 2016; Perozzi et al., 2014; Ristoski et al., 2018) have considered the vertices visited during truncated random walks over the graph as pseudo sentences , and have applied popular word embedding learning algorithms such as skip-gram with negative sampling or continuous bag-of-words model (Mikolov et al., 2013) to learn vertex embeddings.
However, pseudo sentences generated this way are syntactically very different from sentences in natural languages.
On the other hand, our work extends the random walk analysis by Arora et al.
(2016a) that derives a useful connection between the joint co-occurrence probability of two words and the (cid:96) 2 norm of the sum of the corresponding word embeddings.
Speciﬁcally, they proposed a latent variable model where the words in a corpus are generated by a probabilistic model parametrised by a time-dependent discourse vector that performs a random walk.
However, unlike in our work, they do not consider the relations between two co-occurring words in a corpus.
Bollegala et al.
(2018) extended the model proposed by Arora et al.
(2016a) to capture co-occurrences involving more than two words.
They deﬁned the co-occurrence of k unique words in a given context as a k -way co-occurrence, where Arora et al.
(2016a)’s result could be seen as a special case corersponding to k = 2 .
Moreover, Bollegala et al.
(2018) showed that it is possible to learn word embeddings that capture some types of semantic relations such as antonymy and collocation using 3-way co-occurrences more accurately than using 2-way co-occurrences.
However, their model does not explicitly consider the relations between words/entities and uses only a corpus for learning the word embeddings. [Table] [Table]
E MPIRICAL VALIDATION
To empirically evaluate the theoretical result stated in Theorem [Table] we learn KGEs (denoted by
RelWalk ) by minimising the marginal loss objective derived in section 3.
We use the FB15k, FB (subsets of Freebase ) and WN11, WN18, WN18RR (subsets of WordNet ) datasets, which are standard benchmarks for KGE.
We use the standard training, validation and test splits as detailed in Table [Table]
We generate negative triples by replacing a head or a tail entity in a positive triple by a randomly selected different entity and learn KGEs.
We train the model until convergence or at mostepochs over the training data where each epoch is divided into 100 mini-batches.
The best model is selected by early stopping based on the performance of the learnt embeddings on the validation set (evaluated after each 20 epochs).
The training details and hyperparameter settings [-were: λ 
 = λ-]are detailed in
Appendix C. RelWalk is implemented in the open-source toolkit OpenKE (Han et al., 2018).
We conduct two evaluation tasks: link prediction (predict the missing head or tail entity in a given triple ( h, R, ?) or (? , R, t ) )
(Bordes et al., 2011) and triple classiﬁcation (predict whether a relation R holds between h and t in a given triple ( h, R, t ) )
(Socher et al., 2013).
We evaluate the performance in the link prediction task using mean rank ( MR , the average of the rank assigned to the original head or tail entity in a corrupted triple) and H@10 ( hits@10 , percentage of the test cases where the original head or tail entity was found among the top-10 ranked entities), whereas in the triple classiﬁcation task we use accuracy (percentage of the correctly classiﬁed test triples).
We only report scores under the ﬁltered setting Bordes et al.
(2013), which removes all triples appeared in training, validating and testing sets from candidate triples before obtaining the rank of the ground truth triple.
In link prediction, we consider all entities that appear in the corresponding argument in the entire knowledge graph as candidates.
In Tables 2 and 3 we compare the KGEs learnt by RelWalk against prior work using the published results.
For link prediction, RelWalk reports SoTA on WN18, WN18RR and FB15K237 on both
MR and H@10, except in FB15K. WN18RR excludes triples from WN18 that are simply inverted between train and test partitions (Toutanova and Chen, 2015; Dettmers et al., 2017).
RelWalk ’s consistently good performance on both versions of this dataset shows that it is considering the global structure in the knowledge graph when learning KGEs. For triple classiﬁcation, RelWalk reports the best performance on FB13, whereas TransG reports the best performance on WN11. Considering that both TransG and RelWalk are generative models, it would be interesting to further investigate generative approaches for KGE in the future.
Overall, the experimental results support our theoretical claim and emphasise the importance of theoretically motivating the scoring function design process.
C ONCLUSION
We proposed RelWalk , a generative model of KGE and derived a theoretical relationship between the probability of a triple and entity, relation embeddings.
We then proposed a learning objective based on the theoretical relationship we derived.
Experimental results on a link prediction and a triple classiﬁcation tasks show that RelWalk obtains strong performances in multiple benchmark datasets.
A PPENDIX A P ROOF OF T HEOREM
Let us consider the probabilistic event that (1 − (cid:15) z )
Z ≤ Z c ≤ (1 + (cid:15) z ) Z to be F c and (1 − (cid:15) z ) Z ≤
Z c (cid:48) ≤ (1 + (cid:15) z )
Z to be F c (cid:48) .
From Lemma 1 we have Pr c [ F c ] ≥ 1 − δ .
Then from the union bound we have,
[Equation]
Moreover, let F be the probabilistic event that both F c and F c (cid:48) being True.
Then from Pr[ F ] = 1 − Pr[ ¯ F c ∨ ¯ F c (cid:48) ] we have, Pr[ F ] ≥ 1 − 2 δ .
We can decompose the expectation in the R.H.S. in (5) into two terms T 1 and T 2 depending on whether respectively F is True or False as follows:
[Equation]
[Equation]
Here, 1 F and 1 ¯ F are indicator functions given by:
[Equation]
30) and
[Equation]
if F is True
[Equation]
otherwise
[Equation]
Let us ﬁrst show that T 2 is negligibly small.
For two real integrable functions ψ 1 ( x ) and ψ 2 ( x ) in [ a
[Equation]
b ]
[Equation]
the Cauchy-Schwarz’s inequality states that
[Equation]
[Equation]
Applying (32) to T 2 in (29) we have:
[Equation]
Note that Z c ≥ 1 because Z c is the sum of positive numbers and if h (cid:62) R 1 c ≥ 0 for at least one of the h ∈ V , then the total sum will be greater than 1.
Therefore, by dropping Z c term from the denominator we can further increase the ﬁrst term in (33) as given by (34).
[Equation]
[Equation]
Let us split the expectation on the R.H.S. of (34) into two cases depending on whether h (cid:62) R 1 c >or otherwise, indicated respectively by 1 ( h (cid:62) R 1 c > 0) and 1 ( h (cid:62) R 1 c ≤ 0) .
[Equation] [Equation]
The second term of (35) is upper bounded by
[Equation] [Equation]
The ﬁrst term of (35) can be bounded as follows:
[Equation] [Equation]
where α > 1 .
Therefore, it is sufﬁcient to bound E c (cid:104)
[Equation]
when (cid:107) h (cid:107) =
Ω( √ d ) .
Let us denote by z the random variable 2 h (cid:62) R 1 c .
Moreover, let r ( z ) =
E c (cid:48) | z
[ 1 ¯ F ] , which is a function of z between [0 , 1] .
We wish to upper bound E c [exp( z ) r ( z )] .
The worst-case r ( z ) can be quantiﬁed using a continuous version of Abel’s inequality (proved as Lemma A.4 in Arora et al.
(2016b)), we can upper bound E c
[exp( z ) r ( z )] as follows:
[Equation]
38) where t satisﬁes that E c [ 1 [ t, + ∞ ] ( z )] =
Pr[ z ≥ t ] =
E c
[ r ( z )] ≤ exp( − Ω(log 2 n )) .
Here, 1 [ t, + ∞ ] ( z ) is a function that takes the value 1 when z ≥ t and zero elsewhere.
Then, we claim Pr c [ z ≥ t ] ≤ exp( − Ω(log 2 n )) implies that t ≥ Ω(log . 9 n ) .
If c was distributed as N (0 , 1 d I ) , this would be a simple tail bound.
However, as c is distributed uniformly on the sphere, this requires special care, and the claim follows by applying the tail bound for the spherical distribution given by Lemma A.1 in (Arora et al., 2016a) instead.
Finally, applying
Corollary A.3 in (Arora et al., 2016a), we have:
[Equation]
From a similar argument as above we can obtain the same bound for c (cid:48) as well.
Therefore, T 2 in (29) can be upper bounded as follows:
[Equation]
Because n = |V| , the size of the entity vocabulary, is large (ca. n > 10 5 ) in most knowledge graphs, we can ignore the T 2 term in (29).
Combining this with (29) we obtain an upper bound for p ( h, t | R ) given by (41).
[Equation]
[Equation]
where |D| is the number of relational tuples ( h, R, t ) in the KB D and δ 0 = |D| exp( − Ω(log 1 . 8 n )) ≤ exp( − Ω(log 1 . 8 n )) by the fact that Z ≤ exp(2 κ ) n = O ( n ) , where κ is the upper bound on h (cid:62) R 1 c and t (cid:62) R 2 c (cid:48) , which is regarded as a constant.
On the other hand, we can lower bound p ( h, t | R ) as given by (42).
[Equation]
Taking the logarithm of both sides, from (41) and (42), the multiplicative error translates to an additive error given by (43).
[Equation]
where A ( c ) := E c
[Equation]
c (cid:2) exp (cid:0) t (cid:62) R 2 c
[Equation]
(cid:1)(cid:3) .
We assumed that c and c (cid:48) are on the unit sphere and R 1 and R 2 to be orthogonal matrices.
Therefore,
R 1 c and R 2 c (cid:48) are also on the unit sphere.
Moreover, if we let the upper bound of the (cid:96) 2 norm of the entity embeddings to be κ (cid:48) √ d , then we have (cid:107) h (cid:107) ≤ κ (cid:48) √ d and (cid:107) t
[Equation]
≤ κ
[Equation]
√ d .
Therefore, we have
[Equation] [Equation]
Then we can lower bound A ( c ) as follows:
[Equation] [Equation]
For some (cid:15) 2 > 0 .
The last inequality holds because
[Equation]
To obtain a lower bound on A ( c ) from the ﬁrst-order Taylor approximation of exp( x ) ≥ 1 + x we observe that
[Equation]
Therefore
[Equation]
from our model assumptions we have
[Equation]
Hence,
[Equation] [Equation]
Therefore
[Equation]
from (46) and (49) we have
[Equation]
) =
(1 ± (cid:15) 2 ) exp (cid:0) t (cid:62) R 2 c (cid:1) (50)
Plugging A ( c ) back in (43) we obtain log p ( h, t | R )
= log (cid:0) E c (cid:2) exp (cid:0)
h (cid:62) R 1 c (cid:1)
A ( c ) ±
δ 0 (cid:3)(cid:1) − 2 log Z + 2 log(1 ± (cid:15) z ) = log (cid:0) E c (cid:2) exp (cid:0) h (cid:62) R 1 c (cid:1) (1 ± (cid:15) 2 ) exp (cid:0) t (cid:62) R 2 c (cid:1) ± δ 0 (cid:3)(cid:1) − 2 log Z + 2 log(1 ± (cid:15) z ) (51) = log (cid:0) E c (cid:2) exp (cid:0) h (cid:62) R 1 c (cid:1)
exp (cid:0) t (cid:62) R 2 c (cid:1) ±
δ 0 (cid:3)(cid:1) − 2 log Z + 2 log(1 ± (cid:15) z )
+ log(1 ± (cid:15) 2 )
= log (cid:0) E c (cid:2) exp (cid:0) h (cid:62) R 1 c + t (cid:62)
R 2 c (cid:1) ±
δ 0 (cid:3)(cid:1) − 2 log Z + 2 log(1 ± (cid:15) z )
+ log(1 ± (cid:15) 2 )
= log (cid:0) E c (cid:2) exp (cid:0) R 1 (cid:62) h
+ R 2 (cid:62) t (cid:1) (cid:62) c ± δ 0 (cid:3)(cid:1) − 2 log Z + 2 log(1 ± (cid:15) z )
+ log(1 ± (cid:15) 2 )
Note that c has a uniform distribution over the unit sphere.
In this case, from Lemma A.5 in (Arora et al., 2016b), (52) holds approximately.
[Equation]
where (cid:15) 3 =
˜ O
(1 /d ) .
Plugging (52) in (51) we have that
[Equation]
where δ (cid:48)
= δ 0 · (cid:0) E c (cid:2) exp (cid:0) ( R 1 (cid:62) h
+ R
[Equation]
t )
[Equation]
c (cid:1)(cid:3)(cid:1) − 1 = exp( − Ω(log 1 . 8 n )) .
Therefore, δ (cid:48) 0 can be ignored.
Note that (cid:15) 3 = ˜ O (1 /d ) and (cid:15) z
= ˜ O
(1 / √ n ) by assumption.
Therefore, we obtain that
[Equation]
B L EARNING WITH MULTIPLE NEGATIVE TRIPLES
In this section, we show how the margin loss-based learning objective derived in section 3 can be extended to learn from more than one negative triples per each positive triple.
This formulation leads to rank-based loss objective used in prior work on KGE.
Considering that negative triples are generated via random perturbation, it is important to consider multiple negative triples during training to better estimate the classiﬁcation boundary.
Let us consider that we are given a positive triple, ( h, R, t ) and a set of K negative triples { ( h (cid:48)
k , R, t (cid:48) k ) }
Kk =1 .
We would like our model to assign a probability, p ( h, t | R ) , to the positive triple that is higher than that assigned to any of the negative triples.
This requirement can be written as (55).
[Equation]
We could further require the ratio between the probability of the positive triple and maximum probability over all negative triples to be greater than a threshold η ≥ 1 to make the requirement of (55) to be tighter.
[Equation]
By taking the logarithm of (56) we obtain
[Equation]
Therefore, we can deﬁne the margin loss for a misclassiﬁcation as follows:
[Equation]
However, from the monotonicity of the logarithm we have ∀ x 1 , x 2 > 0 , if log( x 1 ) ≥ log( x 2 ) then x 1 ≥ x 2 .
Therefore, the logarithm of the maximum can be replaced by the maximum of the logarithms in (58) as shown in (59).
[Equation]
By substituting (18) for the probabilities in (59) we obtain the rank-based loss given by (60).
[Equation]
In practice, we can use p ( h (cid:48) k , t (cid:48)
k | R ) to select the negative triple with the highest probability for training with the positive triple. [Table]
C T RAINING DETAILS
The statistics of the benchmark datasets are show in Table 4.
We selected the initial learning rate ( α ) for SGD in { 0 . 01 , 0 . 001 } , the regularisation coefﬁcients ( λ 1 , λ 2 ) for the orthogonality constraints of relation matrices in { 0 , 1 , 10 , 100 } .
The number of randomly generated negative triples n neg for each positive example is varied in { 1 , 10 , 20 , 50 , 100 } and d ∈ { 50 , 100 } .
{+Optimal hyperparameter settings were: λ 
+}
=
λ
2 = 10 , n neg = 100 for all the datasets, α = 0 .
001 for FB15K, FB15K237 and FB13, α = 0 .
01 for WN18, WN18RR and WN11.
For FB15K237 and WN18RR d = 100 was the best, whereas for all other datasets d = 50 performed best.