K NOWLEDGE G RAPH E MBEDDING : A P ROBABILISTIC P ERSPECTIVE AND G ENERALIZATION B OUNDS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study theoretical properties of embedding methods for knowledge graph completion under the “missing completely at random” assumption.
We prove generalization error bounds for this setting.
Even though the missing completely at random setting may seem naive, it is actually how knowledge graph embedding methods are typically benchmarked in the literature.
Our results provide, to a certain extent, an explanation for why knowledge graph embedding methods work (as much as classical learning theory results provide explanations for classical learning from i.i d. data).
I NTRODUCTION
Many large knowledge bases such as YAGO (Mahdisoltani et al., 2015) and NELL (Carlson et al., 2010) are represented as knowledge graphs .
A knowledge graph is a set of triples ( head , predicate , tail ) where head and tail are objects (also frequently called entities ) and predicate is a binary relation. Each such triple represents a fact about the world, e.g.
( NewYork , IsLocatedIn , US ) .
The objects can be seen as vertices of a graph and the relations as labeled directed edges (pointing from head to tail), hence the name knowledge graph.
Most knowledge bases are incomplete, i.e. there are facts that should be included in them but are not.
A popular task is to automatically complete such knowledge bases.
This is often done by embedding the objects and relations contained in them into some vector spaces and then using the embedding to predict the missing tuples.
There has been a number of knowledge graph completion methods presented in the literature (we give an overview of existing methods in Section 2.3).
However, to our best knowledge, there has not been any theoretical analysis of the generalization abilities of these methods in the spirit of classical learning theory.
We make the ﬁrst steps in this direction in the present paper.
We derive expected error bounds for a class of knowledge graph embedding methods.
A key challenge in deriving generalization bounds in this setting is that the training and test data do not satisfy assumptions typically used in learning theory.
The knowledge bases that we receive contain only positive examples, that is, the facts already in the knowledge base.
We do not know which of the facts not contained in the knowledge base are just missing and which are not contained in the knowledge base because they are actually false.
This is known as learning from positive and unlabeled data (PU-learning); we refer to (Bekker & Davis, 2018) for a recent survey.
Only a few results have been obtained for generalization bounds for PU-learning so far, e.g. (Niu et al., 2016), but they are not directly applicable to our setting (we discuss the details in Section 10).
The bounds that we derive in this paper rely on an assumption on how the triples are missing from the knowledge graph.
In particular, we assume that the triples are missing “completely at random”, i.e. that each tuple may be missing with probability δ > 0 independently of the others.
Even though this setting may appear naive, it is actually exactly how some famous datasets’ training and test sets are created, e.g. (Bordes et al., 2013).
Thus, the theoretical results actually do relate to what one observes in “practice” (i.e. in the respective papers that use these datasets).
The bounds that we derive here apply to knowledge graph embedding methods that use log-loss for learning the embeddings such as ComplEx (Trouillon & Nickel, 2017), SimplE (Kazemi & Poole, 2018) or
ConvE (Dettmers et al., 2018).
Main Technical Contributions
Our main technical results are expected error bounds on the number of triples ( h, r, t ) that are predicted incorrectly when using knowledge graph embedding for knowledge graph completion (stated in Theorems 4 and 5), i.e. either to be in the knowledge graph when they are not supposed to be (false positives) or not to be in the knowledge graph when they are supposed to be (false negatives).
The bound assumes that the knowledge graph embedding is learned by maximizing the log-likelihood of a certain exponential-family distribution (introduced in
Section 5), which turns out to be equivalent to how several prominent knowledge graph embedding methods (Trouillon & Nickel, 2017; Kazemi & Poole, 2018; Dettmers et al., 2018) are learned using log-loss (this is discussed in detail in Section 10). Our results therefore provide a statistical learning theory style justiﬁcation for these knowledge graph embedding methods. In order to obtain the main results we also derive several auxiliary technical results which might be of independent interest (in particular, Theorems 2 and 3).  P RELIMINARIES 2. K NOWLEDGE G RAPHS
Let O be a set of objects and R a set of relations.
A knowledge graph is a set of triples ( h, r, t ) where h and t are objects and r is a relation.
Here h is called the head of the triple and t is called the tail of the triple.
In the formalism of ﬁrst-order logic, a triple ( h, r, t ) represents a ﬁrst-order logic atom r ( h, t ) .
For instance the triple ( Alice , Knows , Bob ) corresponds to the ﬁrst-order logic atom
Knows ( Alice , Bob ) , which represents the fact that Alice knows Bob .
Here, Alice and Bob are objects and Knows is a relation.
Example 1.
Let us have the following objects and relations:
O = { France , Germany , Berlin , Paris } , R = { CapitalOf , Borders } .
Then the knowledge graph, representing our geographical knowledge about these countries and cities, can look as G =
{ ( France ,
Borders , Germany ) , ( Germany , Borders , France ) , ( Berlin , CapitalOf , Germany ) , ( Paris , CapitalOf ,
France ) } .
Whereas, in this case, the knowledge graph is complete and accurate, in reality most knowledge graphs will lack some of the true tuples.
The task of knowledge base completion is to add such missing facts.
2. K NOWLEDGE G RAPH E MBEDDING
Knowledge graph embedding methods represent knowledge graphs geometrically.
They do so by representing objects and relations as vectors (or matrices).
Given a knowledge graph G on some sets of objects O and relations R , most knowledge graph embedding methods need three components: vector representations of objects, vector representations of relations and a function ψ ( x h , x r , x t ) which takes the representations of h , r and t and produces a score that is used to assess how likely it is that the relation ( h, r, t ) is contained in the knowledge graph.
One can then compare the output of the function with a threshold to decide which triples are (predicted to be) in the knowledge graph.
Note on notation.
For a set of objects O and a set of relations R we will normally denote their vector representations by X .
We will usually treat X as a look-up table.
For an object o we will typically use x o to denote its vector representation from X and analogically for a relation r we will use x r , unless explicitly stated otherwise.
With a slight abuse of notation, we will also sometimes treat X as a set of vectors and, for instance, write x ∈ X .
Relations can be mapped to matrices (denoted M r ) or tensors instead of only vectors, and even if relations are mapped to vectors the dimension may be different from the dimension of entity embeddings.
However, for the sake of simplicity, we will not introduce more notations, and X and ψ will also be used in these cases.
2. E XISTING M ETHODS
One can roughly categorize KG embedding approaches into several groups based on different scoring functions such as distance-based scoring functions and similarity-based scoring functions.
The most well-known class of embedding methods using distance-based scoring functions is TransE (Bordes et al., 2013) and its variants.
The scoring function of TransE is deﬁned as the negative distance between x h + x r and x t , i.e., ψ ( x h , x r , x t )
=
−|| x h + x r − x t || 2 .
In order to effectively model more general relations rather than only one-to-one relations, TransH has been proposed (Wang et al., 2014).
Given a triple ( h, r, t ) , the scoring function is deﬁned as −|| x ⊥ h + x r − x ⊥ t || 2 where x ⊥ h and x ⊥ t are the projections of entity embeddings x h and x t onto some hyperplane, respectively.
TransR (Lin et al., 2015), TransD (Ji et al., 2015) and TranSparse (Ji et al., 2016) are similar to TransH, allowing relation-speciﬁc embeddings in different ways.
TransM (Fan et al., 2014; Xiao et al., 2015), ManifoldE (Xiao et al., 2016) and TransF (Feng et al., 2016) relax the requirement x h + x r ≈ x t .
SE (Bordes et al., 2011) uses two matrices to project head and tail entities for each relation r , and the score is −|| M 1 r x
h − M 2 r x t || 1 .
RotatE (Sun et al., 2019) deﬁnes each relation as a rotation from the source entity to the target entity in the complex vector space, and its scoring function is −|| x h ◦ x r − x t || 2 where ◦ is the entry-wise product.
In the class of similarity-based methods, RESCAL (Nickel et al., 2011) and its variants are among the most well-known ones.
RESCAL also assigns every entity a vector but every relation r a matrix
M r 1 .
The scoring function is deﬁned as x (cid:62) h M r x t .
Jenatton et al.
(2012) further assumes that every
M r has the same set of eigenvectors.
Garc´ıa-Dur´an et al.
(2014) adds other terms into the scoring function x (cid:62)
h M r x t + x (cid:62)
h x r + x (cid:62) r x t + x (cid:62) t D x h where D is a diagonal matrix independent of relations.
DistMult (Yang et al., 2015) is a simpliﬁed version of RESCAL by restricting M r being diagonal, and hence improves the efﬁciency of RESCAL.
To combine the advantages of both methods, i.e., the expressive power of RESCAL and the simplicity of DistMult, HolE (Nickel et al., 2016) has been proposed.
ComplEx (Yang et al., 2015) extends DistMult by allowing complex-valued embeddings, which is more powerful in modeling asymmetric relations.
Hayashi & Shimbo (2017) show that ComplEx is in fact equivalent to HolE. To capture analogical structures, ANALOGY (Liu et al., 2017) further requires M r in RESCAL to be normal and mutually commutative.
A class of KG embeddings is based on tensor factorization approaches such as the Canonical
Polyadic (CP) decomposition (Hitchcock, 1927) whose scoring function is similar to DistMult, but every entity has two independent embeddings.
SimplE
(Kazemi & Poole, 2018) is a recent extension of CP decomposition, which ensures the two embeddings of every entity dependent.
There are also a group of algorithms using neural networks for embedding, e.g., SME (Bordes et al., 2014), NTN and SLM (Bordes et al., 2014), MLP (Dong et al., 2014), NAM (Liu et al., 2016) and
ConvE (Dettmers et al., 2018).
All the aforementioned embedding methods map entities and relations in a deterministic way.
In
KG2E (He et al., 2015), representations of entities and relations are random vectors sampled from multivariate normal distributions,
[Equation]
A possible scoring function is KL ( x t − x h
|| x r ) and another is the probability inner product (Jebara et al., 2004) of x t − x h and x r .
TransG (Han et al., 2016) further models relations using mixtures of Gaussians.
L EARNING S ETTING
We work within the following learning setting.
We assume that there exists some “ground truth” knowledge graph G ∗ and that we observe only a subset of the triples contained in it.
We will denote the observed knowledge graph consisting of a subset of G ∗ ’s triples by (cid:98) G .
Furthermore, we assume that (cid:98) G is obtained from G ∗ by the following sampling process, called data-generating distribution .
Deﬁnition 1 (Data-generating distribution) .
Let G ∗ be a ground-truth knowledge graph and δ >be a real number.
The data-generating distribution is given by the following process.
Iterate over all triples τ ∈ G ∗ and add each of them to (cid:98) G with probability 1 − δ independently of the others.
We call δ the missingness parameter .
The task that we want to solve is then to reconstruct the ground-truth knowledge graph G ∗ as well as possible from a single sample from the data-generating distribution.
Remark 1.
Some knowledge graph embedding methods use different representations for the same object depending on whether it is used as a head or tail of a particular triple.
It is easy to see that our general setting also allows such representations;it is enough to double the dimension of the vectors and assume that the function ψ ( x h , x r , x t ) depends only on the ﬁrst half of dimensions of x h and the second half of the dimensions of x t .
It is easy to see that we can also handle the case when the vector representations of relations have different dimensions than those of objects similarly.
I NTUITION AND S KETCH OF THE A RGUMENT
Our main results are bounds for the expected number of incorrectly predicted triples for knowledge graph completion.
In particular, we show that, assuming certain technical assumption on the knowledge graph embedding method and the way it is trained, there exists a constant C such that the expected error rate in the realizable setting is bounded by C ( (cid:112) |G ∗ | / ( |R||O| 2 ) · (cid:112) ln |O| / |O| +1 / |O| ) , where G ∗ is the ground truth knowledge graph and O is the set of objects.
The ﬁrst term in the product measures sparsity of the graph and the second term has the familiar form of classical error bounds for learning from i.i.d.
data.
We also provide similar bounds for the agnostic setting.
The strategy taken in this paper is based on viewing the problem as learning a distribution on knowledge graphs.
Suppose that we have not only one sample (cid:98) G (one knowledge graph) from the datagenerating distribution, but actually many such samples (cid:98) G 1 , (cid:98) G 2 , . . .
, (cid:98) G n (a collection of knowledge graphs instead of just one).
[-Then we could produce a prediction for the ground-truth knowledge graph by taking the union (cid:98) G 1 ∪ (cid:98) G 2 ∪ · · · ∪ (cid:98) G n . 
-]
We could try to use the knowledge graphs (cid:98) G 1 , (cid:98) G 2 , . . .
, (cid:98) G n to estimate a distribution over knowledge graphs.
Then we would predict the triples with frequency higher than some threshold t as true.
In reality, we do not have a collection of knowledge graphs but only one.
However, to estimate the data-generating distribution, we can exploit the fact that we know its form and the independencies it satisﬁes.
In the next section, we deﬁne a distribution to approximate the data-generating distribution, parameterized by the vector representations X of objects and relations.
For the approach sketched above to work, we need (i) the distribution to be expressive enough and (ii) we need to show that the learned distribution will be close enough to the data-generating distribution if we estimate it from a sufﬁciently large knowledge graph.
The question of expressiveness has been tackled in the literature (e.g. Kazemi & Poole (2018)) and we do not deal with it much in this paper.
We deal with the second question.
Using a concentration inequality for log-likelihood (Theorem 1), we prove a generalization bound for log-likelihood (Theorem 2) from which a bound on KullbackLeibler divergence follows.
What then remains to be done is to show how to bound the number of incorrectly predicted triples as a function of the Kullback-Leibler divergence, which we do with the help of Pinsker’s inequality (Theorem 3).
Combining these two latter results together then yields a bound on the expected number of triples that are predicted incorrectly (both false positives and false negatives) by the learning method sketched here (Theorem 4).
A P ROBABILISTIC P ERSPECTIVE
Following the intuition described in the previous section, we cast the knowledge graph embedding problem as the problem of estimating the distribution induced by the data-generating process.
We seek an approximation of this distribution as an exponential-family model of the form given below in Deﬁnition 2.
The parameters of this exponential family model are the vector embeddings of the objects and relations.
Deﬁnition 2 (KG-distribution) .
Let O be a set of objects, R be a set of relations and Ω = 2 O×R×O be the set of all possible knowledge graphs on these objects and relations.
Let G ∈ Ω be a knowledge graph and X be the vector representation of the objects and relations.
Then we deﬁne [Equation]
where x h , x r and x t are the vector representations of h , r and t given by X , ψ : R d × R d × R d is a function, d is the dimension of the vector representations, and ﬁnally Z = [Equation] exp
[Equation] ψ [Equation]
x [Equation] x r , x t )) is a normalization constant, ensuring that P X ( G ) is a probability distribution.
Unless we allow | ψ | to be inﬁnite, KG-distributions cannot model the data-generating distribution exactly.
However, as we will show it is enough if the KG-distributions are close enough in KullbackLeibler divergence. Remark 2. The KG-distribution from Deﬁnition 2 can equivalently be written as [Equation]
G ENERALIZATION B OUNDS FOR L OG -L IKELIHOOD
In this section, we derive generalization bounds for log-likelihood of KG-distributions learned from one sample (i.e. knowledge graph) from the data-generating distribution described in Section 3.
First, we deﬁne log-likelihood and normalized log-likelihood of the KG-distribution model.
Deﬁnition 3 (Normalized log-likelihood) .
Let G be a knowledge graph on a set of objects O and a set of relations R and X be vector representations of the objects and relations.
The log-likelihood of this model is L ( X |G ) =
ln P X ( G ) and its normalized log-likelihood is then deﬁned as NL ( X |G )
=
|O| 2 |R| · L ( X |G ) .
We start by proving a concentration inequality for the log-likelihood of KG-distributions, which is a rather straightforward consequence of McDiarmid’s inequality.
Theorem 1.
Let G ∗ be the ground-truth knowledge graph.
Let G be sampled from the datagenerating distribution induced by G ∗ and X be ﬁxed vector representations.
Then
[Equation]
where M ≥ sup h , t , r ∈ X | ψ ( x h , x r , x t ) | (we assume sup h , t , r ∈ X | ψ ( x h , x r , x t ) | to be ﬁnite) and the probability P [ . . . ] and the expectation E [ . .
. ] are w.r.t.
the sampling of knowledge graphs from the data-generating distribution.
The next theorem is the main result of this section.
It bounds the expectation of the maximum deviation of the log-likelihood of KG distributions from their mean, where the KG-distributions are selected from a hypothesis class H (which has to satisfy certain assumptions listed in the theorem).
Theorem 2.
Let G ∗ be a ground-truth knowledge graph on a set of objects O and a set of relations
R and let G be sampled by the data-generating distribution.
Let X ⊆ R d .
Finally, let H X (“hypothesis class”) denote the set of vector representations X , X ⊆ X |O| + |R| , of the objects from O and relations from R .
Let D , M and K be real numbers satisfying:
∀ x , x (cid:48) ∈
X : (cid:107) x − x (cid:48) (cid:107) ≤ D , 2.
∀ x h , x r , x t ∈ X : | ψ ( x h , x r , x t ) | ≤ M , 3.
∀ x h , x r , x t , x (cid:48)
h , x (cid:48)
r , x (cid:48) t ∈ X : (cid:107) ψ
( x h , x r , x t ) − ψ ( x (cid:48) h , x (cid:48)
r , x (cid:48) t ) (cid:107) ≤ K ( (cid:107) x
h − x (cid:48) h (cid:107) + (cid:107) x r − x (cid:48) r (cid:107)
[Equation]
Then the following holds:
[Equation]
where both expectations on the l.h.s.
are w.r.t.
sampling of knowledge graphs from the datagenerating distribution.
P INSKER ’ S I NEQUALITY C OMES INTO P LAY
In this section, we use Pinsker’s inequality to show how the Kullback-Leibler divergence between the data-generating distribution, which generates the observed knowledge graph, and the learned distribution can be used to bound the number of incorrectly predicted triples .
In particular, we assume that all triples with probability given by the learned distribution greater than a ﬁxed threshold t are predicted as positive (i.e. predicted to belong in the knowledge graph) and the rest as negative (i.e. predicted not to belong in the knowledge graph).
An incorrectly predicted triple is then either a triple which is included in the ground-truth knowledge graph
but we predict it as false (false negative) or a triple which is not included in the ground-truth knowledge graph
but we predict it as true (false positive).
In combination with the results from the previous section, this will allow us to bound the expected number of incorrectly predicted triples (this is explained in detail in the next section).
Theorem 3, which is the main result of this section, bounds the number of incorrectly predicted triples as a function of the Kullback-Leibler divergence of the data-generating and the learned distributions, of the threshold t (used to decide which triple is predicted as positive) and the missingness parameter δ (the fraction of the triples missing from the ground-truth knowledge graph). The main idea of the proof of this theorem (which is given in the appendix) is as follows. Whenever there is an incorrectly predicted triple τ , the difference between its probability w.r.t. the data-generating distribution and w.r.t.
the learned distribution must be greater than some ﬁxed ∆ (the value of ∆ depends on the threshold t and the missingness parameter δ ), otherwise τ could not be incorrectly predicted.
Pinsker’s inequality (Pinsker, 1964), which relates total variation distance and KullbackLeibler divergence, then allows us to obtain a lower bound on the Kullback-Leibler divergence of the data-generating and the learned marginal distributions corresponding to the incorrectly predicted triple τ . Since triples are sampled independently of each other in both the data-generating and the learned distributions, the sum of the Kullback-Leibler divergences of the marginal distributions corresponding to all possible individual triples will equal the Kullback-Leibler divergence of the datagenerating distribution and the learned distribution. This all together is what allows us to derive the bound stated in the next theorem. Theorem 3.
Let G ∗ be the ground-truth graph on a set of objects O and relations R .
Let P denote the data-generating distribution induced by G ∗ with the missingness parameter δ and let Q denote the learned distribution.
Let t ∈ (0; 1 − δ ) be a threshold for predicting which triples belong to the knowledge graph. Finally, let F be the set of incorrectly predicted triples, i.e. the set of all triples τ ∈ O×R×O such that either τ ∈ G ∗ and the probability of τ given by the learned distribution Q is smaller than t (false negatives) or τ (cid:54)∈ G ∗ and the probability of τ given by the learned distribution Q is greater or equal to t (false positives).
Then the following holds for the cardinality of the set of incorrectly predicted triples F :
[Equation]
Note that the bound from this theorem is minimized for the threshold t :=
(1 − δ ) / 2 .
This suggests how to set the threshold if we knew the missingness parameter δ .
P UTTING E VERYTHING S O F AR T OGETHER
Next we combine the results from the previous sections to obtain a bound on the expected number of incorrectly predicted triples for vector representations X learned by maximizing the likelihood of the respective KG-distribution parametrized by X .
Here we assume that the domain X of the vector representations is a subset of R d of ﬁnite diameter and that there is some pre-ﬁxed scoring function ψ .
Theorem 4.
Let G ∗ be a ground-truth knowledge graph on a set of objects O and a set of relations R and let G be sampled by the data-generating distribution.
Let X ⊆ R d .
Let H X
(“hypothesis class”) denote the set of vector representations X , X
⊆ X |O| + |R| , of the objects from O and relations from R .
Let K , M , D be as in Theorem 2 and t and δ be as in Theorem 3.
Let X ∈ H X be a representation of objects and relations learned by maximizing the log-likelihood L ( X |G ) , i.e. X = arg max X (cid:48) ∈H X L ( X (cid:48) |G ) .
Finally, let X ∗ = arg max X ∈H X E
[ L ( X | . )] .
Then the following holds for the sets of incorrectly predicted triples F X , obtained using the KG-distribution with X : E (cid:20) |F X | |O| 2 |R| (cid:21) ≤ max (cid:26) ln
[Equation]
t 2 , ln
[Equation]
[Equation]
t
[Equation]
(cid:27) · (cid:18)
[Equation]
|O|
[Equation]
M (cid:115)
[Equation]
· (cid:115)
[Equation]
|O| + KL
[Equation]
) |O| [Equation] |R|   where KL ( P || P X ∗ ) is the Kullback-Leibler divergence of the data-generating distribution P and the best possible representable distribution P X ∗ .
T HE “R EALIZABLE ” C ASE
In learning theory, one usually speaks of the “realizable case” when the hypothesis class contains the target concept.
In general, the data generating distribution cannot be represented exactly using the KG-distribution with vector representations from a given set X unless we allow | ψ | to be inﬁnite (which would be needed to model the triples that have zero probability of being sampled from the data generating distribution because they are not contained in the ground-truth knowledge graph
G ∗ ).
Therefore we cannot really speak of the “realizable case” in the usual sense.
In particular it may be the case that the given scoring function ψ and the given domain of vector representations
X are enough to represent the knowledge graph in the sense that there exists a threshold t such that ( ψ ( x h , x r , x t ) ≥ t ) ⇔ ( h, r, t ) ∈ G ∗ ) , and yet the data-generating distribution could not be represented by the corresponding KG-distribution exactly.
Hence, it makes sense to deﬁne the realizable case slightly differently for our setting.
Deﬁnition 4 ( γ -Realizable case) .
For a given data-generating distribution P induced by a groundtruth knowledge graph G ∗ with the missingness parameter δ and a given scoring function ψ and a domain of vector representations X , we say that a KG-distribution P X given by a scoring function ψ and vector representations
X is γ -admissible if the following is satisﬁed: exp( ψ ( x h , x r , x t )) /
(1 + exp( ψ ( x h , x r , x t ))) ≥ 1 − γ for all ( h, r, t ) ∈ G ∗ and 1 / (1 + exp( ψ ( x h , x r , x t )))
≤ γ for all ( h, r, t ) (cid:54)∈ G ∗ , where x h , x r , x t are vector representations given by X .
We say that a learning problem is γ -realizable if all vector representations X ∈ X |O| + |R| that maximize the expected log-likelihood (i.e. the best vector representations) give rise to γ -admissible KG-distributions.
Next we present a version of Theorem 3 suitable for γ -realizable settings.
Theorem 5.
Let G ∗ be the ground-truth graph on a set of objects O and relations R .
Let P denote the data-generating distribution induced by G ∗ and let P γ be a γ -admissible distribution.
Let Q denote the learned distribution and let t ∈ ( γ ; 1 − γ ) be a threshold.
Then the following holds for the cardinality of the set of incorrectly predicted triples F :
[Equation]
In the next theorem we present a tighter version of Theorem 4 for γ -realizable settings.
Theorem 6.
Let everything be as in Theorem 4 but assume a γ -realizable setting.
Then the following holds for the sets of incorrectly predicted triples F X , obtained using the KG-distribution with X :
[Equation]
The next example illustrates that the results from Theorem 6 are asymptotically non-trivial (which is non-obvious as we show below).
Example 2.
Let us assume that we have some ﬁxed H X , K , D and d and that the X ∗ ∈ H X .
For simplicity, let us assume that this is a γ -realizable setting with γ = 0 .
Let the ground knowledge graphs G be sparse and satisfy |G| =
α ( |O| ) · |O| where α is an integer-valued function and O is the set of objects in the knowledge graph.
We suppose that the threshold t satisﬁes t ∈ ( γ, 1 − γ ) .
Then the expected fraction of triples incorrectly predicted by the learned model from our hypothesis class can be bounded by: C 1 · (cid:112) α ( |O| )
(cid:112) ln |O| / |O| where C 1 is a positive constant.
Now consider another “alternative prediction method” which always predicts all triples as false.
The fraction of triples incorrectly predicted by this method will be: E (cid:2) |F| / ( |O| 2 |R| ) (cid:3) ≤ C 2 · α ( |O| ) / |O| where C 2 is a positive constant.
It follows that whenever α ( |O| ) grows faster than ln |O| , the bounds of the maximum likelihood method, derived in this paper, become tighter than the error bound of this trivial method.
Here, we note that α ( |O| ) growing only as a logarithm corresponds to very sparse knowledge graphs (in which prediction of positive examples is likely hard).
E XISTING KGE M ETHODS IN THE L IGHT OF O UR A NALYSIS
In this section, we discuss how existing knowledge graph embedding methods in general ﬁt into our analysis.
10. S CORING F UNCTIONS
We assumed that predictions are made by comparing exp ( ψ ( x h , x r , x t )) with a threshold.
In existing works, one directly compares the output of the scoring function ψ ( x h , x r , x t ) with a threshold.
This differs only by taking a logarithm and, hence, obviously does not harm applicability of our analysis. 10. L OSS F UNCTIONS
Throughout this paper we assumed learning vector representations of objects and relations using maximum likelihood. Here we show that this setting corresponds to learning knowledge graph embeddings using a sigmoidal transformation of the scoring function ψ together with log-loss.
This combination is often used in the knowledge graph embedding literature (Trouillon et al., 2016; Dettmers et al., 2018; Kazemi & Poole, 2018). In particular the loss used in the literature is (using our notation): [Equation]
[Equation] It is not difﬁcult to see that this is the same as the negative log-likelihood of the KG-distribution with scoring function ψ . Thus, our analysis directly applies to methods that use this loss, such as
ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018) and SimplE (Kazemi & Poole, 2018). For other methods, log-loss can in principle be also used and it was observed by Trouillon & Nickel (2017) that the margin-based loss functions, used by many knowledge graph embedding methods, are more prone to overﬁtting compared to log-likelihood.
Our results may shed further light on this latter observation, however, to verify it theoretically would require to perform a similar type of analysis as we did in this paper for the margin-based loss functions (which might require completely different techniques). Therefore we leave this comparison for future work. 10. L IPSCHITZ C ONTINUITY AND B OUNDED D OMAIN A SSUMPTIONS
Our result requires Lipschitz continuity. Most of the scoring functions ψ used by existing knowledge graph methods are not globally Lipschitz continuous. However, they are Lipschitz continuous if we Number of objects in the domain |O|
B ound fr o m T h e o r e m restrict the domain of the embeddings to a bounded subset of R d (this is among others true for the approaches based on matrix or tensor factorization).
For obtaining the generalization bounds, we needed to assume that the learned vector representations come from a set with a bounded diameter.
Existing methods usually do not have such a constraint explicitly but use a regularization term which has a similar effect in practice.
TransE Here we show how our results apply to TransE (Bordes et al., 2013), arguably the simplest embedding method.
In TransE, ψ ( x h , x r , x t ) = −(cid:107) x h + x r − x t (cid:107) .
It is not difﬁcult to show that the Lipschitz constant K = 1 as follows (using the triangle inequality repeatedly):
[Equation]
≤
[Equation]
(cid:48) [Equation] (cid:107) .
Let the dimension of the vector embeddings be d = 3 .
Let us suppose that X = { x |(cid:107) x (cid:107) ≤ ln 10 } .
Then the remaining parameters from Theorem 4 are D
= ln 10 2 and M =
ln 10 3 .
For simplicity, let us assume a γ -realizable setting with γ = 0 .
Let t = 0 .
Finally suppose that |G ∗ | = |O| ln 2 |O| and |R| = 1 .
Now we have all parameters needed to apply Theorem 6.
We plot the resulting bound for a varying number of objects in Figure 1 in red.
Note that, even though the number of dimensions of the vector embeddings is low, the total number of parameters of the model is very high because each object and each relation has its own parameters.
SimplE Next we illustrate how our analysis applies to SimpleE (Kazemi & Poole, 2018), which is a fully-expressive knowledge graph embedding method. In SimplE, each object o ∈ O and each relation r ∈ R are represented by two vectors x (cid:48)
o and x (cid:48)(cid:48) o , respectively, x (cid:48) r and x (cid:48)(cid:48) r . In our formalism, we represent both as single vectors: x o =
[ x (cid:48) o , x (cid:48)(cid:48) o ] and x r = [ x (cid:48) r , x (cid:48)(cid:48) r ] .
The scoring function of SimpleE is ψ ( x h , x r , x t ) = 12 ( (cid:104) x (cid:48) h , x (cid:48)
r , x (cid:48)(cid:48) t (cid:105) + (cid:104) x (cid:48) t , x (cid:48)(cid:48) r , x (cid:48)(cid:48) h (cid:105) ) where (cid:104) x , y , z (cid:105) : = (cid:80) di
=1 x i y
i
z i .
Clearly the Lipschitz constant of ψ ( x o , x r , x t ) is not bounded for unbounded x h , x r and x t .
We therefore need to restrict the domain of the vector representations X , which our theorems assume anyways and which is done implicitly through the L2-regularization term in the original work (Kazemi &
Poole, 2018).
We show in the appendix that the Lipschitz constant K for SimplE can be bounded by √ 62 sup x ∈X (cid:107) x (cid:107) 2 .
For the illustration, we will assume the same dimension d = 3 and the same domain X as we did for TransE, i.e. X = { x |(cid:107) x (cid:107) ≤ ln 10 } .
Then the other parameters needed to compute the generalization bound for SimplE are D
= ln 10 2 , M = (ln 10) 3 , K = √ 62 (ln 10) 2 .
Finally, we assume the same γ , |R| and |G ∗ | as we did for TransE.
The resulting generalization bound for SimplE following from Theorem 5 is shown in Figure 1 in red together with the bound for TransE (in blue).
Not surprisingly, the generalization error bound is looser for SimplE as it is a more ﬂexible model (in fact, a fully-expressive model with a sufﬁciently large dimension of the vector representations).
Our work is very close in spirit to the theoretical analyses of learning from positive and unlabeled examples, known as PU-learning (Niu et al., 2016).
However, there are several distinctions that do not allow direct application of the existing results to the problem that we analyze in this paper.
First, existing works on PU-learning assume the inductive setting whereas our setting is transductive.
Second, the work (Niu et al., 2016), which is one of the few papers analyzing generalization bounds for PU-learning, only applies to losses that satisfy l ( t, +1)
+ l ( t, − 1) = 1 which is not the case for log-loss.
Moreover the losses considered by Niu et al.
(2016) are usually not used for knowledge graph embedding in practice.
Third, even if the previous two differences were not a problem (although they are), it would still not be obvious how to apply the existing results to the knowledge graph embedding setting in which we deal with interconnected examples; i.e. it is not clear what the attributes of the examples and the hypothesis class should be so that we could ﬁt knowledge graph embedding setting into the standard PU learning setting.
Lastly, an advantage of our analysis is that it uses a simple probabilistic model which provides a possible, hopefully intuitive, explanation of why knowledge graph embedding methods work (at least in the standard testing scenarios that use test data generated completely at random).
Another line of works that are related to ours are works on generalization bounds for link prediction, such as (London et al., 2013;
Wang et al., 2018).
The main differences w.r.t.
these works is that (i) they assume availability of both positive and negative examples (whereas we only have positive examples), (ii) the links are predicted based on attributes of nodes and not on the structure of the relations.
Finally, our work is also related to works for tensor completion using tensor decomposition, such as (Nickel & Tresp, 2013).
However, ﬁrst, not all knowledge graph embedding methods can be seen as some low-rank tensor decompositions and, second, as far as we know no analysis of generalization bounds of tensor decomposition applies to the PU-learning setting, in which one observes only positive examples (which is the setting studied in our work).
C ONCLUSIONS
In this paper we have derived generalization bounds for the expected number of triples predicted incorrectly by knowledge graph embedding methods.
The main assumptions that we used were: (i) facts are missing from the knowledge graph “completely at random” and (ii) the vector representations of objects and relations are learned by maximizing log-likelihood of the model (equivalently by minimizing log-loss).
While the “missing completely at random” assumption may seem simplistic, it is actually how training and test sets are usually constructed, e.g. in (Bordes et al., 2013), and only recently there have been works systematically looking beyond datasets constructed in this way, e.g. (Trouillon et al., 2019).
As for the second assumption, it would be interesting to see if one could derive similar bounds for other loss functions that have been used for knowledge graph embedding in the literature, which would then provide a more solid theoretical justiﬁcation for their use
A U SEFUL I NEQUALITIES
We will need the following well-known inequalities.
Lemma 7 (Pinsker’s inequality (Pinsker, 1964)) .
Let P and Q be distributions.
Then
[Equation]
Lemma 8 (McDiarmid’s Inequality (McDiarmid, 1989)) .
Let X 1 , . . .
, X n be independent random variables with values from a set X and let f : X n → R .
If, for all i ∈ { 1 , . . . , n } , and for all x 1 , , x n , x (cid:48) i ∈ X , the function f satisﬁes | f ( x 1 , . . .
, x i , . . .
, x n ) − f ( x 1 , . . .
, x (cid:48)
i , . .
, x n ) | ≤ c i then:
[Equation]
B P ROOF OF T HEOREM
In this section, we prove Theorem 1.
Theorem 1.
Let G ∗ be the ground-truth knowledge graph.
Let G be sampled from the datagenerating distribution induced by G ∗ and X be ﬁxed vector representations.
Then
[Equation]
where M ≥ sup h , t ∈ X o , r ∈ X r | ψ ( x h , x r , x t ) | .
Proof.
Let F :=
|G ∗ | .
Let U 1 , . .
, U F be Bernoulli random variables such that P [ U i = 1] = 1 − δ .
Let us denote by τ i the i -th tuple in G ∗ .
Let g : { 0 , 1 } F → 2 G ∗ be deﬁned as g ( u 1 , . .
, u F ) = { τ i ∈ G ∗ |
u i = 1 } , i.e. g ( u 1 , . . .
, u F ) contains all τ
i ’s for which u i = 1 .
It is not difﬁcult to see that G and g ( U 1 , . . .
, U F ) have the same distribution.
Hence we can bound the probability
P [ | L ( X | g ( U 1 , . . .
, U F ))
− E [ L ( X | . )]
| ≥ ε ] instead of directly P [ | L ( X |G ) − E
[ L ( X | . )]
| ≥ ε ] .
It follows from the assumptions of the theorem that for any ( u 1 , . . .
, u F ) ∈ { 0 , 1 } F the following must hold:
[Equation]
where x h , x r and x t denote vector representations of h , r and t , respectively.
Hence, we can use McDiarmid’s inequality to obtain P [ | L ( X | g
( U 1 , . . .
, U F ))
− E [ L ( X | . )]
| ≥ ε ] ≤ 2 exp (cid:16) − 2 ε 2 M 2 F (cid:17) from which the statement of the theorem follows directly.
C P ROOF OF T HEOREM
In this section, we give a proof of Theorem 2.
First, we prove several lemmas that we need to prove this theorem.
Lemma 9.
Let (cid:102) M > 0 be such that for every object or relation o it holds (cid:107) x o − x (cid:48) o (cid:107) ≤ (cid:102) M where x o and x (cid:48)
o are the vector representations of o in X and X (cid:48) , respectively.
If ψ satisﬁes the assumptions of
Theorem 2 then the following holds: | L ( X o , X r |G ) − L ( X (cid:48)
o , X (cid:48) r |G ) | ≤ 2 K |O| 2 |R| (cid:102) M.
Proof.
Let us ﬁrst introduce some notation.
Let V := |O| and R = |R| .
Whenever we write x h , x r and x t , it denotes the vector representations of h , r and t from X , whereas when we write x (cid:48) h , x (cid:48) r and x (cid:48) t , it denotes the representations from X (cid:48) .
We have
[Equation]
Next we bound the terms on the r.h.s.
of the above inequality separately.
We start by obtaining the bound for the ﬁrst term:
[Equation]
It remains to bound the second term.
We have:
[Equation]
where the last inequality follows from the inequality K ( (cid:107)
x h − x
[Equation]
h [Equation] + (cid:107) x r − x
[Equation]
r (cid:107) + | x t − x (cid:48) t (cid:107) ) ≤
K (cid:102) M .
Combining (1) and (2) we obtain the inequality that we needed to prove.
Lemma 10.
Let G ∗ be a ground-truth knowledge graph on a set of objects O and a set of relations R and let G be sampled by the data-generating distribution.
Let H (“hypothesis class”) be a ﬁnite set of vector representations of the objects from O and relations from R .
Then the following inequality holds
[Equation]
G
[Equation] | [Equation] where M ≥ sup h , t ∈ X o , r ∈ X r | ψ ( x h , x r , x t ) | .
Proof.
We denote W := sup X ∈H | L ( X |G ) − E
[ L ( X | . )
| and B := 2 / ( |G ∗ | · M 2 ) .
Then, using the union bound and Theorem 1, we have: P [ W ≥ ε ] ≤ 2 |H| · exp (cid:0) − ε 2 · B (cid:1) .
Next, we have
[Equation]
The above expression is minimized for u := ln (2 |H| ) /B
(note that u may be arbitrary), yielding the bound:
[Equation]
Lemma 11 (Covering number, e.g., (Shalev-Shwartz & Ben-David, 2014)) .
Let S be a subset of R d of diameter at most k (i.e. for any x 1 , x 2 ∈ S , (cid:107) x 1 − x 2 (cid:107) ≤ k ).
Then S can be covered by (cid:16) (2 k √ d ) /ρ
(cid:17) d balls of radius ρ .
Now we are ready to prove Theorem 2.
Theorem 2.
Let G ∗ be a ground-truth knowledge graph on a set of objects O and a set of relations
R and let G be sampled by the data-generating distribution .
Let X ⊆ R d .
Finally, let H X (“hypothesis class”) denote the set of vector representations X , X ⊆ X |O| + |R| , of the objects from O and relations from R .
Let D , M and K be real numbers satisfying:
∀ x , x (cid:48) ∈
X : (cid:107) x − x (cid:48) (cid:107) ≤ D , 2.
∀ x h , x r , x t ∈ X : | ψ ( x h , x r , x t ) | ≤ M , 3.
∀ x h , x r , x t , x (cid:48)
h , x (cid:48)
r , x (cid:48) t ∈ X : (cid:107) ψ
( x h , x r , x t ) − ψ ( x (cid:48) h , x (cid:48)
r , x (cid:48) t ) (cid:107) ≤ K ( (cid:107) x
h − x (cid:48) h (cid:107)
[Equation]
(cid:107) x r [Equation]
(cid:48) t (cid:107) ) ,
Then the following holds:
[Equation]
Proof.
Let B be a ﬁnite set of vectors of dimension d such that if we place centers of balls of radius ρ in all these points then these balls will cover the set X .
Using Lemma 11, we know that it is possible to ﬁnd such a set B satisfying
[Equation]
because the diameter of the set X is at most D (due to the assumptions of the theorem) and the dimension of the vectors in X is d .
Next we assume that we are only searching for a maximumlikelihood solution X consisting of vectors from B so that we could use Lemma 10 which expects a ﬁnite hypothesis set H .
Even though we have so far been thinking of the vector representations X as lookup tables, it will be more convenient here to think of them as lists of vectors.
Speciﬁcally, we assume that the objects and relations are ordered arbitrarily.
The hypothesis set will then be represented as H B = B |O| + |R| .
Hence, using (3), we obtain |H B | ≤ (cid:16) (2 D √ d ) /ρ
(cid:17) d ( |O| + |R| ) .
Then from Lemma 10 we have:
[Equation]
Using the bound on |H B | , we can bound the r.h.s.
by:
[Equation]
For ﬁxed vector representations X (cid:48) of the objects and relations, let H ( X (cid:48) ) denote the set of vector representations X which satisfy that for every object or relation o : (cid:107) x o − x (cid:48)
o (cid:107) ≤ ρ where x o and x (cid:48)
o are vector representations of o from X and X (cid:48) , respectively.
Now, optimizing over the hypothesis set
H X of all possible vector representations from X instead of just the vectors from the set H B , would yield the next bound:
[Equation]
Next, to bound the ﬁrst and third term [Equation] we use Lemma 9 and, to bound the second term [Equation]
we use
Equations (4) and (5).
After setting ρ := 1 K |O| and after simplifying, this gives us the following bound:
[Equation]
For the expected error of normalized log-likelihood we then obtain
[Equation]
which ﬁnishes the proof of the theorem.
D P ROOF OF T HEOREM
In this section [Equation] we prove Theorem 3.
Theorem 3.
Let G ∗ be the ground-truth graph on a set of objects O and relations R .
Let P denote the data-generating distribution induced by G ∗ with the missingness parameter δ and let Q denote the learned distribution.
Let t ∈ (0; 1 − δ ) be a threshold for predicting which triples belong to the knowledge graph. Finally, let F be the set of incorrectly predicted triples, i.e. the set of all triples τ ∈ O×R×O such that either τ ∈ G ∗ and the probability of τ given by the learned distribution Q is smaller than t (false negatives) or τ (cid:54)∈ G ∗ and the probability of τ given by the learned distribution Q is greater or equal to t (false positives).
Then the following holds for the cardinality of the set of incorrectly predicted triples F :
[Equation]
In the proof of the theorem, we will use the following simple lemma (which can be veriﬁed by computing the discriminant of the respective quadratic equation).
Lemma 12.
The following inequality holds for all x, y ∈ [0; 1] 2 and α ∈ (0; ∞ ) :
[Equation]
Proof of Theorem 3.
We assume that all possible triples ( h, r, t ) ∈
O × R × O are ordered (arbitrarily) and that τ i denotes the i -th triple from this set
and I = { 1 , 2 , . .
, |O| 2 · |R|} .
Let G be sampled from the data-generating distribution and G (cid:48) be sampled from the learned model (i.e. the
KG-distribution).
We deﬁne binary random variables
B i := 1 ( τ i ∈ G ) and B (cid:48)
i := 1 ( τ i ∈ G (cid:48) ) .
Hence the complete vectors B = ( B 1 , . . . , B n ) and B (cid:48) =
( B (cid:48) 1 , . . .
, B (cid:48) n ) represent the knowledge graphs sampled from the data-generating and modelled distributions, respectively.
We denote by
P B and P (cid:48)
B the distributions of B and B (cid:48) .
Due to the independence properties of the two distributions, the following holds: KL ( P B || P B (cid:48) )
= (cid:80) i ∈I KL ( P B
i
|| P B (cid:48) i ) where P B i and P (cid:48)
B i are the distributions of B i and B (cid:48)
i , respectively.
Next we bound the error that is made when we classify all triples τ i with P [ B (cid:48) i = 1] > t as true (i.e. belonging to the knowledge graph) and the rest as false.
We may notice that when misclassifying a triple τ i , we must either have
P [ B i = 1] = 0 and P [ B (cid:48) i = 1] > t (“false positive”) or
P [ B i = 1] = 1 − δ and P [ B (cid:48) i = 1] < t (“false negative”).
Hence, if we denote by FP the set of triples τ i for which we have false positive errors, we must have for all α >
[Equation]
where the last inequality follows from Lemma 12.
It follows that
[Equation]
Letting α = 1 / (2 t ) yields the bound
[Equation]
Similarly for false negatives
[Equation]
if we denote by FN the set of triples for which we have false negative errors, we must have for all α (cid:48) > 0 :
[Equation]
Hence, we also have:
[Equation]
Letting α (cid:48)
= 1 / (2 − 2 δ − 2 t ) gives us
[Equation]
Next we combine (6) and (7) and apply Pinsker’s inequality:
[Equation]
This ﬁnishes the proof of this theorem.
E P ROOF OF T HEOREM
Theorem 4 Let G ∗ be a ground-truth knowledge graph on a set of objects O and a set of relations R and let G be sampled by the data-generating distribution.
Let X ⊆ R d .
Let H X
(“hypothesis class”) denote the set of vector representations X , X
⊆ X |O| + |R| , of the objects from O and relations from R .
Let K , M , D be as in Theorem 2 and t and δ be as in Theorem 3.
Let X ∈ H X be a representation of objects and relations learned by maximizing the log-likelihood L ( X |G ) , i.e. X = arg max X (cid:48) ∈H X L ( X (cid:48) |G ) .
Finally, let X ∗ = arg max X ∈H X E
[ L ( X | . )] .
Then the following holds for the sets of incorrectly predicted triples F X , obtained using the KG-distribution with X : E
[Equation]
where KL ( P || P X ∗ ) is the Kullback-Leibler divergence of the data-generating distribution P and the best possible representable distribution P X ∗ .
Proof.
Let G denote the knowledge graph sampled from the data-generating distribution and P X the learned distribution.
For a ﬁxed X ﬁxed , the following equality holds: KL ( P || P X ﬁxed ) =
H ( P ) + E
P [ L ( X ﬁxed |G )] .
Then we also have KL ( P || P X ﬁxed ) − KL
( P || P X ∗ ) =
E P
[ L ( X ﬁxed |G )]
− E P
[ L ( X ∗ |G )] .
Next we can deﬁne g ( G (cid:48) ) :
= arg max X ∈H X L ( X |G (cid:48) ) , substitute g ( G (cid:48) ) for X ﬁxed and take the expectation over graphs G (cid:48) , which are assumed to be sampled from the data-generating distribution.
This yields:
[Equation]
[ E G∼ P [ L
[Equation]
g ( G [Equation] ) |G )] − E G∼ P [ L ( X ∗ |G )]] . Next we can use Lemma 8.2 from (Devroye et al., 2013) to bound the r.h.s.:
[Equation]
Finally, we use the bound from Theorem 2 to bound the r.h.s.
of this inequality.
Combining this with
Theorem 3 ﬁnishes the proof.
F P ROOF OF T HEOREM
Theorem 5.
Let G ∗ be the ground-truth graph on a set of objects O and relations R .
Let P denote the data-generating distribution induced by G ∗ and let P γ be a γ -admissible distribution.
Let Q denote the learned distribution and let t ∈ ( γ ; 1 − γ ) be a threshold.
Then the following holds for the cardinality of the set of incorrectly predicted triples F :
[Equation]
Proof.
The proof of this theorem is very similar to the proof of Theorem 3.
We highlight the most important differences in bold.
We assume that all possible triples ( h, r, t ) ∈
O × R × O are ordered (arbitrarily) and that τ i denotes the i -th triple from this set
and I = { 1 , 2 , . .
, |O| 2 · |R|} .
Let G be sampled from the distribution P γ and G (cid:48) be sampled from the learned model (i.e. the KG-distribution).
We deﬁne binary random variables
B i := 1 ( τ i ∈ G ) and B (cid:48)
i := 1 ( τ i ∈ G (cid:48) ) .
Hence the complete vectors
B = ( B 1 , . . . , B n ) and B (cid:48) =
( B (cid:48) 1 , . . .
, B (cid:48) n ) represent the knowledge graphs sampled from the distribution P γ and the modelled distribution, respectively.
We denote by P B and P (cid:48)
B the distributions of B and B (cid:48) .
Due to the independence properties of the two distributions ( after all, P γ is still a KG-distribution and therefore inherits its independence structure ), the following holds:
KL
[Equation]
P
[Equation]
i || P B [Equation]
i ) where P B i and P (cid:48)
B i are the distributions of B i and B (cid:48)
i , respectively.
Next we bound the error that is made when we classify all triples τ i with P [ B (cid:48) i = 1] > t as true (i.e. belonging to the knowledge graph) and the rest as false.
We may notice that when misclassifying a triple τ i , we must either have
P [ B i = 1] ≤ γ and P [ B (cid:48) i = 1] > t (“false positive”) or
P [ B i = 1] ≥ 1 − γ and P [ B (cid:48) i = 1] < t (“false negative”).
The rest of differences w.r.t.
the proof of Theorem 3 follows from the differences highlighted so far in a straightforward way (details follow below).
Hence, if we denote by FP the set of triples τ i for which we have false positive errors, we must have for all α >
[Equation]
where the last inequality follows from Lemma 12.
It follows that
[Equation]
Letting 1 / (2 t − 2 γ ) yields the bound
[Equation]
Similarly for false negatives, if we denote by FN the set of triples for which we have false negative errors, we must have for all α (cid:48) > 0 :
[Equation]
γ − t ) ≤ (cid:88)
[Equation]
Hence
[Equation]
we also have:
[Equation]
Letting α (cid:48)
= 1 / (2 − 2 γ − 2 t ) gives us
[Equation]
Next [Equation] we combine (9) and (10) and apply Pinsker’s inequality:
[Equation]
This ﬁnishes the proof of this theorem.
G P ROOF OF T HEOREM
Theorem 6 Let everything be as in Theorem 4 but assume a γ -realizable setting.
Then the following holds for the sets of incorrectly predicted triples F X , obtained using the KG-distribution with X : E
[Equation] 
Proof.
The proof of this theorem is practically identical to the proof of Theorem 4.
The only difference is that in this proof we invoke Theorem 5 instead of Theorem 3.
H T HE L IPSCHITZ C ONSTANT OF S IMPL E’ S S CORING F UNCTION
{+In this section we derive a bound on the Lipschitz constant K of the scoring function of SimplE (Kazemi & Poole, 2018) on a bounded domain X ⊆ R 2 d . 
+}
The scoring function of SimplE is ψ ( x o , x r , x t )
=
12 ( (cid:104) x
(cid:48) h , x (cid:48)
r , x (cid:48)(cid:48) t (cid:105) + (cid:104) x
(cid:48) t , x (cid:48)(cid:48) r , x (cid:48)(cid:48) h (cid:105) )
where (cid:104) x , y , z (cid:105) :
= (cid:80) di
=1 x i y
i
z i .
Let us denote x h
= [ x (cid:48) h , x (cid:48)(cid:48) h ] , x r =
[ x (cid:48) r , x (cid:48)(cid:48) r ] , x t =
[ x (cid:48) t , x (cid:48)(cid:48) t ] and r =
[ x h , x r , x t ] .
Since the function ψ ( r ) is differentiable everywhere, any K satisfying K ≥ (cid:107)∇ ψ ( r ) (cid:107) for all r ∈ X 3 will be a Lipschitz constant for ψ on the bounded domain X .
Next we compute the gradient of ψ ( r ) .
To simplify the notation, we use x [ i ] to denote the i -th element of the vector x .
We then have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
From this we get
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
Thus, we get
[Equation]
Let us now denote y h
=
[ y (cid:48) h , y (cid:48)(cid:48)
h ] , y r
=
[ y
[Equation]
r , y (cid:48)(cid:48) r ] , y t =
[ y (cid:48) t , y (cid:48)(cid:48) t ] , and s =
[ y h , y r , y t ] .
Then we have (cid:107) ψ ( r ) − ψ ( s ) (cid:107) ≤ K (cid:107) r − s (cid:107) ≤ K · ( (cid:107) x
h − y h (cid:107) + (cid:107) x r − y r (cid:107) + (cid:107) x
t − y t (cid:107) ) .
So K =
[Table]
62 sup x ∈X (cid:107) x (cid:107) 2 is indeed the Lipschitz constant that we need in order to be able to obtain generalization bounds using Theorems 4 and 5.