P ROJECTED C ANONICAL D ECOMPOSITION FOR K NOWLEDGE B ASE C OMPLETION
Anonymous authors
Paper under double-blind review
A BSTRACT
The leading approaches to tensor completion and link prediction are based on the canonical polyadic (CP) decomposition of tensors.
While these approaches were originally motivated by low rank approximations, the best performances are usually obtained for ranks as high as permitted by computation constraints.
For large scale factorization problems where the factor dimensions have to be kept small, the performances of these approaches tend to drop drastically.
The other main tensor factorization model, Tucker decomposition, is more ﬂexible than CP for ﬁxed factor dimensions, so we expect Tucker-based approaches to yield better performance under strong constraints on the number of parameters.
However, as we show in this paper through experiments on standard benchmarks of link prediction in knowledge bases, ComplEx, (Trouillon et al., 2016), a variant of CP, achieves similar performances to recent approaches based on Tucker decomposition on all operating points in terms of number of parameters.
In a control experiment, we show that one problem in the practical application of Tucker decomposition to large-scale tensor completion comes from the adaptive optimization algorithms based on diagonal rescaling, such as Adagrad.
We present a new algorithm for a constrained version of Tucker which implicitly applies Adagrad to a CP-based model with an additional projection of the embeddings onto a ﬁxed lower dimensional subspace.
The resulting Tucker-style extension of ComplEx obtains similar best performances as ComplEx, with substantial gains on some datasets under constraints on the number of parameters.
I NTRODUCTION
The problems of representation learning and link prediction in multi-relational data can be formulated as a binary tensor completion problem, where the tensor is obtained by stacking the adjacency matrices of every relations between entities.
This tensor can then be intrepreted as a "knowledge base", and contains triples (subject, predicate, object) representing facts about the world.
Link prediction in knowledge bases aims at automatically discovering missing facts (Bordes et al., 2011;
Nickel et al., 2011; Bordes et al., 2013; Nickel et al., 2016a; Nguyen, 2017).
State of the art methods use the canonical polyadic (CP) decomposition of tensors (Hitchcock, 1927) or variants of it (Trouillon et al., 2016; Kazemi & Poole, 2018;
Lacroix et al., 2018).
While initially motivated by low-rank assumptions on the underlying ground-truth tensor, the best performances are obtained by setting the rank as high as permitted by computational constraints, using tensor norms for regularization (Lacroix et al., 2018).
However, for large scale data where computational or memory constraints require ranks to be low (Lerer et al., 2019), performances drop drastically.
Tucker decomposition is another multilinear model which allows richer interactions between entities and predicate vectors.
A special case of Tucker decomposition is RESCAL (Nickel et al., 2011), in which the relations are represented by matrices and entities factors are shared for subjects and objects.
However, an evaluation of this model in Nickel et al.
(2016b) shows that RESCAL lags behind other methods on several benchmarks of interest.
Recent work have obtained more competitive results with similar models (Balaževi´c et al., 2019b; Wang et al., 2019), using different regularizers or deep learning heuristics such as dropout and label smoothing.
Despite these recent efforts, learning
Tucker decompositions remains mostly unresolved.
Wang et al.
(2019) does not achieve state of the art results on standard benchmarks, and we show (see Figure 3) that the performances reported by
Balaževi´c et al.
(2019b) are actually matched by ComplEx (Trouillon et al., 2016;
Lacroix et al., 2018) optimized with Adam, which has less hyperparameters.
In this work, we overcome some of the difﬁculties associated with learning a Tucker model for knowledge base completion.
Balaževi´c et al.
(2019b) use deep-learning mechanisms such as batch normalization (Ioffe & Szegedy, 2015), dropout (Srivastava et al., 2014) or learning-rate annealing to address both regularization and optimization issues.
Our approach is different: We factorize the core tensor of the Tucker decomposition with CP to obtain a formulation which is closer to CP and better understand what difﬁculties appear.
This yields a simple approach, which has a single regularization hyperparameter to tune for a ﬁxed model speciﬁcation.
The main novelty of our approach is a more careful application of adaptive gradient techniques.
State-of-the-art methods for tensor completion use optimization algorithms with adaptive diagonal rescaling such as Adagrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2014).
Through control experiments in which our model is equivalent to CP up to a ﬁxed rotation of the embeddings, we show that one of the difﬁculties in training Tucker-style decompositions can be attributed to the lack of invariance to rotation of the diagonal rescaling.
Focusing on Adagrad, we propose a different update rule that is equivalent to implicitely applying Adagrad to a CP model with a projection of the embedding to a lower dimensional subspace.
Combining the Tucker formulation and the implicit Adagrad update, we obtain performances that match state-of-the-art methods on the standard benchmarks and achieve signiﬁcanly better results for small embedding sizes on several datasets.
Compared to the best current algorithm for Tucker decomposition of Balaževi´c et al.
(2019b), our approach has less hyperparameters, and we effectively report better performances than the implementation of ComplEx of Lacroix et al.
(2018) in the regime of small embedding dimension.
We discuss the related work in the next section.
In Section 3, we present a variant of the Tucker decomposition which allows to interpolate between Tucker and CP.
The extreme case of this variant, which is equivalent to CP up to a ﬁxed rotation of the embedding, serves as control model to highlight the deﬁciency of the diagonal rescaling of Adagrad for Tucker-style decompositions in experiments reported in Section 4 .
We present the modiﬁed version of Adagrad in Section 5 and present experimental results on standard benchmarks of knowledge base completion in Section 7.
L INK PREDICTION IN KNOWLEDGE BASES
Notation Tensors and matrices are denoted by uppercase letters.
For a matrix U , u i is the vector corresponding to the i -th row of U .
The tensor product is written (cid:10) and the Hadamard product (i.e., elementwise product) is written ⊙ .
2. L EARNING SETUP
A knowledge base consists of a set S of triples (subject, predicate, object) that represent (true) known facts.
The goal of link prediction is to recover facts that are true but not in the database.
The data is represented as a tensor ~
X 2 f 0 ; 1 g N (cid:2) L (cid:2) N for N the number of entities and L the number of predicates.
Given a training set of triples, the goal is to provide a ranking of entities for queries of the type (subject, predicate, ?) and (?, predicate, object).
Following Lacroix et al.
(2018), we use the cross-entropy as a surrogate of the ranking loss.
As proposed by Lacroix et al.
and Kazemi & Poole (2018), we include reciprocal predicates: for each predicate P in the original dataset, and given an item o , each query of the form (?, P , o ) is reformulated as a query ( o , P (cid:0) 1 , ?), where o is now the subject of P (cid:0) 1 .
This doubles the effective number of predicates but reduces the problem to queries of the type (subject, predicate, ?) only.
For a given triple ( i; j; k ) 2 S , the training loss function for a tensor X is then
[Equation]
For a tensor decomposition model X ( (cid:18) ) parameterized by (cid:18) , the parameters ^ (cid:18) are found by minimizing the regularized empirical risk with regularizer (cid:3) :
[Equation]
This work studies speciﬁc models for X ( (cid:18) ) , inspired by CP and Tucker decomposition.
We discuss the related work on tensor decompositions and link prediction in knowledge bases below.
2. R ELATED WORK 2.2. C ANONICAL D ECOMPOSITION AND ITS VARIANTS
The canonical polyadic (CP) decomposition of a tensor X is deﬁned entrywise by
[Equation]
The smallest value of d for which this decomposition exists is the rank of X .
Each element X i;j;k is thus represented as a multi-linear product of the 3 embeddings in R d associated respectively to the i th subject, the j th predictate and the k th object.
CP currently achieves near state-of-the-art performances on standard benchmarks of knowledge base completion (Kazemi & Poole, 2018; Lacroix et al., 2018).
Nonetheless, the best reported results are with the ComplEx model (Trouillon et al., 2016), which learns complex-valued embeddings and sets the embeddings of the objects to be the complex conjugate of the embeddings of subjects, i.e., w k = (cid:22) u k .
Prior to ComplEx, Dismult was proposed (Yang et al., 2014) as a variant of CP with w k = u k .
While this model obtained good performances (Kadlec et al., 2017), it can only model symmetric relations and does not perform as well as ComplEx.
CP-based models are optimized with vanilla Adam or Adagrad and a single regularization parameter (Trouillon et al., 2016; Kadlec et al., 2017; Lacroix et al., 2018) and do not require additional heuristics for training.
2.2. T UCKER D ECOMPOSITION AND ITS VARIANTS
Given a tensor X of size N (cid:2) L (cid:2) N , the Tucker decomposition of X is deﬁned entrywise by 8 i; j; k;
X i
[Equation]
j
[Equation]
k
[Equation]
i
[Equation]
d 1 ∑ r 1 = d 2 ∑ r 2 = d 3 ∑ r 3 =
C r 1 ;r 2 ;r 3 u ir 1 v jr 2 w kr 3 :
The triple ( d 1 ; d 2 ; d 3 ) are the rank parameters of the decomposition.
We also use a multilinear product notation X =
[[ C ; U; V; W ]] , where U; V; W are the matrices whose rows are respectively u j ; v k ; w l and C the three dimensional d 1 (cid:2) d 2 (cid:2)
d 3 core tensor .
Note that the CP decomposition is a Tucker decomposition
in which d 1 = d 2 = d 3 =
d
and C is the identity, which we write [[ U; V; W ]] .
With a non-trivial core tensor, Tucker decomposition is thus more ﬂexible than CP for ﬁxed embedding size.
In knowledge base applications, we typically have d (cid:20) L ≪ N , so the vast majority of the model parameters are in the embedding matrices of the entities U and W .
When constraints on the number of model parameters arise (e.g., memory constraints), Tucker models appear as natural candidates to increase the expressivity of the decomposition compared to CP with limited impact on the total number of parameters.
While many variants of the Tucker decomposition have been proposed in the literature on tensor factorization (see e.g., Kolda & Bader, 2009), the ﬁrst approach based on Tucker for link prediction in knowledge bases is RESCAL (Nickel et al., 2011).
RESCAL uses a special form of Tucker decomposition in which the object and subject embeddings are shared, i.e., U = W , and it does not compress the relation matrices.
In the multilinear product notation above, a RESCAL model is thus written as X =
[[ C ; U; I; U ]] .
Despite some success on a few smaller datasets, RESCAL performances drop on larger datasets (Nickel et al., 2016b).
This decrease in performances has been attributed either to improper regularization (Nickel et al., 2011) or optimization issues (Xue et al., 2018).
Balaževi´c et al.
(2019b) revisits Tucker decomposition in the context of large-scale knowledge bases and resolves some of the optimization and regularization issues using learning rate annealing, batch-normalization and dropout.
It comes at the price of more hyperparameters to tune for each dataset (label smoothing, three different dropouts and a learning rate decay), and as we discuss in our experiments, the results they report are not better than ComplEx for the same number of parameters.
Two methods were previously proposed to interpolate between the expressivity of RESCAL and
CP.
Xue et al.
(2018) expands the HolE model (Nickel et al., 2016b) (and thus the ComplEx model (Hayashi & Shimbo, 2017)) based on cross-correlation of embeddings to close the gap in expressivity with the Tucker decomposition for a ﬁxed embedding size.
Jenatton et al.
(2012) express the relation matrices in RESCAL as low-rank combination of a family of matrices.
We describe the link between these approaches and ours in Appendix 9.4.
None of these approach however studied the effect of their formulation on optimization, and reported results inferior to ours.
2.2. O THER APPROACHES
(Graph) neural networks for link prediction Several methods have introduced models that go beyond the form of Tucker and canonical decompositions.
ConvE (Dettmers et al., 2018) uses a convolution on a 2D tiling of the subject and relation embeddings as input to a 2-layer neural net that produces a new embedding for the pair, then compares to the object embedding.
Graph neural networks (Scarselli et al., 2009; Niepert et al., 2016; Li et al., 2016; Bruna et al., 2014) have recently gained popularity and have been applied to link prediction in knowledge bases by Schlichtkrull et al.
This model uses a graph convolutional architecture to generate a variant of CP.
Poincaré embeddings Poincaré embeddings have been proposed as an alternative to usual tensor decomposition approaches to learn smaller embeddings when the relations are hierarchical (Nickel & Kiela, 2017).
The method has recently been extended to link prediction in relational data with very good performance trade-offs for small dimensional embeddings on the benchmark using WordNet (Balaževi´c et al., 2019a), which contains relationships such as hypernyms and hyponyms which are purely hierarchical.
However, such good results do not extend to other benchmarks.
I NTERPOLATING BETWEEN CP AND T UCKER
In order to better understand the underlying difﬁculties in learning (variants of) Tucker decompositions compared to CP, our analysis starts from a Tucker model in which the core tensor is itself decomposed with CP.
Given a N (cid:2) L (cid:2) N tensor, a ﬁxed d and assuming a ( d; d; d ) Tucker decomposition to simplify notation, a Tucker model where the core tensor is itself decomposed with a rank- D CP can be written as (details are given in Appendix 9.3):
[Equation]
where P 1 ; P 2 ; P 3 are all D (cid:2) d matrices.
Since most knowledge bases have much fewer predicates than entities ( L ≪ N ), the dimension of the predictate factors has little impact on the overall number of model parameters.
So in the remainder of the paper, we always consider P 2 =
I .
Learning matrices U; V; W; P 1 ; P 3 of this decomposition simultaneously leads to the following model, which we call CP-Tucker (CPT):
[Equation]
The CPT model is similar to a CP model except that the embedding matrices U and W have an additional low-rank constraint ( d instead of D ).
We say that the model interpolates between CP and
Tucker because for D =
d it is equivalent to CP (as long as P 1 and P 3 are full rank), whereas for
D =
d 2 we recover a full Tucker model because the matrices P 1 and P 3 can be chosen such that
[Equation]
P 1 u i ; v j ; P 3 w k
[Equation]
= u i Mat( v j ) v Tj , where Mat is the operator that maps a d 2 vector to a d (cid:2)
d matrix (see Appendix 9.5).
CPT is similar to CANDELINC (Carroll et al., 1980), except that in CANDELINC the factors U , V and W are ﬁxed and used to compress the data in order to efﬁciently learn the P i .
Closer to CPT,
Bro & Andersson (1998) ﬁrst learn a Tucker3 decomposition of X before applying CANDELINC using the learned factors.
These methods are only applicable to least-square estimation, and for tensors of smaller scale than knowledge bases.
[Figure]
M RR ( h i g h e
r
i s b e tt e r )
CP (Adagrad
[Figure]
CPT (Adagrad)
PCP (Adagrad)
PCP (Ada imp ) (a) Performances on FB15K-237 in the control exper-
iments for D =
d , i.e., when PCP is a reparameterization of CP.
We observe that PCP and CPT with vanilla Adagrad, which are variants of Tucker, underperform compared to CP.
As expected in this case D =
d , our modiﬁcation of the Adagrad update leads to the same performances for PCP as for CP.
[Figure]
Sorted Adagrad coeﬃcients
[Figure]
PCP Adagrad
CP (b) Adagrad coefﬁcients for subject/object embedding matrices in the control experiments ( D = d ) for CP and PCP, averaged by columns (i.e., embedding di- mension) and sorted by values.
Adagrad coefﬁcients decay exponentially for CP, but the values are similar across most dimensions in PCP: the ﬁxed unitary transform in PCP removes the beneﬁt of Adagrad.
Fixed projection matrices: The Projected Canonical Polyadic (PCP) Decomposition
In order to clarify the difﬁculty that arise when learning a CPT model compared to a CP model, we study a simpler model in which the matrices P 1 and P 3 are not learned but rather ﬁxed during training and taken as random matrices with orthonormal columns.
We call the resulting model the Projected
Canonical Polyadic (PCP) decomposition, since P 1 ; P 3 project the embeddings of dimension
d into a higher dimension D :
[Equation]
j 2 R D ; ﬁxed P 1 ; P 3 2 R D (cid:2) d
When D =
d the matrices P i are then ﬁxed unitary transformations.
The PCP (or CPT) model in this case D =
d is then equivalent to a CP model, up to a ﬁxed invertible transformation of the embeddings.
The capacity of the model grows beyond that of CP as D increases up to d 2 .
M OTIVATION : OPTIMIZATION ISSUES WITH CPT AND PCP
As discussed in the related works, previous results suggest that Tucker models are more difﬁcult to train than CP models.
The goal of this section is to isolate an issue faced with CPT/PCP models when trained with vanilla adaptive gradient methods such as Adagrad or Adam.
4. C ONTROL EXPERIMENT : U NITARY P 1 AND P 3 IN PCP
When D = d in PCP, the model becomes equivalent to CP.
Indeed, the matrices P 1 and P 3 are unitary ( P 1 P ⊤ 1 = P 3 P ⊤ 3 = I ) and so [[( UP 1 ) P ⊤ 1 ; V; ( WP 3 ) P ⊤ 3 ]] =
[[ U; V; W ]] .
There is no practical interest in considering this degenerate case of PCP, we only use it in the following toy experiment to exhibit one of the difﬁculties encountered when training PCP.
We perform a simple control experiment in which we take one of the standard benchmarks of link prediction in knowledge bases, called FB15K-237, and train a CP model for different values of the rank D and a PCP model with D =
d with vanilla Adagrad.
The full experimental protocol, including hyperparameter tuning, is similar to our main experiments and is described in Section 7.2.
Figure 1a plots the performances in terms of the standard metric mean reciprocal rank (higher is better) as a function of D of CP (blue curve) and PCP (red curve, called PCP (Adagrad)).
We observe that CP obtains signiﬁcantly better performances than CPT for larger embedding dimension D .
Since in this toy experiment CP and PCP can represent exactly the same tensors and have equivalent regularizations, the only difference between the algorithms that can explain the difference in performances is in how the optimization is carried out, namely the diagonal rescaling performed by Adagrad: Adagrad adapts the learning rate on a per-parameter basis, depending on previous and current gradients, and is therefore not invariant by the addition of the matrices P 1 and P 2 even if these are unitary (we provide the formal justiﬁcation in the next section).
This is shown experimentally in Figure 1b where we plot the average Adagrad coefﬁcients for each embedding dimensions (i.e., adagrad coefﬁcients of subject/object embedding matrices averaged by column).
The addition of the random P 1 and P 2 ﬂattens the Adagrad weights, which in turn removes all the beneﬁt of the adaptive rescaling of the algorithm.
For reference, we also tried to directly learn all parameters including P 1 and P 3 (i.e., learn a CPT model) with vanilla Adagrad.
The performances obtained are also lower than those of CP, as shown in Figure 1a (orange curve).
A ROTATION INVARIANT A DA G RAD : A DA imp
In this section, we study the optimization problem in more details, and more precisely the effect of the diagonal rescaling performed by Adagrad.
As a remainder, given a sequence of stochastic gradients g ( t ) of L and denoting G ( t ) =
ϵI + ∑ t(cid:28)
=1 g ( (cid:28) )
g ( (cid:28) )
⊤ , the (practical) AdaGrad update is: (cid:18) ( t +1) p
=
(cid:18) ( t ) p (cid:0) (cid:17)g ( t ) p = √
G ( t ) pp or equivalently (cid:18) ( t +1) =
(cid:18) ( t ) (cid:0) (cid:17)
Diag ( G ( t ) ) (cid:0) 1 =where Diag ( G ) is the diagonal matrix obtained by extracting the diagonal elements of G .
5. T WO EQUIVALENT PARAMETRIZATIONS OF PCP
The following decomposition is equivalent to PCP, but its embeddings are expressed in R D :
[Equation]
with u i ; v j ; w k 2 R D ; ﬁxed P 1 ; P 2 2 R D (cid:2) d :
Note that with u i =
P ⊤ 1 u i and w k = P ⊤ 3 w k
, we go from PCP full to PCP.
The practical differences between PCP and PCP full are that PCP full learn embeddings in the high-dimensional space, maintaining the low-rank structure of the overall entity embeddings through the orthogonal projections
P 1 P T 1 and P 3 P T 3 .
The practical interest of PCP full is not in terms of modeling but rather from the optimization perspective with Adagrad because it has a structure that is closer to that of CP.
Indeed, for d = D , P 1 and P 3 disappear in PCP full , so that optimizing a PCP full model with Adagrad is equivalent to optimizing a CP model with Adagrad.
This property suggests an alternative to the vanilla PCP + Adagrad algorithm, which we call implicit Adagrad :
Implicit Adagrad: Ada imp
The approach we propose is to effectively optimize PCP full with Adagrad.
However, when d < D , which is the interesting case for PCP, we notice that we do not need to maintain embeddings in R D .
Our approach, called Ada imp , computes the gradients and Adagrad coefﬁcients with respect to u i ; w k 2 R D , but the full dimension factor matrices U and W are never explicitly stored in memory.
Rather, we store u
i
=
P ⊤ 1 u i and w k = P ⊤ 3 w k 2 R d , which is all that is required for any model computation in PCP full since P 1 and P 3 are ﬁxed.
Overall, the effective model parameters are exactly the same as in PCP, and we call this approach PCP +Ada imp .
An Ada imp update is described in Algorithm 4.
While PCP +Adagrad and PCP +Ada imp work with the same number of model parameters, the fundamental difference is the computation of Adagrad coefﬁcients.
Since Ada imp effectively applies Adagrad to PCP full , we need to maintain the Adagrad coefﬁcients in R D even when d < D : the overall update is ﬁrst computed in R D and projected back to R d after the application of Adagrad rescaling.
In constrat, in vanilla PCP +Adagrad, the gradient is projected to R d before Adagrad rescaling.
5. I MPLICIT OPTIMIZATION OF PCP FULL : A DA imp
In this section, we discuss more formally the Ada imp updates, and how they compare to PCP +Adagrad.
In the following, ~ U; ~ W are in R N (cid:2) d , whereas U; V and W are in R N (cid:2) D .
Using d (cid:20) D ,
P 1 ; P 3 2 R D (cid:2) d , and we use the notation (cid:5) 1 = P 1 P ⊤ 1 and (cid:5) 3 = P 3 P ⊤ 3 .
The empirical risk L can be expressed as a function of three matrices M (1) , M (2) and M (3) corresponding to factors of a CP decomposition.
We focus on the following problems :
[Equation]
We focus on a step at time ( t ) on vectors ~ u i and u i .
We assume that at this time t , the tensor iterates are the same, that is ~ U = UP ⊤ 1 (resp.
~ W =
WP ⊤ 1 )
so that [[ ~ UP ⊤ 1 ; V; ~
WP ⊤ 3 ]] =
[[ U (cid:5) ⊤ 1 ; V; W (cid:5) ⊤ 3 ]] .
In this case, the gradient ∇ M (1) i L is the same in both optimization problems, we denote it by g ( t ) i .
Let G ( t ) i =
ϵI d + ∑ t(cid:28)
=1 g ( (cid:28) )
i g ( (cid:28) )
i ⊤ .
The updates for (PCP) are:
[Equation]
Note that due to the presence of P 1 inside the Diag operator, the update (3) is not rotation invariant.
Moreover, for random P 1 , the matrix P ⊤ 1 G ( t ) i P 1 will be far from diagonal with high probability, making its diagonal meaningless.
This is visualized in Figure 1b.
Similar updates can be derived for (PCP full ):
[Equation]
As a sanity check, clearly, for d = D and (cid:5) 1 =
I , the update (4) is equivalent to the Adagrad update for the CP model.
In the general case d (cid:20) D , in order to avoid storing U 2 R N (cid:2) D , we apply these updates implicitly with Ada imp , by storing u ( t ) i =
P ⊤ 1 u ( t ) i in R d .
Let us compare the updates :
[Equation]
Going back to our control experiment, we note on Figure 1a that PCP +Ada imp matches the performances of CP+Adagrad for all D , indicating that we ﬁxed this optimization issue.
5. A LTERNATIVES TO A DA imp
Another solution would be to use Adagrad projected on the column space of (cid:5) , but we show in
Appendix 9.1 that even with the diagonal approximation, this is impractical.
Note that the version of Adagrad which uses the full matrix G ( t ) is rotation invariant (see Appendix 9.2 for details), but it cannot be used at the scale of our problems.
It could be argued that the strength of the AdaGrad algorithm in our context mostly comes from its adaptation to the different frequencies of updates of each embedding.
In fact, this is one of the examples chosen in Duchi et al.
(2011) to display the advantages of AdaGrad compared to stochastic gradient descent.
A version of AdaGrad that would only keep one coefﬁcient per embedding (we call this version Ada row ) would be invariant to unitary transforms by design and would adapt to the various update frequencies of different embeddings.
In fact, this version of AdaGrad is used to save memory in Lerer et al.
We test this algorithm in Appendix 9.8.
The difference in performances shows that this adaptation is not sufﬁcient to recover the performances of the ﬁner diagonal AdaGrad in our setting.
The claim that the approximations made in Ada imp are indeed better is further backed by the experiments in the next section .
5. C OMPLEXITY
The time complexity of our Ada imp update for a batch of size B is O
( D (cid:1) d (cid:1) B ) which is similar, up to constants, to the complexity of updates for the AdaGrad algorithm.
We do not notice any runtime differences between our algorithm applied in dimensions ( d; D ) and a CP decomposition of dimension D (see Section 7).
The runtime for large enough D is dominated by the matrix product ( O ( D 2 (cid:1) B ) ) required to compute the cross-entropy in Equation (1).
( i; j; k ) sample from S g i ; g j ; g k gradients in ( Pu i ; Pu j ; w k ) u i ; ~
G i
Ada imp ( (cid:17); u i ; g u ; ~ G i ; P ) u j ; ~
G j
Ada imp ( (cid:17); u j ; g j ; ~ G j ; P ) w k ; G k
AdaGrad( (cid:17); w k ; g k ; G k )
Algorithm 2 Ada
Input: (cid:17); x ( t ) ; g ( t ) ; ~ G ( t (cid:0) 1) ; P ~ g ( t )
P ( P ⊤ g ( t ) )
~ G ( t ) ~
G ( t (cid:0) 1) + ~ g ( t ) ⊙ ~ g ( t ) x ( t +1) x ( t ) (cid:0) (cid:17)P ⊤ Diag ( ~ G ( t ) ) (cid:0) 1 = 2 ~ g ( t ) return x ( t +1) ;
~ G ( t )
P ROJECTED C OMPL E X
As the state-of-the-art variant of CP is ComplEx (Trouillon et al., 2016; Lacroix et al., 2018), we propose the following alternative to PCP base on ComplEx with Ada imp in practice.
Given the
ComplEx decomposition X =
Re ([ U; V; U ]) , a low-rank decomposition of the entity factor U as
P ~ U leads to the model PComplEx we use in the experiments of Section 7:
[Equation]
PComplEx is similar to ComplEx but with interactions described by full matrices of rank D that share a same basis.
We learn this decomposition with Algorithms 1 and 4.
For some datasets (WN18, WN18RR), Adam (Kingma & Ba, 2014) is beneﬁcial, in which case we use the implicit adaptation detailed in Appendix 9.7.
E XPERIMENTS
In this section, we compare ComplEx optimized with AdaGrad and PComplEx optimized with
Ada imp .
We optimize the regularized empirical risk of Equation (2).
Following Lacroix et al.
(2018), we regularize ComplEx with the weighted nuclear-3 norm, which is equivalent to regularizing ∥ u i ∥ 33 + ∥ u j ∥ 33 + ∥ w k ∥ 33 for each training example ( i; j; k ) .
For PComplEx based models, we regularize ∥ Pu i ∥ 33 + ∥ v j ∥ 33 + ∥
Pu k ∥ 33 by analogy.
We conduct all experiments on a Quadro GP 100 GPU.
The code for PComplEx and Ada imp is available in the supplementary materials, experiments on ComplEx use the code 1 from (Lacroix et al., 2018).
We include results from TuckER (Balaževi´c et al., 2019b), DRT and SRT which are the two models considered in Wang et al.
ConvE (Dettmers et al., 2018), HolEx (Xue et al., 2018), LFM (Jenatton et al., 2012) and MurP (Balaževi´c et al., 2019a) without re-implementation on our parts.
All the parameters for the experiments in this section are reported in Appendix 9.11.
D ATASETS
WN18 (Bordes et al., 2013) is taken from the Wordnet database which contains words and relations between them.
WN18RR
(Dettmers et al., 2018) is a ﬁltering of this dataset which removes train/test leakage.
YAGO3-10 (Dettmers et al., 2018) is taken from the eponymous knowledge base.
Finally,
SVO (Jenatton et al., 2012) contains observations of Subject, Verb, Object triples.
All statistics for these datasets can be found in Appendix 9.13.
Experiments on the FB15K and FB15K-237 datasets are deferred to Appendix 9.10.
7. R ESULTS
We report the ﬁltered Mean Reciprocal Rank (Nickel et al., 2016b) on Figure 3.
For SVO, we report the only ﬁgure available in previous work which is the ﬁltered hits at 5% (Jenatton et al., 2012).
[Figure] [Figure]
[Figure]
params
[Figure]
entities 0 .0 .0 .
[Figure]
H i t s a t%
SVO 101010params per entities 0.
M RR
YAGO
[Figure]
Parameters per entities 0 .0 .0 .1 .
M RR
WN
ComplEx
[Figure]
TuckER
[Figure]
D/SRT
[Figure]
entities
[Figure]
WN18RR
These measures are detailed in Appendix 9.11.
Only the grid-search parameters were given for
LFM, so we were not able to obtain a precise number of parameters for the number they report.
On WN18, SVO and YAGO3-10 we observe sizable performance gains for low embedding sizes : up to 0 : 14 MRR points on WN18, 0 : 05 MRR points on YAGO and 0 : 03 H@5% points on SVO.
The TuckER (Balaževi´c et al., 2019b) model performs similarly to PComplEx and ComplEx except on FB15K and WN18 where it underperforms (see Appendix 9.10).
We expect this discrepancy to come from a less extensive grid-search rather than any intrinsic differences in the models that are both based on the Tucker decomposition.
The consistency on all operating points of our method with ComplEx shows an advantage of our method, which enjoys the same learning rate robustness as AdaGrad, and does not require choosing a learning-rate decay, leading to easier experiments with only the regularization strength to tune.
The MurP model (Balaževi´c et al., 2019a) provides good performances for low embedding sizes on WN18RR, but underperforms on FB15K-
(see
Appendix 9.10).
All other models fail to match the performances of ComplEx and PComplEx with equivalent number of parameters.
Variance of performances in PComplEx due to random choice of P is similar to the variance of
ComplEx.
We present experiments on the WN18 dataset for 5 different seeds in Appendix 9.9.
C ONCLUSION
By observing that the core tensor of the Tucker decomposition can itself be decomposed, we obtain new models that are reminiscent of the canonical decomposition with low-rank factors.
We provide experimental evidence that a naive application of AdaGrad on this decomposition fails, due to individual coordinates losing their meaning.
We propose a new algorithm, Ada imp , which ﬁxes this issue.
Our model, when optimized with Ada imp , provides better performances than ComplEx in the low-rank regime, and matches its performances in the other regimes.
A PPENDIX
In subsections 9.1 and 9.2 where we discuss the Adagrad algorithm, we do so in a general setting, where we study, for ﬁxed P with orthonormal columns, the problem :
[Equation]
[Equation]
P ROJECTED A DAGRAD UPDATE
Let D denote Diag ( G ) 1 = 2 for the classical version of Adagrad and G 1 = 2 itself for the “full” version of the update.
When the parameter (cid:18) is constrained to a set (cid:2) , the update proposed in Eq.(1) in Duchi et al.
(2011) the one obtained by solving
[Equation]
To enforce the constraint that ~ (cid:18)
= P(cid:18) , we can consider the Lagrangian
[Equation]
whose stationary points satisfy D ( ~ (cid:18) (cid:0) z )
= (cid:21) and P ⊤ (cid:21) =
: So this entails P ⊤ D ( ~ (cid:18) (cid:0) ~ (cid:18) ( t ) ) =
(cid:17)P ⊤ g ( t ) and ﬁnally using ~ (cid:18)
=
P(cid:18)
we obtain an update in (cid:18) as follows
[Equation]
Clearly, PDP ⊤ is in general non-diagonal whether D is diagonal or not, and so this approach does not provide a computationally efﬁcient update.
If D = G 1 = 2 , then since PG 1 = 2 P ⊤ =
( PGP ⊤ ) 1 =
the update is the same as the full Adagrad update (5) that we derive in the following section and replacing ( PGP ⊤ ) 1 = 2 by its diagonal approximation recovers update (3).
9. T HE TWO F ULL A DAGRAD UPDATES AND THE QUALITY OF APPROXIMATIONS OF THE D IAG VERSIONS
If we consider the full versions of the Adagrad updates then (letting again (cid:5) = PP ⊤ ) its application to (cid:18) 7! f
( P(cid:18) ) and ~ (cid:18) 7! f
((cid:5)~ (cid:18) ) yield respectively
[Equation]
and
[Equation] [Equation]
where M y notes the pseudo-inverse of a matrix M .
As it turns out, the two updates are equivalent:
Indeed, ﬁrst ((cid:5) G
[Equation]
) 1 = 2 = P ( P ⊤ GP ) 1 = 2 P ⊤ because P ⊤ P
= I implies that
[Equation]
and the p.s.d.
squareroot is unique.
Second, taking the pseudo-inverse of this identity, we have
[Equation]
because, if H = ( P ⊤ GP )
= 2 is an invertible matrix, then ( PHP ⊤ )
y =
PH (cid:0) 1 P ⊤ given that
PHP ⊤ PH (cid:0) 1 P ⊤ =
PP ⊤ : Finally multiplying both sides of Eq.
(7) by (cid:5) shows that
[Equation]
This shows that although Adagrad is not invariant for any invertible P , it is invariant for any P such that P ⊤ P
= I . Eq.(5) seems in general simpler than (6), but note that if D = d , then (cid:5) is the identity and (6) shows that both full updates are in that case actually equivalent to the full update of
Adagrad applied to plain CP.
Finally, the equivalence of the full updates discussed above strongly suggests that if our proposed update performs better than the naive application of Adagrad to PCP, it is because
P Diag ((cid:5) G ( t ) (cid:5)) (cid:0) 1 = 2 P ⊤ is a better approximation of ( G ( t ) ) (cid:0) 1 = 2 than Diag ( PG ( t ) P ⊤ ) (cid:0) 1 = 2 while not being much more computationally expensive.
9. CP-T UCKER
We have X =
[[ C ; U; V; W ]] and C
=
[[ P 1 ; P 2 ; P 3 ]] .
For all i; j; k , we have:
[Equation]
9. H OL E X AND L ATENT FACTOR MODEL 9.4. H OL E X
The HolEx model (Xue et al., 2018) writes X i;j;k =
∑ Rr
[Equation] [Equation]
c r ⊙ u
i
[Equation]
⋆ u j ; w rk ⟩ , where ⋆ denotes the circular correlation 2 .
Exploiting the equivalence between HolE and ComplEx (Hayashi & Shimbo, 2017), denoting by F the discrete Fourier transform (with values in C ), we can write for embeddings of size d :
[Equation]
For all vectors, we write ^ u i
=
F ( u i ) 2 C d .
We can re-write the circular correlation F ( c r ) ⋆ F ( u i ) as C r ^ u i where C r 2 C d (cid:2)
d is the circulant matrix associated with c r 2 R d .
We have :
[Equation]
Finally, with C 1 =
[ C 1 ; : : : ; C R ] 2 R Rd (cid:2)
d the vertical stacking of all C r , C 2 =
[ I d ; : : : ; I d ] and ^ w k =
[ ^ w 1 k ; : : : ; ^ w Rk ] :
R
[Equation]
HolEx with embeddings of size d is close to the CPT with D =
Rd , allowing for two different complex matrices to act on left and right hand side embeddings.
9.4. L ATENT F ACTOR MODEL
The latent factor model deﬁnes the score for a triple ( i; j; k ) as:
[Equation]
Removing the bias terms z and z ′ and gathering u r and v r into matrices P 1 and P 3 leads to the model CPT.
In the PCP model, we ﬁx P 1 and P 3 instead of learning them.
We do not use a sparsity inducing penalty on (cid:11) but rather a variational form of the nuclear norm on the whole tensor.
[Equation]
9. T UCKER 2 WITH CP-T UCKER
Let for all 1 (cid:20) r (cid:20) d , M ( r ) be a matrix of zeros except its r (cid:0) th column which is all one.
Let P 1 be the vertical concatenation of all ( M ( r ) )
r =1 ::d and P 2 the vertical concatenation of d identity matrix in R d .
Remember that for all k , w k is an element of R d 2 .
For all 0 (cid:20) r < d , let w rk be the restriction of w k to its [ rd; ( r + 1) d ] coordinates.
Then P 1 and P 2 are elements of R d 2 (cid:2)
d
and we have for all i; j; k : ⟨ P 1 u i ; P 2 v j ; w k ⟩ =
d ∑ r 1 = ⟨ M ( r 1 ) u i ; I d v j ; w r 1 k ⟩ =
d ∑ r 1 = u i;r 1 ⟨ v j ; w r 1 k ⟩ by deﬁnition of M ( r 1 ) =
d ∑ r 1 = u i;r( d ∑ r 2 = v j;r 2 w k;r 1 r) by deﬁnition of w r 1 k = u ⊤ i
Mat ( w k ) v j
9. C OMPLETE PC OMPL E X A LGORITHM
Let U 2 C N (cid:2) d be the the entity embedding matrix and V 2 C 2 L (cid:2)
D be the predicate embedding matrix.
Let P : R N (cid:2)
d 7!
R N (cid:2) D such that P
( U i ) =
PU i .
Let G Ut and G Vt be the stochastic gradients with respect to U and V at time t .
Algorithm
Input: learning orthogonal
C
; C V while U; V not converged do ~
G Ut
P
[Equation]
end while
return
U
9. A DAM - I MPLICIT
For WN18RR, we used the ideas presented in this paper, but adapted to the Adam (Kingma &
Ba, 2014) optimization algorithm.
Similar to Ada imp , ﬁrst and second moment estimates are accumulated in R D and we project back to R d only for the parameter update.
For simplicity, we present here the dense version of the algorithm applied to the entity embeddings U 2 R N (cid:2) d .
Let P : R N (cid:2)
d 7!
R N (cid:2) D such that P
( U i ) =
PU i .
Let G t be the stochastic gradient with respect to U at time t .
9. A DA row
For the same control experiment as in Section 5, we observe the performances of Ada row which is rotation invariant by design.
Input: (cid:17); (cid:12) 1 ; (cid:12) 2 ; P ; ϵ m 0 ; v 0 ; t while U not converged do t t + 1 ~
G t
P ( G t ) m t (cid:12) 1 (cid:1) m t (cid:0) 1 + (1 (cid:0) (cid:12) 1 ) (cid:1) ~
G t v t (cid:12) 2 (cid:1) v t (cid:0) 1 + (1 (cid:0) (cid:12) 2 ) (cid:1) ~
G t ⊙ ~
G t ^ m t m t =
(1 (cid:0) (cid:12) t 1 ) ^ v t v t =
(1 (cid:0) (cid:12) t 2 )
U
U (cid:0) (cid:17) (cid:1) P ⊤ ( ^ m t =
( p ^ v t + ϵ )) end while return U
[Figure]
9. V ARIANCE OF PC OMPL E X
We run 5 grid search and plot the 5 associated convex-hulls on the WN18 dataset optimized with
Ada imp .
Note that despite the added randomness of the choice of P , there is not more variance in
PComplEx than in ComplEx.
[Figure]
M RR
Variance of PComplEx on WN
[Figure]
9. FB15 K DATASETS
We use two subsets of the Freebase knowledge base : FB15K (Bordes et al., 2013) and FB15K237 (Toutanova & Chen, 2015).
FB15K-237 is a harder version of FB15K, where some triples have been removed to avoid leakage between the train and test set.
There is no difference of performances between PComplEx and ComplEx on these datasets.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
9. E XPERIMENTAL DETAILS
Metrics Let rank ( ^ X i;j; : ; k ) be the rank of ^ X i;j;k in the sorted list of values of ^ X i;j; : .
We report the MRR for most datasets :
[Equation]
For SVO, the task is slightly different as the ranking happens on the predicate mode.
The metric reported in Jenatton et al.
(2012) is Hits@5% deﬁned as :
[Equation]
The metrics we report are ﬁltered by ignoring other true positives when computing the ranks, as done in Bordes et al.
Number of parameters per entities We count the number of ﬂoats in the model, and divide by the number of entities in the dataset.
For different methods, the number of parameters are :
[Equation]
(cid:1) L (cid:1) d (cid:15)
ConvE: taken from Dettmers et al.
(cid:15) D/SRT: taken from Wang et al.
(2019) (cid:15) HolEx: d (cid:1) ( N + L )
Grid Search For SVO:
(cid:15)
For Complex vary d in [5 ; 25 ; 50 ; 100 ; 500 ; 1000 ; 2000] .
For PComplEx, we vary d in [5 ; 25 ; 50 ; 100 ; 500] .
(cid:15)
The strength of the regularizer, (cid:23) varies in [5 e (cid:0) 4 ; 1 e (cid:0) 3 ; 5 e (cid:0) 3 ; 1 e (cid:0) 2 ; 5 e (cid:0) 2 ; 1 e (cid:0) 1] .
(cid:15) Finally, for PComplEx, we vary the dimension D in [5 ; 25 ; 50 ; 100 ; 500 ; 1000 ; 2000 ; 4000 ; 8000] .
For all other datasets, we run 500 epochs :
(cid:15)
For FB15K and FB15K-237, we vary d in [5 ; 25 ; 50 ; 100 ; 500 ; 1000 ; 2000] .
For YAGO3-10, WN18 and WN18RR, we add ranks 8 and 16 to that list.
(cid:15)
The strength of the regularizer, (cid:23) varies in [5 e (cid:0) 4 ; 1 e (cid:0) 3 ; 5 e (cid:0) 3 ; 1 e (cid:0) 2 ; 5 e (cid:0) 2 ; 1 e (cid:0) 1] .
(cid:15) Finally, for PComplEx, we vary the dimension D in [5 ; 25 ; 50 ; 100 ; 500 ; 1000 ; 2000] .
For
for each
We apply a multiplicative factor (cid:13) to the dropout rates and run a grid search over this parameter
(cid:15)
For WN18RR, we vary d e in [12 ; 15 ; 18 ; 20 ; 40 ; 100 ; 200] .
For FB15K-237, we vary d e in [10 ; 20 ; 30 ; 40 ; 50 ; 80 ; 100 ; 200] .
For WN18, d e varies in [5 ; 8 ; 16 ; 25 ; 50 ; 100] .
For FB15K d e varies in .
(cid:15)
For WN18RR, WN18 we search (cid:13) over [0 ; 0 : 5 ; 1] .
For FB15K-237 where this lead to non increasing performances as a function of number of parameters, we reﬁned this grid to [0 ; 0 : 1 ; 0 : 2 ; :::; 0 : 7] for d e (cid:20) 80 .
For FB15K, we use a grid of [0 ; 0 : 2 ; 0 : 4 ; ::: 1] .
For MurP, we use the hyperparameters described in Balaževi´c et al. (2019a) for each dataset and vary d in [10 ; 12 ; 15 ; 18 ; 20 ; 40 ; 100 ; 200] on WN18RR and in [10 ; 20 ; 40 ; 100 ; 200] for FB15K-237.
Other details Batch-size is ﬁxed to 1000 , the learning-rate is 1 e (cid:0) 1 both for Adagrad and for
Ada imp , we run 100 epochs to ensure convergence.
The best model based on validation MRR is selected and we report the corresponding test MRR.
9. R UNNING TIMES
We give here the running times of each method per epoch, on the WN18RR dataset for comparable dimensionalities and run on a P100 GPU.
We use the original implementation for MurP (Balaževi´c et al., 2019a) and TuckEr (Balaževi´c et al., 2019b)
For ComplEx , we use the implementation from
Lacroix et al.
(cid:15) ComplEx ( d = 200 ) : 4 s/epoch.
(cid:15) PComplEx ( d = D = 200 , Ada imp ) :
5 s/epoch.
(cid:15) MurP ( d = 200 ): 38 s/epoch.
(cid:15) TuckEr
( d e = 200 ; d r = 200 ): 24 s/epoch
Note that these running times are implementation dependent.
In the ﬁgure below, we give the learning-curves of MurP, TuckEr, ComplEx and PComplEx for one operating point on the WN18RR dataset.
The convergence speed of these methods is given in the ﬁgure below at an operating point on WN18RR where all methods are close in ﬁnal performances.
[Figure]
[Figure]
9. D ATASET STATISTICS
[Table]