Y OU CAN T EACH AN O LD D OG N EW T RICKS ! O N T RAINING K NOWLEDGE G RAPH E MBEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations in a knowledge graph.
A vast number of KGE techniques for multi-relational link prediction has been proposed in the recent literature, often with state-of-the-art performance.
These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization.
In this paper, we take a step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance.
We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings.
We found that when trained appropriately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results.
For example, RESCAL (Nickel et al., 2011), one of the ﬁrst KGE models, showed strong performance when trained with state-of-the-art techniques; it was competitive to or outperformed more recent architectures.
We also found that good (and often superior to prior studies) model conﬁgurations can be found by exploring relatively few random samples from a large hyperparameter space.
Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual beneﬁts
I NTRODUCTION
Knowledge graph embedding (KGE) models learn algebraic representations, termed embeddings, of the entities and relations in a knowledge graph.
They have been successfully applied to knowledge graph completion (Nickel et al., 2015) as well as in downstream tasks and applications such as recommender systems (Wang et al., 2017) or visual relationship detection (Baier et al., 2017).
A vast number of different KGE models for multi-relational link prediction has been proposed in the recent literature; e.g., RESCAL (Nickel et al., 2011), TransE (Bordes et al., 2013), DistMult,
ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018), TuckER (Balazevic et al., 2019),
RotatE (Sun et al., 2019), KBGAT (Nathani et al., 2019), and many more.
Model architectures generally differ in the way the entity and relation embeddings are combined to model the presence or absence of an edge (more precisely, a subject-predicate-object triple) in the knowledge graph; they include factorization models (e.g., RESCAL, DistMult, ComplEx, TuckER), translational models (TransE, RotatE), and more advanced models such as convolutional models (ConvE).
In many cases, the introduction of new models went along with the new approaches for training these models— e.g., new training types (such as negative sampling or 1vsAll scoring), new loss functions (such as pairwise margin ranking or binary cross entropy), new forms of regularization (such as unweighted and weighted L2), or the use of reciprocal relations (Dettmers et al., 2018)—and ablation studies were not always performed.
Table 1 shows an overview of selected models and training techniques along with the publications that introduced them.
The diversity in model training makes it difﬁcult to compare performance results for various model architectures, especially when results are reproduced from prior studies that used a different experimental setup.
Model hyperparameters are commonly tuned using grid search on a small grid involving hand-crafted parameter ranges or settings known to “work well” from prior studies.
A grid suitable for one model may be suboptimal for another, however.
Indeed, it has been observed
Publication
[Table]
Nickel et al.
RESCAL
[Table]
ALS
No
Bordes et al.
TransE
[Table]
Yang et al.
[Table]
Trouillon et al.
[Table]
Kadlec et al.
[Table]
Dettmers et al.
[Table] [Table]
that newer training strategies can considerably improve model performance (Kadlec et al., et al
In this paper, we take a step back and aim to summarize and quantify empirically the impact of different model architectures and different training strategies on model performance.
We performed an extensive set of experiments using popular model architectures and training strategies in a common experimental setup.
In contrast to most prior work, we considered many training strategies as well as a large hyperparameter space, and we performed model selection using quasi-random search (instead of grid search) followed by Bayesian optimization.
We found that this approach was able to ﬁnd good (and often superior to prior studies) model conﬁgurations with relatively low effort.
Our study complements and expands on the results of Kotnis & Nastase (2018) (focus on negative sampling) and Mohamed et al.
(2019) (focus on loss functions) as well as similar studies in other areas, including language modeling (Melis et al., 2017), generative adversarial networks (Lucic et al., 2018), or sequence tagging (Reimers & Gurevych, 2017).
We found that when trained appropriately, the performance of a particular model architecture can by far exceed the performance observed in older studies.
For example, RESCAL (Nickel et al., 2011), which constitutes one of the ﬁrst KGE models but is rarely considered in newer work, showed very strong performance in our study: it was competitive to or outperformed more recent architectures such as ConvE (Dettmers et al., 2018) and TuckER (Balazevic et al., 2019).
More generally, we found that the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared to prior results.
This suggests that (at least currently) training strategies have a signiﬁcant impact on model performance and may account for a substantial fraction of the progress made in recent years.
We also found that suitable training strategies and hyperparameter settings vary signiﬁcantly across models and datasets, indicating that a small search grid may bias results on model performance.
Fortunately, as indicated above, large hyperparameter spaces can be (and should be) used with little additional training effort.
Our study focuses solely on pure KGE models, which do not exploit auxiliary information such as textual data or logical rules (Wang et al., 2017).
Since many of the studies on these non-pure models did not (and, to be fair, could not) use current training strategies and consequently underestimated the performance of pure KGE models, their results and conclusions need to be revisited.
K NOWLEDGE G RAPH E MBEDDINGS : M ODELS , T RAINING , E VALUATION
The literature on KGE models is expanding rapidly.
We review selected architectures, training methods, and evaluation protocols; see Table 1.
The table examplarily indicates that new model architectures are sometimes introduced along with new training strategies (marked bold).
Reasonably recent survey articles about KGE models include Nickel et al.
(2015) and Wang et al.
Multi-relational link prediction.
KGE models are typically trained and evaluated in the context of multi-relational link prediction for knowledge graphs (KG).
Generally, given a set E of entities and a set R of relations, a knowledge graph K ⊆
E × R × E is a set of subject-predicate-object (spo) triples.
The goal of multi-relational link prediction is to “complete the KG”, i.e., to predict true but unobserved triples based on the information in K .
Common approaches include rule-based methods methods
, and hybrid methods (
Knowledge graph embeddings (KGE).
A KGE model associates with each entity i ∈ E and relation k ∈ R an embedding e i ∈ R d e and r k ∈ R d r in a low-dimensional vector space, respectively; here d e , d r ∈ N + are hyperparameters for the embedding size .
Each particular model uses a scoring function s : E × R × E
→ R to associate a score s ( i, k, j ) with each potential triple ( i, k,
j ) ∈
E × R × E .
The higher the score of a triple, the more likely it is considered to be true by the model.
The scoring function takes form s ( i, k, j ) =
f ( e i , r k , e j ) , i.e., depends on i , k , and j only through their respective embeddings.
Here f , which represents the model architecture, may be either a ﬁxed function or learned (e.g., f may be a parameterized function).
Evaluation.
The most common evaluation task for KGE methods is entity ranking , which is a form of question answering.
The available data is partitioned into a set of training, validation, and test triples.
Given a test triple ( i, k, j ) (unseen during training), the entity ranking task is to determine the correct answer—i.e., the missing entity j and i , resp.—to questions ( i, k, ?) and (? , k, j ) .
To do so, potential answer triples are ﬁrst ranked by their score in descending order.
All triples but ( i, k, j ) that occur in the training, validation, or test data are subsequently ﬁltered out (so that other triples known to be true do not affect the ranking).
Finally, metrics such as the mean reciprocal rank (MRR) of the true answer or the average HITS@ k are computed; see appendix.
KGE models.
KGE model architectures differ in their scoring function.
We can roughly classify models as decomposable or monolithic : the former only allow element-wise interactions between (relation-speciﬁc) subject and object embeddings, whereas the latter allow arbitrary interactions.
More speciﬁcally, decomposable models use scoring functions of form s ( i, k, j ) =
f ( (cid:80) z g
[ h 1 ( e i , e r ) ◦
h 2 ( e r , e j )] z )) , where ◦ is any element-wise function (e.g., multiplication), h 1 and h 2 are functions that obtain relation-speciﬁc subject and object embeddings, resp., and g and f are scalar functions (e.g., identity or sigmoid).
The most popular models in this category are perhaps RESCAL (Nickel et al., 2011), TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), and ConvE (Dettmers et al., 2018).
RESCAL’s scoring function is bilinear in the entity embeddings: it uses s ( i, k, j ) =
e Ti R k e j , where R k ∈ R d e × d e is a matrix formed from the entries of r k ∈ R d r (where d r = d 2 e ).
DistMult and ComplEx can be seen as constrained variants of RESCAL with smaller relation embeddings ( d r = d e ).
TransE is a translation-based model and uses negative distance −(cid:107)
e i + r k − e j (cid:107) p between e i + r k and e j as score, commonly using the L1 norm ( p = 1 ) or the L2 norm ( p = 2 ).
Finally, ConvE uses a 2D convolutional layer and a large fully connected layer to obtain relation-speciﬁc entity embeddings (i.e., in h 1 above).
Other recent examples for decomposable models include TuckER (Balazevic et al., 2019), RotatE (Sun et al., 2019), and SACN (Shang et al., 2019).
Decomposable models are generally fast to use: once the relation-speciﬁc embeddings are (pre-)computed, score computations are cheap.
Monolithic models—such as KBGAT (Nathani et al., 2019)—do not decompose into relation-speciﬁc embeddings: they take form s ( i, k, j ) =
f ( e i , r k , e j ) .
Such models are more ﬂexible, but also more costly to train and use.
Training type.
There are three commonly used approaches to train KGE models, which differ mainly in the way negative examples are generated.
First, training with negative sampling (NegSamp) (Bordes et al., 2013) obtains for each positive triple t
= ( i, k, j ) from the training data a set of (pseudo-)negative triples obtained by randomly perturbing the subject, relation, or object position in t (and optionally checking that the so-obtained triples does not exist in the KG).
An alternative approach (Lacroix et al., 2018), which we term 1vsAll , is to omit sampling and take all triples that can be obtained by perturbing the subject and object positions as negative examples for t (even if these tuples exist in the KG).
1vsAll is generally more expensive than NegSamp, but it is feasible (and even surprisingly fast in practice) if the number of entities is not excessively large.
Finally, Dettmers et al.
(2018) proposed a training type that we term KvsAll 1 : this approach (i) constructs batches from non-empty rows ( i, k, ∗ ) or ( ∗ , k, j ) (instead of from individual triples), and (ii) labels all such triples as either positive (occurs in training data) or negative (otherwise).
Loss functions .
Several loss functions for training KGEs have been introduced so far.
RESCAL originally used squared error between the score of each triple and its label (positive or negative).
TransE used pairwise margin ranking with hinge loss (MR), where each pair consists of a positive triple and one of its negative triples (only applicable to NegSamp and 1vsAll) and the margin η is a hyperparameter.
Trouillon et al.
(2016) proposed to use binary cross entropy (BCE) loss: it applies a sigmoid to the score of each (positive or negative) triple and uses the cross entropy between the resulting probability and that triple’s label as loss.
BCE is suitable for multi-class and multi-label classiﬁcation.
Finally, Kadlec et al.
(2017) used cross entropy (CE) between the model distribution (softmax distribution over scores) and the data distribution (labels of corresponding triples, normalized to sum to 1).
CE is more suitable for multi-class classiﬁcation (as in NegSamp and 1vsAll), but it has also been used in the multi-label setting (KvsAll).
Mohamed et al.
(2019) found that the choice of loss function can have a signiﬁcant impact on model performance, and that the best choice is data and model dependent.
Our experimental study provides additional evidence for this ﬁnding.
Reciprocal relations.
Kazemi & Poole (2018) and Lacroix et al.
(2018) introduced the technique of reciprocal relations into KGE training.
Observe that during evaluation and also most training methods discussed above, the model is solely asked to score subjects (for questions of form (? , k, j ) ) or objects (for questions of form ( i, k, ?) ).
The idea of reciprocal relations is to use separate scoring functions s sub and s obj for each of these tasks, resp.
Both scoring functions share entity embeddings, but they do not share relation embeddings: each relation thus has two embeddings.
The use of reciprocal relations may decrease the computational cost (as in the case of ConvE), and it may also lead to better model performance Lacroix et al.
(e.g., for relations in which one direction is easier to predict).
On the downside, the use of reciprocal relations means that a model does not provide a single triple score s ( i, k, j ) anymore; generally, s sub ( i, k, j ) (cid:54) =
s obj ( i, k, j ) .
Kazemi & Poole (2018) proposed and explored taking the average of the two triple scores.
Regularization.
The most popular form of regularization in the literature is L2 regularization on the embedding vectors, either unweighted or weighted by the frequency of the corresponding entity or relation (Yang et al., 2015).
Lacroix et al.
(2018) proposed to use L3 regularization.
TransE normalized the embeddings to unit norm after each update.
ConvE used dropout (Srivastava et al., 2014) in its hidden layers (and only in those).
In our study, we additionally consider L1 regularization and the use of dropout on the entity and/or relation embeddings.
Hyperparameters.
Many more hyperparameters have been used in prior work.
This includes, for example, different methods to initialize model parameters, different optimizers, different optimizer parameters such as the learning rate or batch size, the number of negative examples for NegSamp, the regularization weights for entities and relations, and so on.
To deal with such a large search space, most prior studies favor grid search over a small grid where most of these hyperparameters remain ﬁxed.
As discussed before, this approach may lead to bias in the results, however.
E XPERIMENTAL STUDY
In this section, we report on the design and results of our experimental study.
We focus on the most salient points here; more details can be found in the appendix.
3. E XPERIMENTAL S ETUP
Datasets.
We used the FB15K-237 (Toutanova & Chen, 2015) (extracted from Freebase) and
WNRR (Dettmers et al., 2018) (extracted from WordNet) datasets in our study.
We chose these datasets because (i) they are frequently used in prior studies, (ii) they are “hard” datasets that have been designed speciﬁcally to evaluate multi-relational link prediction techniques, (iii) they are diverse in that relative model performance often differs, and (iv) they are of reasonable size for a large study.
Dataset statistics are given in Table 4 in the appendix.
Models.
We selected RESCAL (Nickel et al., 2011), TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016) and ConvE (Dettmers et al., 2018) for our study.
These models are perhaps the most well-known KGE models and include both early and more recent models.
Note that we did not consider monolithic models in our study due to their excessive training cost (for example, KBGAT trained for 14 hours using the best conﬁguration reported by the authors).
Evaluation.
Unless stated otherwise, we report ﬁltered MRR and ﬁltered HITS@10 on test data; see Sec. 2 for details.
Hyperparameters.
We used a large hyperparameter space to ensure sure that suitable hyperparameters for each model are not excluded a priori.
We included all major training types (NegSamp, 1vsAll, KvsAll), use of reciprocal relations, loss functions (MR, BCE, CE), regularization techniques (none/L1/L2/L3, dropout), optimizers (Adam, Adagrad), and initialization methods (4 in total) used in the KGE community as hyperparameters.
We considered three embeddings sizes (128, 256, 512) and used separate weights for dropout/regularization for entity and relation embeddings.
Table 5 in the appendix provides additional details.
To the best of our knowledge, no prior study used such a large hyperparameter search space.
Training.
All models were trained for a maximum of 400 epochs.
We validated models using ﬁltered MRR (on validation data) every ﬁve epochs and performed early stopping with a patience of 50 epochs.
To keep search tractable, we stopped training on models that did not reach ≥ 5% ﬁltered
MRR after 50 epochs; in our experience, such conﬁgurations did not produce good models.
Model selection.
We used the Ax framework ( https://ax.dev/ ) to conduct quasi-random hyperparameter search via a Sobol sequence.
Quasi-random search methods aim to distribute hyperparameter settings evenly and try avoid “clumping” effects (Bergstra & Bengio, 2012).
More speciﬁcally, for each dataset and model, we generated 30 different conﬁgurations per valid combination of training type and loss function (2 for TransE, which only supports NegSamp+MR and NegSamp+CE; 7 for all other models).
After the quasi-random hyperparameter search, we added a short Bayesian optimization phase (20 trials, using expected improvement; also provided by Ax) to tune the numerical hyperparameters further.
Reproducibility.
We reimplemented all models, training strategies, and hyperparameter search in a framework based on PyTorch.
The framework is easily extendable and will be released as open source.
Model selection and training is completely controlled via conﬁguration ﬁles.
All conﬁguration ﬁles used in our study, as well as the raw data that we obtained when running our experiments, will also be openly released.
3. C OMPARISON OF M ODEL P ERFORMANCE
Table 2 shows the ﬁltered MRR and ﬁltered Hits@10 on test data of various models both from prior studies and the best models (on validation MRR) found in our study.
First reported performance vs. our observed performance.
We ﬁrst compare the performance results of models when they were ﬁrst run on the FB15K-237 and the WNRR datasets (“First” block of Table 2) and the performance we obtained in our study (“Ours” block).
We found that the performance of a single model can vary wildly across studies.
For example, ComplEx was ﬁrst run on FB15k-237 by Dettmers et al.
(2018), where it achieved a ﬁltered MRR of 24.7 % .
This is a relatively low number by today’s standards (Lacroix et al., 2018).
In our study, ComplEx achieved a competitive MRR of 35.1 % , which is also a large improvement over the ﬁrst reports.
Studies that report the lower performance number of ComplEx (i.e., 24.7 % ) thus do not adequately capture its performance.
Similar remarks hold for RESCAL and DistMult as well as (albeit to a smaller extent)
ConvE and TransE. To the best of our knowledge, our results for RESCAL, TransE, and ConvE constitute the best results for these models obtained so far.
Relative model performance (our study).
Next, we compared the performance across the models models used in our study (“Ours” block).
We found that the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared to prior results (“First” block).
For example, ConvE showed the best overall performance in prior studies, but is consistently outperformed by RESCAL, DistMult, and ComplEx in our study.
As another example, RESCAL (Nickel et al., 2011), which constitutes one of the ﬁrst KGE models but is rarely considered in newer work, showed strong performance in our study and outperforms all alternative models except ComplEx, which also showed strong performance.
This suggests that (at least currently) training strategies have a signiﬁcant impact on model performance and may account for a large fraction of the progress made in recent years.
[Table]
RESCAL (Wang et al., 2019)
[Table]
TransE (Nguyen et al., 2018)
[Table]
Dettmers
[Table]
[Table]
[Table]
Dettmers
[Table] [Table]
(Balazevic et al., 2019)
[Table]
SACN (Shang
[Table] [Table]
Salehi et al., 2018)
[Table]
N
[Table]
Lacroix
[Table] [Table]
Nathani et al
[Table]
Relative model performance (overall).
Table 2 also shows the best performance results obtained in prior studies of selected recent models (“Recent” block) and very large models with very good performance (“Large” block).
We found that TuckER (Balazevic et al., 2019), RotatE
(Sun et al., 2019), and SACN (Shang et al., 2019)
all achieve state-of-the-art performance, but the performance difference to the best prior models (“Ours” block) in terms of MRR is small or even vanishes.
Even for HITS@10, which we did not consider for model selection, the advantage of more recent models is often small, if present.
The models in the last block (“Large”) show the best performance numbers we have seen in the literature.
For DistMult and ComplEx, these numbers have been obtained using very large embedding sizes (up to 4000 ).
KBGAT showed exceptional performance on FB15K-237, clearly outperforming all other models.
The model is monolithic, however, and has at least an order of magnitude higher training cost.
Limitations.
We note that all models used in our study are likely to yield better performance when hyperparameter tuning is further improved.
For example, we found a ComplEx conﬁguration (of size 100) that achieved 49 .
0% MRR and 56 .
5% Hits@10 on WNRR, and a RESCAL conﬁguration that achieved 36 . 1% MRR and 54 .
6% Hits@10 on FB15k-237 in preliminary experiments.
Sun et al.
(2019) report on a TransE conﬁguration that achieved 33 . 3% MRR (52.2% Hits@10) on FB15K237.
Since the focus of this study is to compare models in a common experimental setup and without manual intervention, we do not further report on these results.
3. I MPACT OF H YPERPARAMETERS
Anatomy of search space.
Figure 1 shows the distribution ﬁltered MRR for each model and dataset (on validation data, Sobol conﬁgurations only).
Each distribution consists of roughly 200 different quasi-random hyperparameter conﬁgurations (except TransE, for which 60 conﬁgurations have been used).
Perhaps unsurprisingly, we found that all models showed a wide dispersion on both datasets and only very few conﬁgurations achieved state-of-the-art results.
There are, however, notable differences across datasets and models.
On FB15K-237, for example, the median ConvE conﬁguration is best; on WNRR, DistMult and Complex have a much higher median MRR.
Generally, the impact of the hyperparameter choice is more pronounced on WNRR (higher variance) than on FB15K-237.
[-Adagrad and the 1vsN training type seem to be beneﬁcial on FB15K-
-]
RESCAL
TransE
DistMult
ComplEx
ConvE
FB 15 KMRR (valid) 36.
Embedding size
[Table]
[Table]
[Table]
[Table]
[Table]
Batch size
[Table]
Loss
CE
[Table]
CE
[Table]
CE
[Table]
CE
[Table]
Optimizer
Adag
[Table]
Initializer
[Table]
Reciprocal
[Table]
W N
RR
MRR
[Table]
valid)
[Table]
[Table]
Embedding [Table] [Table]
[Table] [Table]
[Table]
Best conﬁgurations (quasi-random search).
The hyperparameters of the best performing models during our quasi-random search are reported in Table 3 (selected hyperparameters) and Tablesand 7 (in the appendix, all hyperparameters).
First, we found that the optimum choice of hyperparameters is often model and dataset dependent: with the exception of the loss function (discussed below), almost no hyperparameter setting was consistent across datasets and models.
For example,
{+Adagrad and the 1vsN training type seem to be beneﬁcial on FB15K-
+}
but not on WNRR, where
Adam and KvsAll were more prominent.
Our ﬁndings provide further evidence that grid search on a small search space is not suitable to compare model performance because the result may be heavily inﬂuenced by the speciﬁc grid points being used.
Moreover, the budget that we used for quasi-random search was comparable to a grid search over a small (or at least not very large) grid: we tested merely 30 different model conﬁgurations for every combination of training type and loss function.
It is therefore both feasible and beneﬁcial to work with a large search space.
Best conﬁgurations (Bayesian optimization).
We already obtained good conﬁgurations solely from quasi-random search.
More precisely, roughly half of our best models (as reported in Table 2) were found during quasi-random search, the other half beneﬁted from subsequent tuning with
Bayesian optimization.
But even in the latter cases, the performance on test data almost always im- proved only marginally.
A notable exception is TransE, for which Bayesian optimization improved every metric on both datasets by about 2 percentage points (4 points for Hits@10 on WNRR).
The best hyperparameters after the Bayesian optimization phase are reported in Table 8 in the appendix.
Impact of speciﬁc hyperparameters.
It is difﬁcult to asses the impact of each hyperparameter individually.
As a proxy for the importance of each hyperparameter setting, we report in Table 3 and
Tables 6 and 7 (appendix) the best performance in terms of ﬁltered MRR on validation data for a different choice of value for each hyperparameter (difference in parenthesis).
For example, the best conﬁguration during quasi-random search for ComplEx on FB15K-
used reciprocal relations.
The best conﬁguration that does not use reciprocal relations has an MRR that is 0 .
8 points lower (34.3 instead of 35.1).
This does not mean that the gap is explained by the use of the reciprocal relations alone, as other hyperparameters may also be different, but it does show the best performance we obtained without relying on reciprocal relations.
We can see that most hyperparameters appear to have moderate impact on model performance: no hyperparameter setting produced a signiﬁcant performance improvement that others did not also produce.
The clear exception here is the use of the loss function, which is consistent across models and datasets (always CE), partly with large performance improvements.
Another notable exception is the initializer for TransE: in our experiments,
TransE performed best without embedding normalization but with uniform Xavier initialization.
Impact of loss function.
In order to further explore the impact of the loss function, we show the distribution of the ﬁltered MRR on validation data for every (valid) combination of training type and loss function in Figure 9.
We found that the use of CE generally leads to better conﬁgurations than using other losses.
This is surprising (and somewhat unsatisfying) in that we use the CE loss, which is a multi-class loss, for a multi-label task.
Similar observations have been made for computer vision tasks (Joulin et al., 2016; Mahajan et al., 2018) though.
Note that the combination of KvsAll with
CE (on the very right in ﬁgure) has not been explored previously.
C ONCLUSION
We reported the results of an extensive experimental study with popular KGE model architectures and training strategies across a wide range of hyperparameter settings.
We found that when trained appropriately, the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared to prior results.
This suggests that (at least currently) training strategies have a signiﬁcant impact on model performance and may account for a substantial fraction of the progress made in recent years.
To enable an insightful model comparison, we recommend that new studies compare models in a common experimental setup using a sufﬁciently large hyperparameter search space.
Our study shows that such an approach is possible with reasonable computational cost.
Moreover, we feel that many of the more advanced architectures and techniques proposed in the literature need to be revisited to reassess their individual beneﬁts
A A PPENDIX
Evaluation metrics.
We formally deﬁne the evaluation metrics used in our study. Given a triple ( i, k, j ) , denote by rank( j | i, k ) the ﬁltered rank of object j , i.e., the rank of model score s ( i, k, j ) among the collection of all pseudo-negative object scores { s ( i, k, j (cid:48) ) : j (cid:48) ∈ E and ( i, k, j (cid:48) ) does not occur in training, validation, or test data } .
If there are ties, we take the mean rank of all triples with score s ( i, k, j ) .
Deﬁne rank( i | k, j ) likewise.
Denote by K test the set of test triples.
Then
[Equation]
[Equation]
[Equation]
[Equation]
|
[Equation]
(cid:17) , where indicator 1 ( E ) is 1 if condition E is true, else 0.
[Table]
[Table]
[Table]
ng
[Table]
ub
[Table]
s E ) – – – –
O p ti m i ze r
A d a g .
A d a m (-. 1 )
A d a g .
A d a g .
A d a g .
B a t
c h
s i ze (- 0 .)
L ea r n i ng r a t e 0 .0 .
S c
h e du l e r p a ti e n ce 10 (-. 5 ) 6 (-. 1 ) 5 (-. 1 ) 7 (-. 1 ) 5 (-. 5 )
L p r e gu l a r i za ti on
N on e (-. 5 )
L 2 (- 1 . 1 )
L 3 (- 0 . 5 )
L 3 (- 0 . 8 )
L 3 (- 0 . 5 )
E n tit y e m b .
w e i gh t 2 .−2 .−1 .−1 .−4 .−
R e l a ti on e m b .
w e i gh t 4 .−2 .−4 .−3 .−3 .−
F r e qu e n c y w e i gh ti ng
Y e s (- 0 .)
Y e s (- 1 .)
N o (- 0 .)
N o (- 0 .)
N o (- 0 .)
E m b e dd
i ng no r m a li
za ti o n ( T r a n s E )
E n tit y –
N o –
[Table]
R e l a ti on –
N o –
[Table]
D r opou t
E n tit
y
e m b e dd i ng 0 .0 .0 .
[Table]
[Table]
R e l a ti on e m b e dd i ng 0 .0 .0 .0 .
[Table]
P r o j ec ti on ( C onv
E ) – – –
[Table]
F ea t u r e m a p ( C onv
E ) – – – – 0 .
E m b e dd
i
ng i n iti a li za ti on
U n i f .
X v U n i f ( - 4 . 4 )
U n i f .
X v U n i f ( - 0 . 1 )
N o r m a l (- 0 .)
S t d .
d e v i a ti on ( N o r m a l ) – – – – 0 .
I n t e r v a l (
U n i f)
[Table]
T a b l e 6 :
H yp e r p a r a m e t e r s o f b e s t p e rf o r m i ng m od e l s f ound w it
h qu a s ir a ndo
m hyp e r p a r a m e t e r op
ti m i za ti on o n
F B 15 K237 .
S e tti ng s t
h a t a pp l y on l y t
o ce r t a
i n c on ﬁ gu r a ti on s a r e i nd
i ca t e d i n p a r e n t
h
e s i s .
F o r e ac h
hyp e r p a r a m e t e r , w e a l s o g
i v e t h e r e du c ti on i n ﬁ lt e r e d
M
RR f o r t h e b e s t c on ﬁ gu r a ti on t h a t do e s no t
u s e t h
i s v a l u e o f t
h e hyp e r p a r a m e t e r ( i n p a r e n t h
e s i s ) .
[Table]
n s E ) – – – –
O p ti m i ze r
A d a g .
A d a m (-. 8 )
A d a m (-. 6 )
A d a m (-. 3 )
A d a m (-. 3 )
B a t
c h
s i ze (- 0 .)
L ea r n i ng r a t e 0 .0 .
S c
h e du l e r p a ti e n ce 4 (-. 8 ) 6 (-. 8 ) 1 (-. 1 ) 1 (-. 2 ) 6 (-. 3 )
L p r e gu l a r i za ti on
N on e (-. 8 )
L 2 (- 1 . 8 )
N on e (-. 1 )
N on e (-. 2 )
L 2 (- 1 . 3 )
E n tit y e m b .
w e i gh t 5 .−2 .−
[Table]
−
[Table]
−
[Table]
[Table]
R e l a ti on e m b .
w e i g h t 4 .−2 .−
[Table]
−
[Table]
−
[Table]
−
[Table]
F r e qu e n c y w e i gh ti n g
N o (- 1 .)
Y e s (- 1 .)
Y e s (- 0 .)
Y e s (- 1 .
[Table]
Y e s (
[Table]
[Table]
E m b e dd
i ng no r m a li z a ti on ( T r a n s E )
E n tit y –
N o
[Table]
R e l a ti on –
N o
[Table]
D r opou t
E n tit
y
e m b e dd i ng 0 .0 .0 .
[Table]
[Table]
R e l a ti on e m b e dd i n g 0 .0 .0 .0 .
[Table]
P r o j ec ti on ( C onv
E ) – – – – 0 .
F ea t u r e m a p ( C onv
E ) – – – – 0 .
E m b e dd
i
ng i n iti a li za t i on
X v U n i f ( - 0 . 8 )
X v U n i f ( - 2 . 0 )
N o r m a l (- 0 .)
N o r m a l (- 1 .)
X v N o r m (- 1 .)
S t d .
d e v i a ti on ( N o r m a l ) – – 0 .
[Table]
[Table]
I n t e r v a l (
U n i f) – –
[Table] T a b l e 7 : H yp e r p a r a m e t e r s o f b e s t p e rf o r m i ng m o d e l s f ound w it h qu a s ir a ndo m hyp e r p a r a m e t e r op ti m i za ti on on W N
RR .
S e tti ng s t
h a t a pp l y on l y t
o ce r t a
i n c on ﬁ gu r a ti on s a r e i nd
i ca t e d i n p a r e n t
h
e s i s .
F o r eac h
hy p e r p a r a m e t e r , w e a l s o g i v e t h e r e du c ti on i n ﬁ lt e r e d
M
RR f o r t h e b e s t c on ﬁ gu r a ti on t h a t do e s no t
u s e t h
i s v a l u e o f t
h e hyp e r p a r a m e t e r ( i n p a r e n t h
e s i s ) .
RESCAL
TransE
[Table]
ComplEx
ConvE
[Table]
MRR valid
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Projection
[Table]
[Table]
[Table]
best performing models found with Bayesian optimization.
All other hyperparameters are the same as in Tables 6 (FB15k-237) and 7 (WNRR).