Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations
Abstract
Learning good representations on multi-relational graphs is essential to knowledge base completion (KBC).
In this paper, we propose a new self-supervised training objective for multi-relational graph representation learning, via simply incorporating relation prediction into the commonly used 1vsAll objective.
The new training objective contains not only terms for predicting the subject and object of a given triple, but also a term for predicting the relation type.
We analyse how this new objective impacts multi-relational learning in KBC: experiments on a variety of datasets and models show that relation prediction can signiﬁcantly improve entity ranking, the most widely used evaluation task for KBC, yielding a 6.1% increase in MRR and 9.9% increase in Hits@ on FB15k-237 as well as a 3.1% increase in MRR and 3.4% in Hits@1 on Aristo-v4.
Moreover, we observe that the proposed objective is especially effective on highly multi-relational datasets, i.e. datasets with a large number of predicates, and generates better representations when larger embedding sizes are used.
1. Introduction
Aiming at completing missing entries, Knowledge Base Completion (KBC), also known as Link
Prediction, plays a crucial role in constructing large-scale knowledge graphs [Nickel et al., 2016, Ji et al., 2020, Li et al., 2020].
Over the past years, most of the research on KBC has been focusing on
Knowledge Graph Embedding models, which learn representations for all entities and relations in a Knowledge Graph, and use them for scoring whether an edge exists or not [Nickel et al., 2016].
Numerous models and architectural innovations have been proposed in the literature, including but not limited to translation-based models [Bordes et al., 2013], latent factorisation models [Nickel et al., 2011, Trouillon et al., 2016, Balazevic et al., 2019], and neural network-based models [Dettmers et al., 2018, Schlichtkrull et al., 2018, Xu et al., 2020].
Other more recent research has been making complementary efforts on analysing the evaluation procedures for these KBC models.
For instance, Sun et al.
[2020] call for standardisation of evaluation protocols; Kadlec et al.
[2017], Rufﬁnelli et al.
[2020] and Jain et al.
[2020] highlight the importance of training strategies and show that careful hyper-parameter tuning can produce more accurate results than adopting more elaborate model architectures; Lacroix et al.
[2018] suggests that a simple model can produce state-of-the-art results when its training objective is properly selected.
Taking inspiration from these ﬁndings, this paper explores relation prediction : a simple auxiliary training objective that signiﬁcantly improves multi-relational graph representation learning across several KBC models.
Aside from training models to predict the subject and object entities for triples in a Knowledge Graph, we also train them to predict relation types , leading to a self-supervised training objective.
Intuitively, this approach is akin to using a masked language model-like training objective [Devlin et al., 2019] instead of the commonly used auto-regressive training objective for KBC.
In our experiments, we ﬁnd that the new auxiliary training objective signiﬁcantly improves downstream link prediction accuracy.
Empirical evaluations on various models and datasets support the effectiveness of our new training objective: the largest improvements were observed on ComplEx-N3 [Trouillon et al., 2016] and CP-N3 [Lacroix et al., 2018] with embedding sizes between 2K and 4K, providing up to 9 .
9% boost in Hits@1 and 6 .
1% boost in MRR on FB15k-237 with negligible computational overhead.
We further experiment on datasets with varying numbers of predicates and ﬁnd that relation prediction helps more when the dataset is highly multi-relational, i.e. contains a larger number of predicates.
Moreover, our qualitative analysis demonstrates improved prediction of some M ANY - TOM ANY [Bordes et al., 2013] predicates and more diversiﬁed relation representations.
2. Background and Related Work
A Knowledge Graph G ⊆
E × R × E contains a set of subject-predicate-object (cid:104) s, p, o (cid:105) triples, where each triple represents a relationship of type p ∈ R between the subject s ∈ E and the object o ∈ E of the triple.
Here, E and R denote the set of all entities and relation types, respectively.
Knowledge Graph Embedding Models
A Knowledge Graph Embedding (KGE) model, also referred to as neural link predictor , is a differentiable model where entities in E and relation types in R are represented in a continuous embedding space, and the likelihood of a link between two entities is a function of their representations.
More formally, KGE models are deﬁned by a parametric scoring function φ θ : E × R × E (cid:55)→ R , with parameters θ that, given a triple (cid:104) s, p, o (cid:105) , produces the likelihood that entities s and o are related by the relationship p .
Scoring Functions
KGE models can be characterised by their scoring function φ θ .
For example, in
TransE [Bordes et al., 2013], the score of a triple (cid:104) s, p, o (cid:105) is given by φ θ ( s, p, o )
= − (cid:107) s +
p − o (cid:107) 2 , where s , p , o ∈ R
k denote the embedding representations of s , p , and o , respectively.
In DistMult [Yang et al., 2015], the scoring function is deﬁned as φ θ ( s, p, o )
= (cid:104) s , p , o (cid:105)
=
(cid:80) ki
=1 s i p
i
o i , where (cid:104) · , · , · (cid:105) denotes the tri-linear dot product.
Canonical Tensor Decomposition [CP, Hitchcock, 1927] is similar to DistMult, with the difference that each entity x has two representations, x s ∈ R k and x o ∈ R k , depending on whether it is being used as a subject or object: φ θ ( s, p, o )
= (cid:104) s s , p , o o (cid:105) .
In RESCAL [Nickel et al., 2011], the scoring function is a bilinear model given by φ θ ( s, p, o )
=
s (cid:62) Po , where s , o ∈ R k is the embedding representation of s and p , and
P ∈ R k × k is the representation of p .
Note that DistMult is equivalent to RESCAL if P is constrained to be diagonal.
Another variation of this model is ComplEx [Trouillon et al., 2016], where the embedding representations of s , p , and o are complex vectors – i.e. s , p , o ∈ C k – and the scoring function is given by φ θ ( s, p, o ) =
(cid:60) ( (cid:104) s , p , o (cid:105) ) , where (cid:60) ( x ) represents the real part of x , and x denotes the complex conjugate of x .
In TuckER [Balazevic et al., 2019], the scoring function is deﬁned as φ θ ( s, p, o )
=
W × 1 s × 2 p × 3 o , where W ∈ R k s × k p × k o is a three-way tensor of parameters, and s ∈ R k s , p ∈ R k p , and o ∈ R
k
o are the embedding representations of s , p , and o .
In this work, we mainly focus on DistMult, CP, ComplEx, and TuckER, due to their effectiveness on several link prediction benchmarks [Rufﬁnelli et al., 2020, Jain et al., 2020].
Training Objectives
Another dimension for characterising KGE models is their training objective .
Early tensor factorisation models such as RESCAL and CP were trained to minimise the reconstruction error of the whole adjacency tensor [Nickel et al., 2011].
To scale to larger Knowledge Graphs, subsequent approaches such as Bordes et al.
[2013] and Yang et al.
[2015] simpliﬁed the training objective by using negative sampling : for each training triple, a corruption process generates a batch of negative examples by corrupting the subject and object of the triple, and the model is trained by increasing the score of the training triple while decreasing the score of its corruptions.
This approach was later extended by Dettmers et al.
[2018] where, given a subject s and a predicate p , the task of predicting the correct objects is cast as a |E| -dimensional multi-label classiﬁcation task, where each label corresponds to a distinct object and multiple labels can be assigned to the ( s, p ) pair.
This approach is referred to as KvsAll by Rufﬁnelli et al.
Another extension was proposed by
Lacroix et al.
[2018] where, given a subject s and an object p , the task of predicting the correct object o in the training triple is cast as a |E| -dimensional multi-class classiﬁcation task, where each class corresponds to a distinct object and only one class can be assigned to the ( s, p ) pair.
This is referred to as 1vsAll by Rufﬁnelli et al.
Note that, for factorisation-based models like DistMult, ComplEx, and CP, KvsAll and 1vsAll objectives can be computed efﬁciently on GPUs [Lacroix et al., 2018, Jain et al., 2020].
For example for DistMult, the score of all triples with subject s and predicate p can be computed via E ( s (cid:12) p ) , where (cid:12) denotes the element-wise product, and E ∈ R
|E|× k is the entity embedding matrix.
In this work, we follow Lacroix et al.
[2018] and adopt the 1vsAll loss, so as to be able to compare with their results, and since Rufﬁnelli et al.
[2020] showed that they produce similar results in terms of downstream link prediction accuracy.
Recent work on standardised evaluation protocols for KBC models [Sun et al., 2020] and their systematic evaluation [Kadlec et al., 2017, Mohamed et al., 2019, Jain et al., 2020, Rufﬁnelli et al., 2020] shows that latent factorisation based models such as RESCAL, ComplEx, and CP are very competitive when their hyper-parameters are tuned properly [Kadlec et al., 2017, Rufﬁnelli et al., 2020].
In this work, we show that using relation prediction as an auxiliary training task can further improve their downstream link prediction accuracy.
3. Relation Prediction as An Auxiliary Training Objective
In what is referred to as the 1vsAll setting [Rufﬁnelli et al., 2020], KBC models are trained using a self-supervised training objective by maximising the conditional likelihood of the subject s (resp.
object o ) of training triples, given the predicate and the object o (resp.
subject s ).
More formally,
KBC models are trained by maximising the following objective:
[Equation]
with
[Equation]
where θ ∈ Θ are the model parameters, including entity and relation embeddings, and φ θ is a scoring function parameterised by θ .
Such an objective limits predicting positions in the training objective to either the ﬁrst ( s ) or the last ( o ) element of the triple.
In this work, we propose relation prediction as an auxiliary task for training KBC models.
The new training objective not only contains terms for predicting the subject and the object of the triple – log P ( s | p, o ) and log P ( o | s, p ) in Equation (1) – but also an objective log P ( p | s, o ) for predicting the relation type p :
[Equation]
with
[Equation]
where λ ∈ R + is a user-speciﬁed hyper-parameter that determines the contribution of the relation prediction objective; we assume λ = 1 unless speciﬁed otherwise.
This new training objective adds very little overhead to the training process, and can be easily added to existing KBC implementations;
PyTorch examples are included in Appendix A. Compared to conventional approaches, relation prediction can help the model learn to further distinguish among different predicates.
4. Empirical Study
In this section, we conduct several experiments to verify the effectiveness of incorporating relation prediction as an auxiliary training objective.
We are interested in the following research questions:
RQ1: How does the new training objective impact the results of downstream knowledge base completion tasks across different datasets?
How does the number of relation types on the datasets affect the new training objective?
RQ2: How does the new training objective impact different models?
Does it beneﬁt all the models uniformly, or it particularly helps some of them?
RQ3: Does the new training objective produce better entity and relation representations?
Datasets
We use Nations, UMLS, and Kinship from Kok and Domingos [2007], WN18RR
[Dettmers et al., 2018], and FB15k-237 [Toutanova et al., 2015], which are commonly used in the KBC literature.
As these datasets contain a relatively small number of predicates, we also experiment with
Aristo-v4, the 4 -th version of Aristo Tuple KB
[Mishra et al., 2017], which has more than 1 . 6 k predicates.
Since Aristo-v4 has no standardised splits for KBC, we randomly sample 20 k triples for test and 20 k for validation.
Table 1 summarises the statistics of these datasets. [Table]
Metrics
Entity ranking is the most commonly used evaluation protocol for knowledge base completion.
For a given query ( s, p, ?) or (? , p, o ) , all the candidate entities are ranked based on the scores produced by the models, and the resulting ordering is used to compute the rank of the true answer.
We use the standard ﬁltered Mean Reciprocal Rank (MRR) and Hits@
K (Hit ratios of the top-K ranked results), with K ∈ { 1 , 3 , 10 } , as metrics.
Models
We use several competitive and reproducible [Rufﬁnelli et al., 2020, Sun et al., 2020] models: RESCAL [Nickel et al., 2011], ComplEx [Trouillon et al., 2016], CP [Lacroix et al., 2018], and TuckER [Balazevic et al., 2019].
To ensure fairness in various comparisons, we did an extensive tuning of hyper-parameters using the validation sets, which consists of 41,316 training runs in total.
For the main results on all the datasets, we tuned λ using grid-search.
For the ablation experiments on the number of predicates and for different choices of models, we set λ to 1 for simplicity.
Details regarding the hyper-parameter sweeps can be found in Appendix B.
4.1 RQ1: Impacts of Relation Prediction on Different Datasets
How does the proposed training objective impact knowledge base completion on different datasets?
To answer this research question, we compare the performance of training with relation prediction and training without relation prediction on several popular KBC datasets.
For the smaller datasets (Kinship, Nations and UMLS), we selected the best one from RESCAL, ComplEx, CP, and TuckER.
For larger datasets (WN18RR, FB15k-237, and Aristo-v4), due to a limited computation budget, we used ComplEx, which outperformed other models in our preliminary experiments.
Table 2 summarises the results on the smaller datasets, where (cid:34) indicates training with relation prediction while (cid:37) indicates training without relation prediction.
We can observe that relation prediction brings a 2% – 4% improvement in MRR and Hits@1, as well as keeping a competitive
Hits@3 and Hits@10.
Table 3 summarises the results on the larger datasets.
Including relation prediction as an auxiliary training objective brings a consistent improvement on the 3 datasets with respect to all metrics, except for Hits@10 on WN18RR.
Particularly, relation prediction leads to increases of 6 . 1% in MRR, 9 . 9% in Hits@1, 6 . 1% in Hits@3 on FB15k-237 and 3 . 1% in MRR, 3 . 4% in Hits@1, 3 . 8% in Hits@3 on Aristo-v4. Compared to WN18RR, we observe a larger improvement on FB15kand Aristo-v4.
One potential reason is that on FB15k-237 there is a more diverse set of predicates ( |R| = 237 ) and Aristo-v4 ( |R| = 1605 ) than in WN18RR ( |R| = 11 ).
The number of predicates |R| on WN18RR is comparatively small, and the model gains more from distinguishing different entities than distinguishing relations. In other words, using lower values for λ (the weight of the
[Table]
[Table]
[Table]
Including relation prediction as an auxiliary training objective brings consistent improvements across the three datasets on all metrics except Hits@10 on WN18RR.
On FB15kand Aristo-v4, adding relation prediction yields larger improvements in downstream link prediction tasks.
More details on the hyper-parameter selection process are available in
Appendix B.1.2.
relation prediction objective) is more suitable for datasets with fewer predicates but a large number of entities.
We include ablations on |R| in Section 4.1.2.
[Table]
4.1.1 S IGNIFICANCE T ESTING
In order to show that the improvements brought by relation perturbation are signiﬁcant, we run the experiments with 5 random seeds and perform Wilcoxon signed-rank test [Wilcoxon, 1992] over the metrics obtained with and without relation prediction.
The test is performed as follows.
First, we computed the differences between results obtained with ComplEx trained with and without relation prediction.
The null hypothesis is that the median of the differences is negative.
Table 4 summarises the result.
We can observe that almost all p-values are about 0.03, which means we can reject the null hypothesis at a conﬁdence level of about 97%.
The new training objective that incorporates relation prediction as an auxiliary training objective signiﬁcantly improves the performance of KBC models except for Hits@10 on WN18RR.
4.1.2 A BLATION ON THE N UMBER OF P REDICATES
As previously discussed, relation prediction brings different impacts to WN18RR, FB15k-237, and
Aristo-v4.
Since one of the biggest differences among these datasets is the number of different predicates |R| ( 1 , 605 for Aristo-v4 and 237 for FB15k-237, while only 11 for WN18RR), we would like to determine the impact of perturbing relations with various |R| .
In order to achieve this, we construct a series of datasets with different |R| by sampling triples containing a subset of predicates from FB15k-237.
For example, to construct a dataset with only 5 predicates, we ﬁrst sampledpredicates from the set of 237 predicates and then extracted triples containing these 5 predicates as the new dataset.
In total, we have datasets with |R| ∈ [5 , 25 , 50 , 100 , 150 , 200] predicates.
To address the noise introduced in predicate sampling during datasets construction, we experimented with 3 random seeds.
For convenience, we set the weight of relation prediction λ to 1 and performed a similar grid-search over the regularisation and other hyper-parameters to ensure that the models were regularised and trained appropriately with the different amounts of training and test data points.
Results are summarised in Figure 1.
As shown in the right portion of Figure 1, predicting relations helps datasets with more predicates, resulting in a 2% – 4% boost in MRR, Hits@1, and
Hits@3.
For datasets with fewer than 50 predicates, there is considerable ﬂuctuation in the relative change as shown in the left portion of the ﬁgure – but a clear downward trend.
These results verify our hypothesis that relation prediction brings beneﬁts to datasets with a larger number of predicates.
Note that we did not tune the weight of relation prediction objective λ (and ﬁxed it to 1), and this choice might have been sub-optimal on datasets with a fewer number of predicates. |R| in the Dataset - - - - - R e l a t i v e C h a n g e i n P
e r ce n t ag e %
MRR Hits@ Hits@ Hits@
Model
Relation Prediction
MRR
Hits@
Hits@
Hits@
CP (cid:37) 0.
(cid:34) 0.
ComplEx (cid:37) 0.
(cid:34) 0.
RESCAL (cid:37) 0.
(cid:34) 0.
TuckER (cid:37) 0.
(cid:34) 0.
4 models, it brings more beneﬁt to CP and ComplEx compared to TuckER and RESCAL
4.2 RQ2: Impacts of Relation Prediction on Different KBC Models
For measuring how does relation prediction inﬂuences the downstream accuracy of KBC models, we run experiments on FB15k-237 with several models – namely ComplEx, CP, TuckER, and RESCAL. [Figure]
For simplicity, we set the weight of relation prediction λ to 1.
As shown in Table 5, including relation prediction as an auxiliary training objective brings consistent improvement to all models.
Notably, up to a 4.4% and a 6.6% increase in Hits@1 can be observed respectively for CP and ComplEx.
For
TuckER and RESCAL, the improvements brought by relation perturbation are relatively small.
This may be due to the fact that we had to use smaller embedding sizes for TuckER and RESCAL, since these models are known to suffer from scalability problems when used with larger embedding sizes.
We include the ablation on embedding sizes of the models in Section 4.2.1.
As for the computational cost, in our experiments, adopting the new loss only added an average 2% increase in training time per epoch, though it might require more epochs to converge.
4.2.1 A BLATIONS ON E MBEDDING S IZES
In our experiments, increasing the embedding size of the model leads to better performance.
However, there might exist a saturating point where larger embedding sizes stop boosting the performance.
We are interested in how perturbing relations will impact the saturating point and which embedding sizes beneﬁt most from it.
Figure 2 shows the relationship between the embedding size and the MRR for
CP on FB15k-237.
At small embedding sizes, perturbing relations makes little difference.
However, it does help CP with larger embedding sizes and delays the saturating point.
As we can see, the slope of the blue curve is larger than the red one, which bends little between an embedding size of1, and an embedding size of 4,000.
We can thus observe that perturbing relations leaves more headroom to improve the model by increasing embedding sizes.
4.3 RQ3: Qualitative Analysis of the Learned Entity and Relation Representations
In our experiments, we observe that relation prediction improves the link prediction accuracy for
MANY - TO - MANY predicates, which are known to be difﬁcult for KBC models [Bordes et al., 2013].
Table 6 lists the top 10 predicates that beneﬁt most from relation prediction. We rank the predicates
/ice hockey/hockey team/ team roster/position /sports/sports team/roster./baseball/baseball roster position/position /location/country/second level divisions /tv/tv producer/programs produced./tv/tv producer term/program /olympics/olympic sport/athletes./olympics/olympic athlete afﬁliation/olympics /award/award winning work/awards won./award/award honor/honored for /music/instrument/family /olympics/olympic games/sports /base/biblioness/bibs location/state /soccer/football team/current roster./soccer/football roster position/position by averaging the associated MRR of ( s, p, ?) and (? , p, o ) queries.
Table 14 and Table 15 list the top 20 queries of ( s, p, ? and (? , p, o ) that are improved most by relation prediction. We can see that relation prediction helps the queries like “Where was ﬁlm Magic Mike released?” , “Where was Paramount Pictures founded?” , “Which person appear in the ﬁlm The Dictator 2012?” , “Which places are located in UK?”
and “Which award did Vera Drake win?” .
To intuitively understand why it helps with these predicates, we ran t-SNE over the learned entity and predicate representations.
Reciprocal predicates are also included in the t-SNE visualisations.
We set the embedding size to 1000, and use N3 regularisation.
Hyper-parameters were chosen based on the validation MRR.
We run t-SNE for 5000 steps with 50 as perplexity.
As we can see from
Figure 3, there are more predicate clusters in the t-SNE visualisation for relation prediction compared to without relation prediction.
This demonstrates relation prediction helps the model distinguish between different predicates: Most predicates are separated from the entities (the pink region) while some predicates with similar semantics or subject-object contexts form a cluster (the red region);
There are also a few predicates, which are not close to their predicate counterparts but instead close to highly related entities (the green region).
Table 7 lists 3 example predicates for each region.
Though there can be information loss during the process of projecting high-dimensional embedding vectors into 2-dimensional space, we hope this visualisation will help illustrate how relation prediction helps to learn more diversiﬁed predicate representations.
[Table]
5. Discussion and Conclusion
In this paper, we propose to use a new self-supervised objective for training KBC models - by simply incorporating relation prediction into the commonly used 1vsAll objective.
In our experiments, we show that adding such a simple learning objective is signiﬁcantly helpful to various KBC models.
It brings up to 9 .
9% boost in Hits@1 for ComplEx trained on FB15k-237, even though the evaluation task of entity ranking might seem irrelevant to relation prediction .
Our work suggests a worthwhile direction towards devising relation-aware self-supervised objectives for KBC.
In this paper, we mainly focus on simple factorisation-based models. Future work will consider analysing the proposed objective for more complex KBC models, such as graph neural network-based KBC models, and on more datasets.
Another interesting future work direction is analysing the proposed auxiliary objective on more downstream applications besides link prediction, and evaluate whether it can be used to learn useful multi-relational graph representations. Acknowledgements
We would like to thank the reviewers for their constructive feedback.
We would also want to thank all the other members in UCL NLP for their support throughout COVID.
PM and PS are supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 875160.
Appendix A. Code Snippet for Relation Prediction as an Auxiliary Training Objective
Figure 4 demonstrates how to add relation prediction to the existing implementation of ComplEx.
Appendix B. Hyper-parameters Sweeps
In this section
[Table]
we summarise all the hyper-parameters used in our experiments.
We used Tesla
P100 and Tesla V100 GPUs to run the experiments.
We implemented each model by PyTorch.
Our codebase is based on https://github.com/facebookresearch/kbc .
B.1 Hyper-parameter Ranges of Relation Prediction Across Datasets B.1.1 K INSHIP , N ATIONS , AND UMLS
[Table] [Table]
lr is the learning rate.
bsz is the batch size.
reg is the regularization strength.
λ is the weighting of relation prediction.
NA indicates not applicable.
For all small datasets (Kinship, Nations, UMLS), we trained RESCAL, ComplEx, CP and
TuckER with Adagrad optimiser and N3 regularisation for at most 400 epochs.
Reciprocal triples were included since they are reported to be helpful
[Dettmers et al., 2018, Lacroix et al., 2018].
We did grid searches over hyper-parameter combinations and chose the best conﬁguration for each dataset based on validation MRR.
We listed the grids of hyper-parameter search in Table 8 and report the best-searched conﬁguration in Table 9.
As for the balancing between relation prediction and entity prediction, we searched the weight of relation prediction over { 4 , 2 , 0 . 5 , 0 . 25 , 0 . 125 } .
B.1.2 WN18RR, FB15 K -237, AND A RISTO - V
For all datasets, we trained ComplEx with N3 regularizer and Adagrad optimiser and N3 regularisation for at most 400 epochs.
Reciprocal triples were included since they are reported to be helpful
[Dettmers et al., 2018, Lacroix et al., 2018].
As for the weight of relation prediction, we searched over different zones on different datasets.
For WN18RR, we searched the weight of relation prediction over [0 . 005 , 0 .
For FB15k-237, we searched over [0 . 125 , 0 .
For Aristo-v4, we searched over [0 . 125 , 0 . 25 , 0 .
We did grid searches over hyper-parameter combinations and chose the best conﬁguration for each dataset based on validation MRR.
We report the grids for each dataset in Table 10, and the best found conﬁguration in Table 11.
[Table] [Table]
lr is the learning rate. bsz is the batch size. reg is the regularization strength. λ is the weighting
B.2 Hyper-parameter Ranges of Relation Prediction Across Models
We experiment with each model on FB15k-237.
Note that the original TucKER [Balazevic et al., 2019] includes some training strategies which are not used in CP, ComplEx and TuckER, like dropout, learning rate decay etc.
However, for a fair comparison of how relation prediction affects each model, we trained all the models conditioned on similar settings with Adagrad optimizer and N3 regularisation for at most 400 epochs.
We did grid searches and selected the best hyperparameter conﬁgurations according to validation MRR.
We set the weight of relation prediction to 1 in this experiment .
Table 12 lists the grid of the shared hyper-parameters.
For RESCAL, the regularisation over predicate matrices can be normalised over the rank to achieve better results.
Also
F2 regularisation empirically performed better than N3 regulariser for RESCAL.
For TuckER, the ranks for predicate and entity are different.
Table 13 lists the best hyper-parameter conﬁguration found by our search
[Table]
[Table]
[Table]
[Figure]
Appendix C. Additional Results C.1 More Metrics for Ablation on Rank
Figure 5 (MRR), Figure 6 (Hits@3) and Figure 7 (Hits@10) show the additional metric for the experiments ablating ranks.
Blue indicates training with relation prediction while red indicates training without prediction.
The range of the rank is [25 , 50 , 100 , 500 , 1000 , 2000 , 3000 , 4000]
C.2 Top 20 Queries That Are Improved Most by Relation Prediction
Table 14 shows the top 20 queries of (? , p, o ) form that are improved most by relation prediction while
Table 15 shows the top 20 queries of ( s, p, ?) form that are improved most by relation prediction.
[Figure] [Figure]
Subject
Predicate
Object ∆ 1 / Rank
Paramount Pictures /common
/topic/webpage /common/webpage/category
NA 1.
Midﬁelder /sports/sports position/players /sports/sports team roster/team
Gaziantepspor 0.
The Dictator (2012 ﬁlm)
/ﬁlm/ﬁlm/personal appearances /ﬁlm/personal ﬁlm appearance/person Hillary Clinton 0.
Academy Award for Best
Supporting Actress /award/award category/winners /award/award honor/award winner
Maureen Stapleton 0.
Christopher Columbus /user/tsegaran/random/taxonomy subject/entry /user/tsegaran/random/taxonomy entry/taxonomy
Library of Congress Classiﬁcation 0.
United Kingdom /location/location/contains
Westminster 0.
President /organization/role/leaders /organization/leadership/organization
West Virginia University 0.
Academy Award for Best
Supporting Actor /award/award category/winners /award/award honor/award winner
Christopher Walken 0.
President /organization/role/leaders /organization/leadership/organization
Bryn Mawr College 0.
President /organization/role/leaders /organization/leadership/organization
Dickinson College 0.
National Society of Film
Critics Award for Best Actress /award/award category/winners /award/award honor/award winner
Reese Witherspoon 0.
President /organization/role/leaders /organization/leadership/organization
Louisiana State University 0.
Vera Drake /award/award winning work/awards won /award/award honor/award
Los Angeles Film Critics Association Award for Best Actress 0.
President /organization/role/leaders /organization/leadership/organization
University of Oklahoma 0.
United States /location/country/second level divisions
Marion County, Indiana 0.
President /organization/role/leaders /organization/leadership/organization
University of Southern California 0.
Travis Tritt /ﬁlm/actor/ﬁlm /ﬁlm/performance/ﬁlm
Blues Brothers0.
London Film Critics’ Circle
Award for Director of the
Year /award/award category/winners /award/award honor/award winner
Neil Jordan 0.
United States /location/country/second level divisions
Niagara County, New York 0.
Deion Sanders /people/person/places lived /people/place lived/location
Atlanta 0.
[Table]
[Table]
EL
A T I
ON
P
[Table]
E
[Table]
I C
T I ON
A S
AN
A UX
I L I A
R Y
[Table]
I
NG
[Table]
BC
[Table]
[Table]