Pretrain-KGEs: Learning Knowledge Representation from Pretrained Models for Knowledge Graph Embeddings
Abstract
Learning knowledge graph embeddings (KGEs) is an efﬁcient approach to knowledge graph completion.
Conventional KGEs often suffer from limited knowledge representation, which causes less accuracy especially when training on sparse knowledge graphs.
To remedy this, we present Pretrain-KGEs , a training framework for learning better knowledgeable entity and relation embeddings, leveraging the abundant linguistic knowledge from pretrained language models.
Speciﬁcally, we propose a uniﬁed approach in which we ﬁrst learn entity and relation representations via pretrained language models and use the representations to initialize entity and relation embeddings for training KGE models.
Our proposed method is model agnostic in the sense that it can be applied to any variant of
KGE models.
Experimental results show that our method can consistently improve results and achieve state-of-the-art performance using different KGE models such as TransE and
QuatE, across four benchmark KG datasets in link prediction and triplet classiﬁcation tasks.
Introduction
Knowledge graphs (KGs) constitute an effective access to world knowledge for a wide variety of
NLP tasks, such as question-answering, entity linking and information retrieval.
A typical KG such as Freebase (Bollacker et al., 2008) and WordNet (Miller, 1995) consists of a set of triplets in the form of ( h, r, t ) with the head entity h and the tail entity t as nodes and relations r as edges in the graph.
A triplet represents the relation between two entities, e.g., (Steve Jobs, founded, Apple Inc.) .
Despite their effectiveness, KGs in real applications suffer from incompleteness and there have been several attempts for knowledge graph completion among which knowledge graph embedding is one of prominent approaches.
Knowledge graph embedding (KGE) models have been designed extensively in recent years (Bordes et al., 2013; Ji et al., 2015; Lin et al., 2015;
Sun et al., 2019; Ebisu and Ichise, 2018; Nickel et al., 2011; Yang et al., 2015; Kazemi and Poole, 2018; Trouillon et al., 2016; Zhang et al., 2019).
The general methodology of these models is to model entities and relations in vector spaces based on a score function for triplets ( h, r, t ) .
The score function measures the plausibility of each candidate triplet ( h, r, t ) compared to corrupted false triplets ( h (cid:48) , r, t ) or ( h, r, t (cid:48) ) .
However, traditional
KGE models often suffer from limited knowledge representation due to the simply symbolic representation of entities and relations.
Some recent works take advantages of both fact triplets and textual description to enrich knowledge representation (Socher et al., 2013a; Xu et al., 2017; Xiao et al., 2017; Xie et al., 2016;
An et al., 2018), but without exploitation of contextual information of the textual descriptions.
Moreover, much of this research effort has been dedicated to developing novel architectures for knowledge representation without applications to KGE models.
Unlike many existing works which try to propose new architectures for KGEs or knowledge representation, we focus on model-agnostic pretraining technique for KGE models.
We present a uniﬁed training framework named as PretrainKGEs which consists of three phases: ﬁne-tuning phase, initializing phase and training phase (see
Fig. 1).
During the ﬁne-tuning phase, we learn better knowledgeable entity and relation representations via pretrained language models using textual descriptions as input sequence.
Different from previous works incorporating textual information into knowledge representation, we use pretrained langauge models such as BERT (Devlin et al., 2019) to better understand textual description by making full use of syntactic and semantic information in large- head : dog.n.
Semantic Descriptions dog : a member of the genus Canis (probably descended from the common wolf) that has been domesticated by man since prehistoric times; occurs in many breeds.
Canine : one of the four pointed conical teeth (two in each jaw) located between the incisors and the premolars.
Knowledge Extracting Semantic-based Fine-tuning tail : canine.n. KGE Training hypernym relation : _hypernym
BERT Encoder relation head tail
KGE loss
Entity Embedding …
Relation Embedding
Fine-tuned BERT Encoder … Entity Embedding
RelationEmbedding
KGE Model scale corpora on which BERT is pretrained.
Thus, we enable to incorporate rich linguistic knowledge learned by BERT into entity and relation representations.
Then during the initializing phase, we use knowledgeable entity and relation representations to initialize entity and relation embeddings so that the initialized KGEs inherit the rich knowledge.
Finally, during the training phase, we train a KGE model the same way as a traditional KGE model to learn entity and relation embeddings.
Extensive experiments using six public KGE models across four benchmark KG datasets show that our proposed training framework can consistently improve results and achieve state-of-the-art performance in link prediction and triplet classiﬁcation tasks.
Our contributions are as follows:
• We propose a model-agnostic training framework for learning knowledge graph embeddings by ﬁrst learning knowledge representation via pretrained language models.
• Results on several benchmark datasets show that our method can improve results and achieve state-of-the-art performance over variants of knowledge graph embedding models in link prediction and triplet classiﬁcation tasks.
• Further analysis demonstrates the effects of knowledge incorporation in our method and shows that our Pretrain-KGEs outperforms baselines especially in the case of fewer training triplets, low-frequency and the out-ofknowledge-base (OOKB)
entities.
Background and Related Work 2. Knowledge Graph Embedding
For each head entity h and tail entity t with their corresponding entity embeddings E h , E t , and each relation r with its relation embeddings R r , we formulate KGE models as follows:
[Equation]
where v h , v r , v t ∈ F d are the learnt vectors for each head entity, relation, and tail entity respectively,
The model is then optimized to calculate a higher score for true triplets than corrupted false ones.
According to the score function, KGE models can be roughly divided into translational models and semantic matching models (Wang et al., 2017).
Translational models popularized by TransE (Bordes et al., 2013) learn vector embeddings of the entities and the relations, and consider the relation between the head and tail entity as a translation between the two entity embeddings, i.e., in the form of v h + v r ≈ v t when the candidate triplet ( h, r, t ) holds.
Since TransE has problems when dealing with 1-to-N, N-to-1 and N-to-N relations, different translational models are proposed subsequently to deﬁne various relational patterns, such as TransH (Wang et al., 2014), TransR (Lin et al., 2015), TransD (Ji et al., 2015), RotatE (Sun et al., 2019), and TorusE (Ebisu and Ichise, 2018).
On the other hand, semantic matching models deﬁne a score function to match latent semantics of the head, tail entity and the relation.
For instance,
RESCAL (Nickel et al., 2011), DistMult (Yang et al., 2015), SimplE (Kazemi and Poole, 2018), and ComplEx (Trouillon et al., 2016) adopt a bilinear approach to model entities and relations for
KGEs.
Speciﬁcally, ComplEx learns complexvalued representations of entities and relations in complex space, while DistMult, SimplE, and
RESCAL embed entities and relations in the traditional real number ﬁeld.
The recent state-of-the-art,
QuatE (Zhang et al., 2019) represents entities as hypercomplex-valued embeddings and models relations as rotations in the quaternion space.
Both translational models and semantic matching models learn entity and relation embeddings in spite of different embedding spaces.
However, these KGE models only use structural information observed in triplets without incorporating external knowledge resources into KGEs, such as textual description of entities and relations.
Thus, the embeddings of entities and relations suffer from limited knowledge representation.
We instead propose a uniﬁed approach to introduce rich linguistic knowledge into KGEs via pretrained language models.
2. Text mining for Knowledge Representation
In a knowledge graph dataset, names of each entity and relation are provided as textual description of entities and relations.
Socher et al.
(2013a)
ﬁrst utilize textual information to represent entities by averaging word embeddings of entity names.
Following the word averaging method, Li et al.
(2016) improve the coverage of commonsense resources in ConceptNet (Speer and Havasi, 2012) by mining candidate triplets from Wikipedia.
They leverage a word averaging model to convert entity and relation names into name vectors.
Other recent works also leverage textual description to enrich knowledge representation but ignore contextual information of the textual descriptions (Socher et al., 2013a;
Xu et al., 2017; Xiao et al., 2017; Xie et al., 2016;
An et al., 2018).
Instead, our method exploits rich contextual information via pretrained models.
2. Deep Contextualized Word Embeddings
Recent approaches to modeling language representations offer signiﬁcant improvements over embeddings, especially pretrained deep contextualized lanaguge representation models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), GPT2 (Radford et al., 2019), and T5 (Raffel et al., 2019).
These deep language models learn better contextualized word presentations, since they are pretrained on large-scale free text data, which make full use of syntactic and semantic information in the large corpora.
In this work, we use BERT, a bidirectional
Transformer encoder to learn entity and relation representation given textual description.
Therefore, by incorporating the plentiful linguistic knowledge learned by pretrained language models, our proposed method can learn better knowledgeable entity and relation representations for subsequent
KGE learning.
Method
In this section, we will introduce our uniﬁed training framework Pretrain-KGEs and provide details of learning knowledgeable entity and relation representations via BERT.
3. Training Framework
An overview of Pretrain-KGEs is shown in Fig.
The framework consists of three phases: ﬁnetuning phase, initializing phase, and training phase.
Our major contribution is the ﬁne-tuning phase with the initializing phase, which incorporates rich knowledge into KGEs via pretained language models, i.e., BERT that enables to exploit contextual information of textual description for entities and relations.
By initializing embeddings with knowledgeable entity and relation representations, our training framework improves KGE models to learn better entity and relation embeddings.
Fine-tuning Phase
Given textual description of entities and relations such as entity names and relation names, we ﬁrst encode the textual descriptions into vectors via pretrained language models to represent entities and relations respectively.
We then project the entity and relation representations into two separate vector spaces to get the entity encoder
Enc e ( · ) for each entity e and the relation encoder
Enc r ( · ) for each relation r .
Formally, Enc e ( · ) and
Enc r ( · ) output entity and relation representations as:
[Equation]
where v h , v r , and v t represents encoding vectors of the head entity, the relation, and the tail entity in a triplet ( h, r, t ) respectively.
For details of Enc e ( · ) and Enc r ( · ) , see section 3.2.
Given the entity and relation representations, we then calculate the score of a triplet to measure its plausibility in Eq. 2.
For instance, if TransE is adopted, the score function is (cid:107) v h + v r − v t (cid:107) .
After ﬁne-tuning, the knowledge representation is used in the following initializing phase.
Initializing Phase
Given the knowledgeable entity and relation representation, we initialize entity embeddings E and relation embeddings R for a
KGE model instead of random initialization.
Speciﬁcally, E =
[ E 1 ; E 2 ; · · · ; E k ] ∈ F k × d and
R =
[ R 1 ; R 2 ; · · · ; R l ] ∈ F l × d in which “;” denotes concatenating column vectors into a matrix.
k and l denote the total number of entities and
[Table]
0.951 ♠ 0.
2290 Pretrain-QuatE 0.899 ♠ 0.
17.2 ♠ 0.554 ♠ 0.350 ♠ 84.4 ♠ 0.964 ♠ 0.
72 ♠ 0.586 ♠ 0.488 ♠ relations respectively.
F satisﬁes R ⊆ F and d denotes the embedding dimension.
Then E i ∈ F d represents the embedding of entity with index i and
R j ∈ F d represents the embedding of relation with index j .
During the initializing phase, we use the representation vector of entity with index i encoded by the entity encoder Enc e ( · ) as the initialized embedding E i for training KGE models to learn entity embeddings.
Likewise, the representation vector of relation with index j encoded by the relation encoder Enc r ( · ) is considered as the initialized embedding R j for training KGE models to learn relation embeddings.
Training Phase
After initializing entity and relation embeddings with knowledgeable entity and relation representations, we train a KGE model in the same way as a traditional KGE model.
We calculate the score of each training triplet in Eq.and Eq. 2 with the same score function in the ﬁnetuning phase.
Finally, we optimize the entity embedding E and the relation embedding R using the same loss function of the corresponding KGE model.
For example, if TransE and the max-margin loss function with negative sampling are adopted, the loss in the training phase is calculated as:
[Equation]
f ( v h , v r , v t ) − f ( v h (cid:48) , v r (cid:48) , v t (cid:48) ) (cid:3) + (4) where ( h, r, t ) and ( h (cid:48) , r (cid:48) , t (cid:48) ) represent a candidate and a corrupted false triplet respectively, γ denotes the margin, (cid:2) · (cid:3) + = max( · , 0) , and f ( · ) denotes score function of TransE (Bordes et al., 2013).
3. Learning knowledgeable Entity and Relation Representations via BERT
To learn better knowledge representation of entities and relations given textual description, we ﬁrst encode the textual description through Bert (Devlin et al., 2019), a bidirectional Transformer encoder which is pretrained on large-scale corpora and thus learns rich contextual information of texts by making full use of syntactic and semantic information in the large corpora.
We deﬁne T ( e ) and T ( r ) as the textual description of entities and relations respectively.
The textual description can be words, phrases, or sentences providing information about entities and relations such as names of entities and relations or deﬁnitions of word senses.
For example, the deﬁnition of entity e = Nyala.n.1 in WordNet is city in Sudan .
Then T ( Nyala.n.1 ) =
Nyala : city in Sudan .
Given the textual descriptions of entities and relations T ( e ) and T ( r ) , Bert ( · ) converts T ( e ) and
T ( r ) into entity representation and relation representation respectively in a vector space R n ( n denotes the vector size).
We then project the entity and relation representations into two separate vector spaces F d through linear transformations.
Formally, we get the entity encoder Enc e ( · ) for each entity e and the relation encoder Enc r ( · ) for each relation r as: [Equation] Enc
[Equation]
(6) where W e , W r ∈ F d × n , b e , b r ∈ F d , and σ : F d →
F d denotes a nonlinear function 1 .
The entity and relation representation encoded by Enc e ( · ) and Enc r ( · ) are then used to initialize entity and relation embeddings for a KGE model.
Experiments 4. Datasets and Evaluation Metrics
We evaluate our proposed training framework on four benchmark KG datasets: WN18 (Bor-
[Table]
des et al., 2013), WN18RR (Dettmers et al., 2018),
FB15K (Bordes et al., 2013) and FB15K237 (Toutanova and Chen, 2015).
2 WN18 and
WN18RR are two subsets of WordNet; FB15K and
FB15K-237 are two subsets of FreeBase.
WordNet is a large KG where entities are synsets corresponding to word senses and relations represents lexical relations between entities.
Freebase is a large KG of general world facts.
We use entity names and relation names provided by the four datasets as input textual descriptions for BERT, and we also utilize synsets deﬁnitions provided by WordNet as additional textual descriptions of entities.
In our experiments, we perform link prediction task (ﬁltered setting) mainly with triplet classiﬁcation task.
The link prediction task aims to predict either the head entity h given the relation r and the tail entity t or the tail entity given the head entity and the relation, while triplet classiﬁcation aims to judge whether a candidate triplet is correct or not.
For the link prediction task, we generate corrupted false triplets ( h (cid:48) , r, t ) and ( h, r, t (cid:48) ) using negative sampling.
For n test triplets, we get their ranks r = ( r 1 , r 2 , · · · , r n ) and calculate standard evaluation metrics: Mean Rank (MR), Mean Reciprocal Rank (MRR) and Hits at N (H@N).
For triplet classiﬁcation, we follow the evaluation protocol in Socher et al.
(2013b)
and adopt the accuracy metric (Acc) to evaluate our training method.
4. Baselines
To evaluate the universality of our training framework Pretrain-KGEs [Table] we select multiple public
KGE models as baselines including translational models :
• TransE (Bordes et al., 2013), the translationalbased model which models the relation as translations
between entities
[Table]
as • RotatE (Sun et al., 2019), the extension of translational-based models which introduces complex-valued embeddings to model the relations as rotations in complex vector space; • pRotatE (Sun et al., 2019), a variant of RotatE where the modulus of complex entity embeddings are constrained and only phase information is involved;
and semantic matching models
• DistMult (Yang et al., 2015), a semantic matching model where each relation is represented with a diagonal matrix; • ComplEx (Trouillon et al., 2016), the extension of semantic matching model which embedds entities and relations in complex space.
QuatE (Zhang et al., 2019), the recent stateof-the-art KGE model which learns entity and relation embeddings in the quaternion space.
4. Main Results
We present results for the Pretrain-KGEs algorithm in Table 1, Table 2 and Table 3.
Tableshows the link prediction results on four benchmark KG datasets using six public KGE models.
Table 2 compares the results on WordNet of using entity names and relation names to the results of adding deﬁnitions of word senses as additional textual description for entities.
Table 3 demonstrates the state-of-the-art performance of our proposed method in both link prediction and triplet classiﬁcation tasks 3 .
From the results, we can observe that:
(1) Our uniﬁed training framework can be applied to multiple variants of KGE models in spite of different embedding spaces, and achieves improvements over TransE, DistMult, ComplEx, RotatE, pRotatE and QuatE on most evaluation metrics, especially on MR but still being competitive on
MRR (see detailed analysis of MR and MRR in section 5.2.1).
Yet, it veriﬁes the universality of our training framework.
The reason is that our method incorporates rich linguistic knowledge into entity and relation representation via pretrained language models to learn better knowledgeable representation for the embedding initialization in KGE models.
For the effects of knowledge incorporation, see detailed analysis in section 5.2.
(2) Our training framework can also facilitate in improving the recent state-of-the-art even further over QuatE on most evaluation metrics in link prediction and triplet classiﬁcation tasks.
It veriﬁes the effectiveness of our proposed training framework.
Analysis
In this section, we provide further analysis of
Pretrain-KGEs’ performance in the case of fewer training triplets, low-frequency entities and the out-of-knowledge-base (OOKB) entities which are particularly hard to handle due to lack of knowledge representation.
We also evaluate the effects of knowledge incorporation into entity and relation embeddings by demonstrating the sensitivity of
MR and MRR metrics and visualizing the process of knowledge incorporation.
5. Performance on Fewer Training Triplets, Low-frequency and OOKB Entities
We also evaluate our training framework in the case of fewer training triplets on WordNet and test its performance on entities of varying frequency in test triplets on FB15K as well as the performance on the OOKB entities in test triplets on WordNet as shown in Fig.
2a- 2e.
To test the performance of our training framework given fewer training triplets, we conduct experiments on WN18 and WN18RR by feeding varying number of training triplets to a KGE model.
We use traditional TransE as one of the baselines.
Baseline-TransE does not utilize any textual description and randomly initializes entity and relation embeddings before the training phase.
Thus, it suffers from learn knowledgeable KGEs when training triplets become fewer.
In contrast, our
Pretrain-TransE ﬁrst learns knowledgeable entity and relation representations by encoding textual description through BERT, and uses the learned representations to initialize KGEs for TransE.
In this way, we enable to incorporate rich linguistic knowledge from BERT into initizalized entity and relation embeddings so that TransE can perform better given fewer training triplets.
On the other hand, to verify the effectiveness of
BERT during the ﬁne-tuning phase, we also set the word averaging model following Li et al.
(2016) to be the entity encoder Enc e ( · ) in Eq.
3 for comparison 4 .
From the results, we can observe that although the word averaging model contributes to better performance of TransE on fewer training triplets compared to Baseline-TransE, it does not learn knowledgeable entity and relation representations as well as BERT because BERT can better understand textual descriptions of entities and relations by exploiting rich contextual information of the textual descriptions.
Moreover, by utilizing deﬁnitions of word senses as additional textual description of entities, the results show that our training method achieves the best performance in the case of fewer training triplets.
Besides, we also evaluate our training framework for its performance on entities of varying frequency in training triplets on FB15K. From the results in
Fig.
2c, we can observe that our training framework outperforms Baseline-TransE especially on infrequent entities.
The reason is that traditional TransE method cannot learn good representation of infrequent entities due to inadquate dataset information and lack of textual description of entities.
When training triplets becomes fewer, there can be increasing OOKB entities in test triplets not observed at training time.
Traditional training method of KGE models cannot address the OOKB entity problem since it randomly gives scores of test triplets containing OOKB entities due to random initialization of entity embeddings before training.
In contrast, our training method initializes entity embeddings with knowledgeable entity representation.
Thus, we also evaluate our training method [Figure] [Figure] [Figure]
[Figure]
in the case of OOKB entities.
From the results in Fig.
2d- 2e, we can observe that our training framework can solve the OOKB entity problem on WordNet dataset and performs best when using
BERT to encode textual description of entities and relations including their names and deﬁnitions of word senses.
5. Effects of Knowledge Incorporation
Our training framework has natural advantages over traditional training method of KGE models since we learn better knowledgeable entity and relation representation via BERT before training a
KGE model.
This section veriﬁes the effectiveness of knowledge incorporation during the ﬁne-tuning phase.
5.2. Analysis of the Sensitivity of MR and MRR
We show the performance of Baseline-TransE and
Pretrain-TransE on WN18RR as iteration increases during the training phase in Fig. 2f.
From the results, we can observe a new changing trend of
MR and MRR for Pretrain-TransE. Compared to
Baseline-TransE for which MR decreases (better performance) and MRR increases (better performance) at training time, Pretrain-TransE shows both increasing results of MR and MRR.
We analyze the changing trend of MR and MRR in Theorem 1.
Formally, for n test triplets, we get corresponding ranks in link prediction task r
= ( r 1 , r 2 , · · · , r n ) , and MR ( r ) =
n (cid:80) i = r i (cid:14) n ; MRR ( r ) = n (cid:80) i = r − 1 i (cid:14) n .
Theorem 1.
5 Sensitivity of MR and MRR metrics
MR is more sensitive to tricky triplets than MRR.
Formally, for r = ( r 1 , r 2 , · · · , r n ) and r i > r j (triplet i is worse-learnt than triplet j ):
[Equation]
where f (cid:48) k
( r ) denotes ∂f∂r k ( f ∈ { MR, MRR } ) and means the sensitivity of metric f to triplet k .
In Figure 2c, we can observe that there is better performance on high-frequency triplets than
[Figure]
low-frequency ones which are more tricky to handle, since there is less information in datasets provided for low-frequency triplets.
According to Theorem 1, we can thus suggest that MR is more sensitive to low-frequency triplets while MRR is more sensitive to high-frequency triplets.
Reasons for the increasing MR of Pretrain-TransE in Fig.
2f are illustrated in the following.
5.2. Visualization of Knowledge Incorporation
We visualize the knowledge learning process of
Baseline-TransE and our Pretrain-TransE in Fig.
3a3c.
We select top ﬁve common supersenses in
WN18: plant , animal , act , person and artifact , among which the last three supersenses are all relevant to the concept of human beings and thus can be considered to constitute one common supersense.
In Fig.
3a, we can observe that Baseline-TransE learns entity and relation embeddings for triplets containing the ﬁve supersenses but does not distinguish embeddings between plant , animal and the other three supersenses.
In contrast, Fig.
3b shows that our Pretrain-TransE can further distinguish embeddings between different supersenses, especially separating supersenses related to human beings from others.
The main reason is that we can learn better knowledgeable entity and relation representation via BERT by incorporating rich linguistic knowledge into entity and relation embeddings during the initializing phase.
However, during the training phase, our PretrainTransE gradually learns different KGEs from those in the initializing phase.
Fig.
3c shows that it is due to the oblivion of partial linguistic knowledge incorporated into entity and relation embeddings as the
KGEs learn more information contained in datasets at training time.
This process can account for the increasing MR results of Pretrain-TransE during the training phase in Fig.
2f.
But the absolute values of MR and MRR for our Pretrain-TransE are overtly lower than those for TransE baseline, which demonstrates that our training framework enables to learn better knowledgeable entity and relation representation and there still remains incorporated knowledge in entity and relation embeddings during the training phase.
To conclude, during the training phase, TransE baseline learns original knowledge contained in datasets.
Instead, our proposed method ﬁrst learns rich linguistic knowledge from BERT, and continues to learn knowledge from datasets while losing partial knowledge learned from BERT.
Finally, there still remains knowledge incorporated in entity and relation embeddings during the training phase.
Conclusion
We present Pretrain-KGEs, a simple and efﬁcient pretraining technique for knowledge graph embedding models.
Pretrain-KGEs is a general technique that can be applied to any KGE model.
It contributes to learn better knowledgeable entity and relation representations from pretrained language models, which are leveraged during the initializing and the training phases for a KGE model to learn entity and relation embeddings.
Through extensive experiments, we demonstrate state-of-the-art performances using this effective pretraining technique on various benchmark datasets.
Further, we verify the effectiveness of our method by demonstrating promising results in the case of fewer training triplets, infrequent and OOKB entities which are particularly hard to handle due to lack of knowledge representation.
We ﬁnally analyze the effects of knowledge incorporation by demonstrating the sensitivity of MR and MRR metrics and visualizing the process of knowledge incorporation.
Acknowledgments
This work is partly supported by Beijing Academy of Artiﬁcial Intelligence (BAAI).
Xu Sun is the corresponding author.
A Detailed Implementation A. Implementation
Our implementations of TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), RotatE (Sun et al., 2019), pRotatE (Sun et al., 2019) are based on the framework provided by
Sun et al.
Our implementation of QuatE is based on on the framework provided by Zhang et al.
In ﬁne-tuning phase, we adopt the following non-linear pointwise function σ ( · ) : for x = x 0 +
K − 1 (cid:80) i
= x i e i ∈ F (where F can be real number ﬁled R , complex number ﬁled C or quaternion number ring
H ):
[Equation] [Equation]
where x i ∈ R and e i is the K -dimension hypercomplex-value unit.
For instance, when K = 1 , F = R ; when K = 2 , F = C , e 1 = i (the imaginary unit); when K = 4 , F = H , e 1 , 2 , 3 = i , j , k
(the quaternion units).
The score functions of baselines are listed in Table 4.
[Table]
A. Implementation of Word-averaging Baseline
We also implement the word-averaging baseline to utilize the entitiy names and entity deﬁnition in
WordNet to represent the entity embedding better.
Formally, for entitiy e and its textual description
T ( e ) =
w 1 w 2 · · · w L , where w i denotes the i -th token in sentence T ( e ) and T ( e ) here together utilizing the entitiy names and entity deﬁnition in WordNet.
[Equation]
where u i denotes the word embedding of token w i , which is a trainable randomly initialized parameter and will be trained in the pretraining phase.
We also adopt our three-phase training method to train word-averaging baseline.
Similarly, E =
[ E 1 ; E 2 ; · · · ; E k ] ∈ F k × d and R =
[ R 1 ; R 2 ; · · · ; R l ] ∈ F l × d denote entity and relation embeddings.
In pretraining phase, for head entity h , tail entity t and relation r , the score function is calculated as:
[Equation]
where R r denotes the relation embedding of relation r .
In initializing phase, similar to our proposed model, we initialize E i with Avg ( e i ) .
In training phase, we optimize E and R with the same training method to TransE baseline.
A. Dataset Statistics
We evaluate our proposed training framework on four benchmark KG datasets: WN18 (Bordes et al., 2013), WN18RR (Dettmers et al., 2018), FB15K (Bordes et al., 2013) and FB15K-
(Toutanova and
Chen, 2015).
We list detailed statistics of datasets are in Table
[Figure]
[Table]
[Figure]
[Table]
A. Experimental Settings
The hyper-parameters of are listed in Table 6.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
adam adam
[Figure]
adam
[Figure]
adam
[Figure]
adam adam
[Figure]
adam adam
[Figure]
adam adam
[Figure]
adam
[Figure]
adam
[Figure]
WN18RR
TransE
[Figure]
B Proof of Theorem
Proof.
According to deﬁnitions
[Equation]
derive with respect to r k ,
[Equation]
[Equation]
therefore, | MR (cid:48) i ( r ) | | MR (cid:48) j ( r ) | = 1 , | MRR (cid:48) i ( r ) | | MRR (cid:48)
j ( r ) | = ( r j r i ) 2 < 1 ( r i > r j ) , | MR (cid:48) i ( r ) | | MR (cid:48) j ( r ) | > | MRR (cid:48) i ( r ) | | MRR (cid:48)
j ( r ) | (13)