H IERARCHICAL R ELATIONAL L EARNING FOR F EWS HOT K NOWLEDGE G RAPH C OMPLETION
Anonymous authors
Paper under double-blind review
A BSTRACT
Knowledge graphs (KGs) are powerful in terms of their inference abilities, but are also notorious for their incompleteness and long-tail distribution of relations.
To address these challenges and expand the coverage of KGs, few-shot KG completion aims to make predictions for triplets involving novel relations when only a few training triplets are provided as reference.
Previous methods have focused on designing local neighbor aggregators to learn entity-level information and/or imposing sequential dependency assumption at the triplet level to learn meta relation information.
However, pairwise triplet-level interactions and context-level relational information have been largely overlooked for learning meta representations of few-shot relations.
In this paper, we propose a hierarchical relational learning method (HiRe) for few-shot KG completion.
By jointly capturing three levels of relational information (entity-level, triplet-level and context-level), HiRe can effectively learn and refine the meta representation of few-shot relations, and thus generalize well to new unseen relations.
Extensive experiments on benchmark datasets validate the superiority of HiRe over state-of-the-art methods.
The code of HiRe can be found in supplementary material and will be released upon acceptance.
I NTRODUCTION
Knowledge graphs (KGs) comprise of a collection of factual triplets, ( h, r, t ) , where each triplet expresses the relationship r between a head entity h and a tail entity t .
Large-scale KGs (Vrandeˇci´c & Kr¨otzsch, 2014; Mitchell et al., 2018; Suchanek et al., 2007; Bollacker et al., 2008) can provide powerful inference capabilities for many intelligent applications, including question answering (Yao & Van Durme, 2014), web search (Eder, 2012) and recommendation systems (Wang et al., 2019).
As KGs are often built semi-automatically from unstructured data, real-world KGs are far from complete and suffer from the notorious long-tail problem — a considerable proportion of relations are associated with only very few triplets.
As a result, the performance of current KG completion methods significantly degrades when predicting relations with a limited number (few-shot) of training triplets.
To tackle this challenge, few-shot KG completion methods have been proposed including
GMatching (Xiong et al., 2018), MetaR(Chen et al., 2019), FSRL(Zhang et al., 2020), FAAN (Sheng et al., 2020) and GANA (Niu et al., 2021).
These methods focus on predicting the missing tail entity t for query triplets by learning from only K reference triplets about a target relation r .
Given a target relation r and K reference triplets, K -shot KG completion aims to correctly predict the tail entity t for each query triplet ( h, r, ?)
using the generalizable knowledge learned from reference triplets.
Thus, the crucial aspect of few-shot KG completion is to learn the meta representation of each few-shot relation from the limited amount of reference triplets that can generalize to novel relations.
To facilitate the learning of meta relation representation, we identify three levels of relational information (see Figure 1).
At the context level , each reference triplet is closely related to its wider contexts, providing crucial evidence for enriching entity and relation embeddings.
At the triplet level , capturing the commonality among limited reference triplets is essential for learning meta relation representations.
At the entity level , the learned meta relation representations should well generalize to unseen query triplets.
Current few-shot KG methods have, however, focused on designing local neighbor aggregators to learn entity-level information, and/or imposing sequential assumption at the triplet level to learn meta relation information (See Table 1).
The potential of leveraging pairwise triplet-level interactions and context-level relational information has been largely unexplored.
ℎ 𝑡𝑡 …
[Figure]
ℎ𝑡𝑡ℎ 𝑡𝑡 context
Context-level relational information triplet ℎ𝑡𝑡ℎ 𝑡𝑡ℎ 𝑘𝑘 𝑡𝑡 k Triplet-level relational information ℎ𝑡𝑡ℎ𝑡𝑡
Entity-level relational information
[Figure]
[Figure]
GMatching ✓ ✗ ✗ ✗
MetaR ✓ ✗ ✗ ✗
FSRL ✓ ✓ ✗ ✗
FAAN ✓ ✓ ✗ ✗
GANA ✓ ✓ ✗ ✗
HiRe (ours) ✓ ✗ ✓ ✓
In this paper, we propose a Hi erarchical Re lational learning framework (HiRe) for few-shot KG completion.
HiRe jointly models three levels of relational information (entity-level, triplet-level, and context-level) within each few-shot task as mutually reinforcing sources of information to generalize to few-shot relations.
Here, ”hierarchical” references relational learning performed at three different levels of granularity.
Specifically, we make the following contributions:
• We propose a contrastive learning based context-level relational learning method to learn expressive entity/relation embeddings by modeling correlations between the target triplet and its true/false contexts.
We argue that a triplet itself has a close relationship with its true context.
Thus, we take a contrastive approach — a given triplet should be pulled close to its true context, but pushed apart from its false contexts — to learn better entity embeddings.
• We propose a transformer-based meta relation learner (MRL) to learn generalizable meta relation representations.
Our proposed MRL is capable of capturing pairwise interactions among reference triplets, while preserving the permutation-invariance property and being insensitive to the size of the reference set.
• We devise a meta representation based embedding learner named MTransD that constrains the learned meta relation representations to hold between unseen query triplets, enabling better generalization to novel relations.
Lastly, we adopt a model agnostic meta learning (MAML) based training strategy (Finn et al., 2017) to optimize HiRe on each meta task within a unified framework.
By performing relational learning at three granular levels, HiRe offers significant advantages for extracting expressive meta relation representations and improving model generalizability for few-shot KG completion.
Extensive experiments on two benchmark datasets validate the superiority of HiRe over state-of-the-art methods.
R ELATED W ORK 2. R ELATIONAL L EARNING IN K NOWLEDGE G RAPHS
KG completion methods utilize relational information available in KGs to learn a unified lowdimensional embedding space for the input triplets.
TransE (Bordes et al., 2013) is the first to use relation r as a translation for learning an embedding space, i.e. h + r ≈ t when ( h, r, t ) ∈ T P .
A scoring function is then used to measure the quality of the translation and to learn a unified embedding space.
TransH (Wang et al., 2014) and TransR (Lin et al., 2015) further model relation-specific information for learning an embedding space.
ComplEx (Trouillon et al., 2016), RotatE (Sun et al., 2019b), and ComplEx-N3 (Lacroix et al., 2018) improve the modeling of relation patterns in a vector/complex space.
ConvE (Dettmers et al., 2018) and ConvKB (Nguyen et al., 2018) employ convolution operators to enhance entity/relation embedding learning.
However, these methods require a large number of triplets for each relation to learn a unified embedding space.
Their performance significantly degrades at few-shot settings, where only very few triplets are available for each relation.
2. F EW - SHOT KG C OMPLETION
Existing few-shot KG completion methods can be grouped into two main categories: (1) Metric learning based methods: GMatching (Xiong et al., 2018) is the first work to formulate few-shot (one-shot) KG completion.
GMatching comprises of two parts: a neighbor encoder that aggregates one-hop neighbors of any given entity, and a matching processor that compares similarity between query and reference entity pairs.
FSRL (Zhang et al., 2020) relaxes the setting to more shots and explores how to integrate the information learned from multiple reference triplets.
FAAN (Sheng et al., 2020) proposes a dynamic attention mechanism for designing one-hop neighbor aggregators.
(2) Meta learning based methods: MetaR (Chen et al., 2019) learns to transfer relation-specific meta information, but it simply generates meta relation representations by averaging the representations of all reference triplets.
GANA (Niu et al., 2021) puts more emphasis on neighboring information and accordingly proposes a gated and attentive neighbor aggregator.
Despite excellent empirical performance, the aforementioned methods suffer from two major limitations.
First, they focus on designing local neighbor aggregators to learn entity-level information.
Second, they impose a potentially invalid sequential dependency assumption and utilize recurrent processors ( i.e. , LSTMs (Hochreiter & Schmidhuber, 1996)) to learn meta relation representations.
Thus, current methods fail to capture pairwise triplet-level interactions and context-level relational information.
Our work is proposed to fill this important research gap in the literature.
2. C ONTRASTIVE L EARNING ON G RAPHS
As a self-supervised learning scheme, contrastive learning follows the instance discrimination principle that pairs instances based on whether they are derived from the same instance ( i.e. positive pairs) or not ( i.e. negative pairs) (Hadsell et al., 2006; Dosovitskiy et al., 2014).
Contrastive methods have been recently proposed to learn expressive node embeddings on graphs.
In general, these methods train a graph encoder that produces node embeddings and a discriminator that distinguishes similar node embedding pairs from those dissimilar ones.
DGI (Velickovic et al., 2019) trains a node encoder to maximize mutual information between patch representations and high-level graph summaries.
InfoGraph (Sun et al., 2019a) contrasts a global graph representation with substructure representations.
(Hassani & Khasahmadi, 2020) propose to contrast encodings from one-hop neighbors and a graph diffusion.
GCC (Qiu et al., 2020) is a pre-training framework that leverages contrastive learning to capture structural properties across multiple networks.
In KGs, we note that positive and negative pairs naturally exist in the few-shot KG completion problem.
However, the potential of contrastive learning in this task is under-explored.
In our work, we adopt the idea of contrastive learning at the context level to capture correlations between a target triplet and its wider context.
This enables enriching the expressiveness of entity embeddings and improving model generalisation for few-shot KG completion.
To the best of our knowledge, we are the first to integrate contrastive learning with KG embedding learning for few-shot KG completion.
P ROBLEM F ORMULATION
In this section, we formally define the few-shot KG completion task and problem setting.
The notations used in the paper can be found in Appendix A.
Definition 1 Knowledge Graph G .
A knowledge graph (KG) can be denoted as G = {E , R , T P} .
E and R are the entity set and the relation set, respectively.
T P = { ( h, r, t ) ∈
E × R × E} denotes the set of all triplets in the knowledge graph.
Definition 2 Few-shot KG Completion .
Given (i) a knowledge graph G = {E , R , T P} , (ii) a reference set S r = { ( h i , t i ) ∈
E × E|∃ r, s.t.
( h i , r, t i ) ∈ T P} that corresponds to a given relation r ∈ R , where |S r | = K , and (iii) a query set Q r = { (˜ h j , r, ?) } that also corresponds to relation r , the K -shot KG completion task aims to predict the true tail entities for triplets from Q r based on the knowledge learned from G and S r .
For each query triplet (˜ h j , r, ?)
∈ Q r , given a set of candidates
C ˜ h
j ,r for the missing tail entity, the goal is to rank the true tail entity highest among C ˜ h
j ,r .
As the definition states, few-shot KG completion is a relation-specific task.
The goal of a few-shot
KG completion model is to correctly predict new triplets about a relation r when only a few triplets about r are available.
Therefore, the training process is based on the unit of tasks, where each task is to predict new triplets for a given relation r , denoted as T r .
Each meta training task T r corresponds to a given relation r and is composed of a reference set S r and a query set Q r , i.e. T r = {S r , Q r } :
[Equation]
where M is the size of Q r .
Mathematically, the training set can be denoted as T train =
{T i } Ii =1 .
The test set T test =
{T j } Jj =1 can be similarly denoted.
Note that all triplets corresponding to the relations in test set are unseen during training, i.e. T train ∩ T test = ∅ .
[Figure]
T HE P ROPOSED M ETHOD
In this section, we present our proposed learning framework in details.
As discussed earlier, we identify the research gap in K -shot KG completion, where learning only entity-level relational information and capturing sequential dependencies between reference triplets prevent the model from capturing a more stereoscopic and generalizable representation for the target relation.
To fill this gap, we propose to perform three hierarchical levels of relational learning within each meta task (context-level, triplet-level and entity-level) for few-shot KG completion.
An overview of our proposed hierarchical relational learning (HiRe) framework can be found in Appendix C.
4. C ONTRASTIVE LEARNING BASED CONTEXT - LEVEL RELATIONAL LEARNING
Given a reference set S r with K training triplets, existing works ( e.g. , (Xiong et al., 2018; Niu et al., 2021)) seek to learn better head/tail entity embeddings by aggregating information from its respective local neighbors, and then concatenate them as triplet representation.
Although treating the neighborhoods of head/tail entity separately is a common practice in homogeneous graphs, we argue that this approach is suboptimal for few-shot KG completion due to the loss of critical information.
Taking Figure 2 as an example, jointly considering the wider context shared by head/tail entity would reveal crucial information – both “Lionel Messi” and “Sergio Ag¨uero” playFor “Argentina’s
National Team” – for determining whether the relation workWith holds between the given entity pair (“Lionel Messi”, “Sergio Ag ¨ uero”).
Notably, our statistical analysis further affirms that the triplets on KGs indeed share a significant amount of context information (see detailed results in Appendix B.
Motivated by this important observation, we propose the idea of jointly considering the neighborhoods of head/tail entity as the context of the given triplet to exploit more meticulous relational information.
To imbue the embedding of the target triplet with such contextual information, a contrastive loss is employed by contrasting the triplet with its true context against false ones.
Formally, given a target triplet ( h, r, t ) , we denote its wider context as C ( h,r,t ) =
N h ∪ N t , where
N h
= { ( r i , t i ) | ( h, r i , t i ) ∈ T P} and N t
= { ( r j , t j ) | ( t, r j , t j ) ∈ T P} .
Our goal is to capture context-level relational information — the correlation between the given triplet ( h, r, t ) and its true context C ( h,r,t ) .
We further propose a multi-head self-attention (MSA) based context encoder, which models the interactions among the given context C ( h,r,t ) and assigns larger weights to more important relation-entity tuples within the context shared by head entity h and tail entity t .
Specifically, given a target triplet ( h, r, t ) and its context C ( h,r,t ) , each relation-entity tuple ( r i , t i ) ∈
C ( h,r,t ) is first encoded as re i =
r i ⊕ t i , where r i ∈ R d and t i ∈ R
d is the relation and entity embedding, respectively, and r i ⊕ t i indicates the concatenation of two vectors r i and t i .
An MSA block is then employed to uncover the underlying relationships within the context and generate context embedding c :
[Equation]
where c 0 is the concatenation of the embeddings of all relation-entity tuples and | x | is the size of set x .
The self-attention scores among all relation-entity tuples from C ( h,r,t ) can be computed by
Eq. 4.
Tuples with higher correlations would be given larger weights and contribute more towards the embedding of C ( h,r,t ) .
The detailed implementation of MSA is given in Appendix D.1.
Additionally, we synthesize a group of false contexts { ˜ C ( h,r,t ) i } by randomly corrupting the relation or entity of each relation-entity tuple ( r i , t i
) ∈ C ( h,r,t ) .
Each false context ˜ C ( h,r,t ) i can be learned via the context encoder as ˜ c i .
Then, we use a contrastive loss to pull close the embedding of the target triplet with its true context and to push away from its false contexts.
The contrastive loss function is defined as follows:
[Equation]
where N is the number of false contexts for ( h, r, t ) , τ denotes the temperature parameter, h ⊕ t indicates the triplet embedding represented as the concatenation of its head and tail entity embeddings, sim( x, y ) measures the cosine similarity between x and y .
Thus, we can inject context-level knowledge into entity embeddings attending to key elements within the context of the given triplet.
4. T RANSFORMER - BASED TRIPLET - LEVEL RELATIONAL LEARNING
After obtaining the embeddings of all reference triplets, the next focus is to learn meta representation for the target relation r .
State-of-the-art models ( e.g. FSRL (Zhang et al., 2020), GANA (Niu et al., 2021)) utilize LSTMs for this purpose, which inevitably imposes an unrealistic sequential dependency assumption on reference triplets since LSTMs are designed to model sequence data.
However, these triplets are associated with the same relation and are not sequentially dependent on each other; the occurrence of one reference triplet does not necessarily lead to other triplets in the reference set.
Consequently, these LSTM-based models violate two important properties.
First, the model should be insensitive to the size of reference set ( i.e. few-shot size K ).
Second, the triplets in the reference set should be permutation-invariant.
To address these issues, we resort to modeling complex interactions among reference triplets for learning generalizable meta relational knowledge.
To this end, we propose a transformer-based meta relation learner (MRL) that effectively models pairwise interactions among reference triplets and learns meta relation representations that generalize well to new unseen relations.
There are two main considerations in our model design.
(1) reference triplets are permutation-invariant; (2) reference triplets that are more representative should be given higher weights when learning meta relation representation.
Inspired by Set Transformer (Lee et al., 2019), we design our MRL based on the idea of set attention block (SAB), which takes a set of objects as input and perform self-attention mechanism between the elements.
Therefore, our proposed MRL can model pairwise triplet-triplet interactions within S r so as to cultivate the ability to learn generalizable meta representation of the target relation.
Mathematically, given a meta training task T r targeting at relation r , our proposed MRL takes the head/tail entity pairs from its reference set as input, i.e. { ( h i , t i ) ∈ S r } .
Each reference triplet is encoded as x i
= h
i ⊕
t i , where h i ∈ R d and t i ∈ R
d are embeddings of entity
h
i and tail entity t i with dimension d .
Note that the same entity embeddings h
i and t i are used here as in Eq.
Since all reference triplets are associated with the same relation r , our proposed MRL aims to capture the commonality among these reference triplets and obtain meta representation for relation r .
To comprehensively incorporate triplet-level relational information in the reference set, we leverage an
SAB on the embeddings of all reference triplets from S r (see the details of SAB in Appendix D.2) :
[Equation]
where x i denotes the i -th reference triplet.
The output of SAB has the same size of the input X , but contains pairwise triplet-triplet interactions among X .
The transformed embeddings of reference triplets, X ′ , are then fed into a two-layer MLP to obtain the meta representation R T r , i.e. :
[Equation]
[Equation]
where the meta representation R T r is generated by averaging the transformed embeddings of all reference triplets.
This ensures that R T r contains the fused pairwise triplet-triplet interactions among the reference set in a permutation-invariant manner.
4. M ETA REPRESENTATION BASED ENTITY - LEVEL RELATIONAL LEARNING
A crucial aspect of few-shot KG completion is to warrant the generalizability of the learned meta representation.
The learned meta representation R T r should hold between ( h i , t i ) if h i and t i are associated with r .
This motivates us to refine R T r under the constraints of true/false entity pairs.
Translational models provide an intuitive solution by using the relation as a translation, enabling to explicitly model and constrain the learning of generalizable meta knowledge at the entity level.
Following the paradigm of KG translational models (Bordes et al., 2013; Wang et al., 2014; Ji et al., 2015), we design a score function that accounts for the diversity of entities and relations in a dynamic manner to satisfy such constraints.
This meta representation based embedding constraint is referred to as MTransD for effectively building meta translational relationship at the entity level.
Given a target relation r and its corresponding reference set S r , after obtaining its meta representation R T r , we can now calculate a score for each entity pair ( h i , t i ) ∈ S r by projecting the embedding of head/tail entity into a latent space determined by its corresponding entities and relation simultaneously.
Mathematically, the projection process and the score function can be formulated as:
[Equation]
where || x || 2 represents the ℓ 2 norm of vector x ,
h i / t i are the head/tail entity embeddings, h pi / t pi are their corresponding projection vectors.
r pi is the projection vector of R T r , and I m × n is an identity matrix (Ji et al., 2015).
In this way, the projection matrices of each head/tail entity are determined by both the entity itself and the relation that connects it.
As a result, the projected tail entity embedding should be closest to the projected head entity embedding after being translated by R T r .
That is to say, for these triplets associated with relation r , the corresponding meta representation R T r should hold between h pi and t pi at entity level in the projection space.
Considering the whole reference set, we can further define a loss function as follows:
[Equation]
where γ is a hyper-parameter that determines the margin to separate positive pairs from negative pairs.
score ( h i , t ′ i ) calculates the score of a negative pair ( h i , t ′ i ) which results from negative sampling of the positive pair ( h i , t i ) ∈ S r , i.e. ( h i , r, t ′ i ) / ∈ G .
Till now, we have obtained meta relation representation R T r for each few-shot relation r , along with a loss function on the reference set.
4. MAML- BASED TRAINING STRATEGY
Noting that L ( S r ) in Eq.
12 is task-specific and should be minimized on the target task T r , we adopt an MAML-based training strategy (Finn et al., 2017) to optimize the parameters on each task T r .
The obtained loss on reference set L ( S r ) is not used to train the whole model but to update intermediate parameters.
Please refer to Appendix E for detailed training scheme of MAML.
Specifically, the learned meta representation R T r can be further refined based on the gradient of L ( S r ) :
[Equation]
(13) where l r indicates the learning rate.
Furthermore, for each target relation T r , the projection vectors h pi , r pi and t pi can also be optimized in the same manner of MAML so that the model can generalize and adapt to a new target relation.
Following MAML, the projection vectors are updated as follows:
[Equation]
[Equation] [Equation]
With the updated parameters, we can project and score each entity pair ( h j , t j ) from the query set
Q r following the same scheme as reference set and obtain the entity-level loss function L ( Q r ) :
[Equation]
where ( h j , t ′ j ) is also a negative triplet generated in the same way as ( h i , t ′ i ) .
The optimization objective for training the whole model is to minimize L ( Q r ) and L c together, i.e. :
[Equation]
where λ is a trade-off hyper-parameter that balances the contributions of L ( Q r ) and L c .
E XPERIMENTS 5. D ATASETS AND E VALUATION M ETRICS
We conduct experiments on two widely-used few-shot KG completion datasets, Nell-One and
Wiki-One, which are constructed by (Xiong et al., 2018).
For fair comparison, we follow the experimental setup of GMatching (Xiong et al., 2018), where relations associated with more than 50 but less than 500 triplets are chosen for few-shot completion tasks.
For each target relation, the candidate entity set provided by GMatching is used.
The statistics of both datasets are provided in Table 2.
We use 51/5/11 and 133/16/34 tasks for training/validation/test on Nell-One and Wiki-One, respectively, following the common setting in the literature.
[Table]
We report both MRR (mean reciprocal rank) and Hits@n ( n = 1 , 5 , 10 ) on both datasets for the evaluation of performance.
MRR is the mean reciprocal rank of the correct entities, and Hits@n is the ratio of correct entities that rank in top n .
We compare the proposed method against other baseline methods in 1-shot and 5-shot settings, which are the most common settings in the literature.
5. B ASELINES
For evaluation, we compare our proposed method against two groups of state-of-the-art baselines:
Conventional KG completion methods : TransE (Bordes et al., 2013), TransH (Wang et al., 2014),
DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016) and ComplEx-N3 (Lacroix et al., 2018).
We use OpenKE (Han et al., 2018) to reproduce the results of these models with hyperparameters reported in the original papers.
The models are trained using all triplets from background relations (Xiong et al., 2018) and training relations, as well as relations from all reference sets.
State-of-the-art few-shot KG completion methods : GMatching (Xiong et al., 2018), MetaR (Chen et al., 2019), FAAN (Sheng et al., 2020) and FSRL (Zhang et al., 2020).
For MetaR (both InTrain and Pre-Train) and FAAN, we directly report results obtained from the original papers.
For
GMatching, we report the results provided by (Chen et al., 2019) for both 1-shot and 5-shot.
As
FSRL was initially reported in different settings, where the candidate set is much smaller, we report the results re-implemented by (Sheng et al., 2020) under the same setting with other methods.
Due to the fact that the reproduced results of GANA (Niu et al., 2021) is less competitive, we leave GANA out in our comparison.
All reported results are produced based on the same experimental setting.
5. E XPERIMENTAL S ETUP
For fair comparison, we use the entity and relation embeddings pretrained by TransE (Bordes et al., 2013) on both datasets, released by GMatching (Xiong et al., 2018), for the initialization of our proposed HiRe.
Following the literature, the embedding dimension is set to 100 and 50 for NellOne and Wiki-One, respectively.
On both datasets, we set the number of SAB to 1 and each SAB contains one self-attention head.
We apply drop path to avoid overfitting with a drop rate of 0 . 2 .
The maximum number of neighbors of one given entity is set to 50, the same as prior works.
For all experiments except for the sensitivity test on the trade-off parameter λ from Eq. 21, λ is set to 0 .and
the number of false contexts for each reference triplet is set to 1 .
The margin γ from Eq.
12 is set to 1 .
We apply mini-batch gradient descent to train the network with a batch size of 1 , 024 for both datasets.
Adam optimizer is used with a learning rate of 0 .
We evaluate HiRe on validation set every 1 , 000 steps and choose the best model within 30 , 000 steps based on MRR.
All models are implemented by PyTorch and trained on 1 Tesla P100 GPU.
5. C OMPARISON WITH STATE - OF - THE - ART METHODS
Table 3 compares HiRe against baselines on Nell-One and Wiki-One under 1 -shot and 5 -shot settings.
In general, conventional KG completion methods are inferior to few-shot KG completion methods, especially udner 1 -shot setting.
This is expected because conventional KG completion methods are designed for scenarios with sufficient training data.
Overall, our HiRe method outperforms all baseline methods under two settings on both datasets, which validates its efficacy for few-shot KG completion.
Especially, as the number of reference triplets increases, HiRe achieves
[Table] [Table]
w
[Table]
larger performance gains because our transformer-based MRL can capture more complex tripletlevel interactions.
This further reinforces HiRe’s multi-level relational learning process in return.
As for performance gains in terms of MRR, Hits@10, Hits@5, and Hits@1, HiRe surpasses the second best performer by +3.8%,
+7.1%, +6.7%, and +1.4% in 1 -shot setting, and by +2.7%, +8.3%, +7.5%, and +0.7% in 5 -shot setting on Nell-One.
For performance gains on Wiki-One, HiRe outperforms the second best method by +0.8%, +2.9%, +0.8%, and +0.5% in 1 -shot setting, and by +3.0%, +3.3%, +2.4%, and +3.8% in 5 -shot setting.
HiRe achieves large performance improvements in terms of all metrics, proving that leveraging hierarchical relational information enhances the model’s generalizability and leads to an overall improvement in performance.
5. A BLATION S TUDY
Our proposed HiRe framework is composed of three key components.
To investigate the contributions of each component to the overall performance, we conduct a thorough ablation study on both datasets under 3-shot and 5-shot settings.
The detailed results on Wiki-One are reported in Table 4: w/o MTransD-MTransE and w/o MTransD-MTransH : To study the effectiveness of MTransD, we substitute MTransD with TransE and TransH respectively, retaining the MAML-based training strategy.
Substituting MTransD leads to performance drops at a significant level under both settings, indicating the necessity of constraining entity and relation embeddings while simultaneously considering the diversity of entities and relations.
w/o MAML : To demonstrate the efficacy of MAML-based training strategy, we removed MAMLbased training strategy from MTransD and replacing it with TransD. In this ablated variant, TransD is applied on the query set after the meta relation representation is learned from the reference set.
The significant performance drop suggests that MAML-based training strategy is essential for the model to learn generalizable meta knowledge to predict unseen relations in few-shot settings.
This conclusion has also been affirmed by the ablation study in MetaR (Chen et al., 2019).
w/o MRL-AVG : To study the impact of transformer-based MRL, we replace MRL by simply averaging the embeddings of all reference triplets to generate meta relation representations.
This has a profoundly negative effect, resulting in a performance drop of 4% in terms of MRR under 3 -shot setting and 5.4% under 5 -shot setting.
The performance under 3 -shot and 5 -shot settings are similar, indicating that simplistic averaging fails to take advantage of more training triplets.
This validates the importance of MRL to capture triplet-level interactions in learning meta relation representations.
w/o MRL-LSTM : To further validate the advantages of leveraging pairwise relational information over sequential information, we replace transformer-based MRL with an LSTM to generate meta re-
[Figure] [Figure]
lation representations.
The resultant performance drop is significant; the MRR drops 4.3% and 5.1% respectively under 3 -shot and 5 -shot settings.
Although the use of LSTM brings some improvements over simplistic averaging through capturing triplet-level relational information, the imposed unrealistic sequential dependency assumption results in limited performance gains.
This demonstrates the necessity and superiority of our proposed transformer-based MRL in capturing pairwise triplet-level relational information to learn meta representations of few-shot relations.
w/o Context :
By ablating L c , we remove contrastive learning based context-level relational learning but retain triplet-level and entity-level relational information.
As compared to jointly considering the context of the target triplet, the resultant performance drop verifies our assumption that the semantic contextual information plays a crucial role in few-shot KG completion.
Similar conclusions can also be drawn from the ablation results on Nell-One (See Appendix F).
5. H YPER - PARAMETER S ENTITIVITY
We conduct two sensitivity tests for the number of false contexts N and the trade-off parameter λ in
Eq.
21 on both datasets under 1 / 3 / 5 -shot settings.
See Appendix G for detailed results on Nell-One.
For hyper-parameter N , we set N as 1 , 2 , 4 , and 6 .
As Figure 3 shows, HiRe performs the best when
N = 1 on all settings, and its performance slightly drops when N = 2 .
As N continues to increase, the performance of HiRe drops accordingly.
One main reason is that, too many false contexts would dominate model training, causing the model to quickly converge to a sub-optimal state.
For hyper-parameter λ , since the value of contrastive loss is significantly larger than that of the margin loss, λ should be small to retain effective supervision from the margin loss.
Thus, we ablated the impact of different values of λ between 0 and 0.5.
As Figure 4 shows, HiRe achieves the best performance when λ = 0 .
With the contrastive loss ( i.e. , λ > 0 ), HiRe consistently yields better performance, proving the efficacy of our contrastive learning based context-level relational learning.
C ONCLUSION
This paper presents a hierarchical relational learning framework (HiRe) for few- shot KG completion.
We investigate the limitations of current few-shot KG completion methods and identify that jointly capturing three levels of relational information is crucial for enriching entity and relation embeddings, which ultimately leads to better meta representation of the target relation and model generalizability.
Experimental results on two commonly used benchmark datasets show that HiRe consistently outperforms current state-of-the-art methods, demonstrating its superiority and efficacy for few-shot KG completion.
The ablation analysis and hyper-parameter sensitivity study verify the significance of the key components of HiRe.
A N OTATIONS
The notations used in this paper are summarized in Table 5.
[Table]
B M OTIVATION : S HARED C ONTEXT S TATISTICS
One of our key motivations is that jointly considering the wider context shared by head/tail entity would reveal crucial information for learning expressive entity embeddings.
To justify our motivation, we perform a statistical analysis on Nell-One and Wiki-One dataset and the results are summarized in Table 6.
Overall, out of the 189,635 triplets in Nell-One dataset, up to 39,234 triplets have shared entities in their contexts.
That means, there exists at least one entity that is connected to both the head entity and the tail entity of the given triplet.
These 39,234 triplets share 117,386 entities in all, making each triplet have almost three shared entities by average.
Triplets that have shared entities in their contexts constitute more than 20.68% and 9.2% of the total triplets, respectively.
More strictly, the triplets that have shared (relation, entity) pairs in their contexts (meaning the head and tail entity are connected to the same entity by the same relation in the context) constitute 13.77% on Nell-One and 4.6% on Wiki-One, respectively.
Our analysis affirms that the triplets on KGs indeed share a significant amount of context information.
Our method is thus designed to leverage such crucial information for learning more expressive entity embeddings.
[Equation]
[Table] [Table]
[Table] [Table]
[Table]
C O VERVIEW OF THE P ROPOSED H I R E F RAMEWORK
Figure 5 shows an overview of our proposed hierarchical relational learning (HiRe) framework.
C o n t e x t - l e v e l r e l a t
i o n a l l e a r n i n g 𝒮𝒮 𝑟𝑟 ℛ 𝒯𝒯 𝑟𝑟 𝓛𝓛 𝒄𝒄
M T r a n s D ℒ 𝒮𝒮 𝑟𝑟
Avg … 𝑟𝑟 ℎ 1 𝑡𝑡
T r a n s f o r m e rb a s e d
M R L 𝑟𝑟 ℎ 2 𝑡𝑡𝑟𝑟 ℎ 𝑘𝑘 𝑡𝑡 𝑘𝑘 𝒮𝒮 𝑟𝑟 ℒ 𝒮𝒮 𝑟𝑟 ℛ 𝒯𝒯 𝑟𝑟 𝛁𝛁 ℛ 𝒯𝒯𝑟𝑟 ℒ 𝒮𝒮 𝑟𝑟 ℛ′ 𝒯𝒯 𝑟𝑟 = 𝒬𝒬 𝑟𝑟 𝓛𝓛
[Figure]
D D ETAILS : M ULTI -H EAD S ELF -A TTENTION AND S ET A TTENTION B LOCK D. D ETAILS OF M ULTI -H EAT S ELF A TTENTION For
context-level relational learning, we employ a Multi-Head Self-Attention (MSA) block (Vaswani et al., 2017) to uncover the underlying relationships within the context C ( h,r,t ) of a given triplet ( h, r, t ) and generate the context embedding c .
Specifically, take a context that contains k entity-relation tuples as an example, the context embedding is initialized as c ∈ R k × d c ( d c = 100 and d c = 200 for WiKi-One and Nell-One, respectively).
This input is first transformed into three different matrices: the query matrix Q ∈ R k × d v , the key matrix K ∈ R k × d k and the value matrix V ∈ R k × d v with dimension d q = d k = d v = d c .
Subsequently, the attention function is calculated as follows:
• Step 1 : Compute the scores between different input matrices as S = Q · K ⊤ ; • Step 2 : Normalize the scores for the stability of gradient as S n = S / √ d k ; • Step 3 : Translate the scores into probabilities with softmax function A = softmax( S n ) ; • Step 4 :
Obtain the weighted value matrix C
=
A · V
The above process can be unified into a single function:
[Equation]
The logic behind Eq.
22 is straightforward.
Step 1 computes a score between each pair of different relation-entity tuples from the context.
Step 2 normalizes the scores to enhance gradient stability for improved training, and Step 3 translates the scores into probabilities.
Finally, each relation-entity tuple is updated by the weighted sum based on the probabilities.
Overall, Eq. 3 and Eq. 4 in the main paper can be re-formulated as:
[Equation]
where A =
[ α 1 ; α 2 ; ..., ; α k ] .
Instead of performing a single attention function with d c -dimensional queries, keys and values, it would be beneficial to linearly project the queries, keys and values h times with different linear projections (called “multi-head”)
(Vaswani et al., 2017).
On each of these projected versions of queries, keys and values, the attention function is executed in parallel.
[Equation]
where head i =
Attention( QW
[Equation]
KW
[Equation]
25) where the projections are parameter matrices W Qi ∈ R d × d k , W Ki ∈ R d × d k , W Vi ∈ R d × d k and
W O ∈ R hd v × d .
In each paralleld attention layer, d k = d v = d/h .
D. D ETAILS OF S ET A TTENTION B LOCK
To comprehensively incorporate triplet-level relational information in the reference set S r , we design a transformer-based MRL based on a set attention block (Lee et al., 2019) to model interactions among all triplets in S r .
SAB is built based on multi-head attention, defined as:
[Equation]
26) where H = LayerNorm( X + MultiHead(X , X , X)) , rFF is any row-wise feedforward layer and
LayerNorm is layer normalization (Ba et al., 2016).
E MAML- BASED T RAINING S TRATEGY
The detailed MAML-based training framework can be described as following:
Algorithm 1: Training framework of HiRe.
Input:
F A BLATION S TUDY ON N ELL -O NE
Table 7 illustrates the contribution of each component to the overall performance on Nell-One.
As discussed in Section 5.5, each component in our proposed HiRe framework plays an important role in few-shot KG completion.
The removal of any component leads to performance drops in terms of all evaluation metrics.
[Table]
G H YPER -P ARAMETER S ENSITIVITY S TUDY ON N ELL -O NE
The sensitivity test on hyper-parameter N and λ is shown in Figure 6 and Figure7.
As illustrated in
Figure 6, as the number of false contexts increases, the performance of HiRe drops slightly because its model training converges to a sub-optimal state.
In addition, the best λ value is also 0 .
05 on NellOne.
The overall results are consistent with the conclusion we draw from the results on Wiki-One in Section 5.
[Figure]
[Figure]
H C OMPLEXITY A NALYSIS
[Table]
Table 8 lists the complexity of all the hierarchical relational learning modules, where d denotes the dimension of entity embeddings, k denotes the number of relation-entity tuples in the context, and n denotes the number of reference triplets in each task.
As can be seen, our proposed
HiRe quadratically scales with the number of reference triplets in each task.
Given the number of reference triplets is often very small ( ∼ 5 ) , our proposed HiRe scales reasonably well.