RNNL OGIC : L EARNING L OGIC R ULES FOR R EASONING ON K NOWLEDGE G RAPHS
A BSTRACT
This paper studies learning logic rules for reasoning on knowledge graphs.
Logic rules provide interpretable explanations when used for prediction as well as being able to generalize to other tasks, and hence are critical to learn.
Existing methods either suffer from the problem of searching in a large search space (e.g., neural logic programming) or ineffective optimization due to sparse rewards (e.g., techniques based on reinforcement learning).
To address these limitations, this paper proposes a probabilistic model called RNNLogic.
RNNLogic treats logic rules as a latent variable, and simultaneously trains a rule generator as well as a reasoning predictor with logic rules.
We develop an EM-based algorithm for optimization.
In each iteration, the reasoning predictor is ﬁrst updated to explore some generated logic rules for reasoning.
Then in the E-step, we select a set of high-quality rules from all generated rules with both the rule generator and reasoning predictor via posterior inference; and in the M-step, the rule generator is updated with the rules selected in the E-step.
Experiments on four datasets prove the effectiveness of RNNLogic.
I NTRODUCTION
Knowledge graphs are collections of real-world facts, which are useful in various applications.
Each fact is typically speciﬁed as a triplet ( h, r , t ) or equivalently r ( h, t ) , meaning entity h has relation r with entity t .
For example, Bill Gates is the Co-founder of Microsoft .
As it is impossible to collect all facts, knowledge graphs are incomplete.
Therefore, a fundamental problem on knowledge graphs is to predict missing facts by reasoning with existing ones, a.k.a.
knowledge graph reasoning.
This paper studies learning logic rules for reasoning on knowledge graphs.
For example, one may extract a rule ∀ X, Y, Z hobby ( X, Y ) ← friend ( X, Z ) ∧ hobby ( Z, Y ) , meaning that if Z is a friend of X and Z has hobby Y , then Y is also likely the hobby of X .
Then the rule can be applied to infer new hobbies of people.
Such logic rules are able to improve interpretability and precision of reasoning (Qu & Tang, 2019; Zhang et al., 2020).
Moreover, logic rules can also be reused and generalized to other domains and data (Teru & Hamilton, 2020).
However, due to the large search space, inferring high-quality logic rules for reasoning on knowledge graphs is a challenging task.
Indeed, a variety of methods have been proposed for learning logic rules from knowledge graphs.
Most traditional methods such as path ranking (Lao & Cohen, 2010) and Markov logic networks (Richardson & Domingos, 2006)
enumerate relational paths on graphs as candidate logic rules, and then learn a weight for each rule as an assessment of rule qualities.
There are also some recent methods based on neural logic programming (Yang et al., 2017) and neural theorem provers (Rockt¨aschel & Riedel, 2017), which are able to learn logic rules and their weights simultaneously in a differentiable way.
Though empirically effective for prediction, the search space of these methods is exponentially large, making it hard to identify high-quality logic rules.
Besides, some recent efforts (Xiong et al., 2017) formulate the problem as a sequential decision making process, and use reinforcement learning to search for logic rules, which signiﬁcantly reduces search complexity.
However, due to the large action space and sparse reward in training, the performance of these methods is not yet satisfying.
In this paper, we propose a principled probabilistic approach called RNNLogic which overcomes the above limitations.
Our approach consists of a rule generator as well as a reasoning predictor with logic rules, which are simultaneously trained to enhance each other.
The rule generator provides logic rules which are used by the reasoning predictor for reasoning, while the reasoning predictor provides effective reward to train the rule generator, which helps signiﬁcantly reduce the search space.
Speciﬁcally, for each query-answer pair, e.g., q = ( h, r, ?) and a = t , we model the probability of the answer conditioned on the query and existing knowledge graph G , i.e., p ( a |G , q ) , where a set of logic rules z 1 is treated as a latent variable.
The rule generator deﬁnes a prior distribution over logic rules for each query, i.e., p ( z | q ), which is parameterized by a recurrent neural network.
The reasoning predictor computes the likelihood of the answer conditioned on the logic rules and the existing knowledge graph G , i.e., p ( a |G , q , z ) .
At each training iteration, we ﬁrst sample a few logic rules from the rule generator, and further update the reasoning predictor to try out these rules for prediction.
Then an EM algorithm (Neal & Hinton, 1998) is used to optimize the rule generator.
In the E-step, a set of high-quality logic rules are selected from all the generated rules according to their posterior probabilities.
In the M-step, the rule generator is updated to imitate the high-quality rules selected in the E-step.
Extensive experimental results show that RNNLogic outperforms state-of-the-art methods for knowledge graph reasoning 2 .
Besides, RNNLogic is able to generate high-quality logic rules.
R ELATED W ORK
Our work is related to existing efforts on learning logic rules for knowledge graph reasoning.
Most traditional methods enumerate relational paths between query entities and answer entities as candidate logic rules, and further learn a scalar weight for each rule to assess the quality.
Representative methods include Markov logic networks (Kok & Domingos, 2005; Richardson & Domingos, 2006; Khot et al., 2011), relational dependency networks (Neville & Jensen, 2007; Natarajan et al., 2010), rule mining algorithms (Gal´arraga et al., 2013; Meilicke et al., 2019), path ranking (Lao & Cohen, 2010; Lao et al., 2011) and probabilistic personalized page rank (ProPPR) algorithms (Wang et al., 2013; 2014a;b).
Some recent methods extend the idea by simultaneously learning logic rules and the weights in a differentiable way, and most of them are based on neural logic programming (Rockt ¨ aschel &
Riedel, 2017; Yang et al., 2017; Cohen et al., 2018; Sadeghian et al., 2019; Yang & Song, 2020) or neural theorem provers (Rockt ¨ aschel & Riedel, 2017; Minervini et al., 2020).
These methods and our approach are similar in spirit, as they are all able to learn the weights of logic rules efﬁciently.
However, these existing methods try to simultaneously learn logic rules and their weights, which is nontrivial in terms of optimization.
The main innovation of our approach is to separate rule generation and rule weight learning by introducing a rule generator and a reasoning predictor respectively, which can mutually enhance each other.
The rule generator generates a few high-quality logic rules, and the reasoning predictor only focuses on learning the weights of such high-quality rules, which signiﬁcantly reduces the search space and leads to better reasoning results.
Meanwhile, the reasoning predictor can in turn help identify some useful logic rules to improve the rule generator.
The other kind of rule learning method is based on reinforcement learning.
The general idea is to train pathﬁnding agents, which search for reasoning paths in knowledge graphs to answer questions, and then extract logic rules from reasoning paths (Xiong et al., 2017; Chen et al., 2018; Das et al., 2018;
Lin et al., 2018;
Shen et al., 2018).
However, training effective pathﬁnding agents is highly challenging, as the reward signal (i.e., whether a path ends at the correct answer) can be extremely sparse.
Although some studies (Lin et al., 2018) try to get better reward by using embedding-based methods for reward shaping, the performance is still worse than most embedding-based methods.
In our approach, the rule generator has a similar role to those pathﬁnding agents.
The major difference is that we simultaneously train the rule generator and a reasoning predictor with logic rules, which mutually enhance each other.
The reasoning predictor provides effective reward for training the rule generator, and the rule generator offers high-quality rules to improve the reasoning predictor.
Our work is also related to knowledge graph embedding, which solves knowledge graph reasoning by learning entity and relation embeddings in latent spaces (
Bordes et al., 2013; Wang et al., 2014c;
Yang et al., 2015; Nickel et al., 2016; Trouillon et al., 2016; Cai & Wang, 2018; Dettmers et al., 2018;
Balazevic et al., 2019; Sun et al., 2019).
With proper architectures, these methods are able to learn [Figure] some simple logic rules.
For example, TransE (Bordes et al., 2013) can learn some composition rules.
RotatE (Sun et al., 2019) can mine some composition rules, symmetric rules and inverse rules.
However, these methods can only ﬁnd some simple rules in an implicit way.
In contrast, our approach explicitly trains a rule generator, which is able to generate more complicated logic rules.
There are some works studying boosting rule-based models (Goldberg & Eckstein, 2010; Eckstein et al., 2017), where they dynamically add new rules according to the rule weights learned so far.
These methods have been proven effective in binary classiﬁcation and regression.
Compared with them, our approach shares similar ideas, as we dynamically update the rule generator with the feedback from the reasoning predictor, but we focus on a different task, i.e., reasoning on knowledge graphs.
M ODEL
In this section, we introduce the proposed approach RNNLogic which learns logic rules for knowledge graph reasoning.
We ﬁrst formally deﬁne knowledge graph reasoning and logic rules.
Knowledge Graph Reasoning.
Let p data ( G , q , a ) be a training data distribution, where G is a background knowledge graph characterized by a set of ( h, r , t ) -triplets which we may also write as r ( h, t ) , q = ( h, r , ?) is a query, and a = t is the answer.
Given G and the query q , the goal is to predict the correct answer a .
More formally, we aim to model the probabilistic distribution p ( a |G , q ) .
Logic Rule.
We perform knowledge graph reasoning by learning logic rules, where logic rules in this paper have the conjunctive form ∀{ X i } li =
r ( X 0 , X l ) ← r 1 ( X 0 , X 1 ) ∧ · · · ∧ r l ( X l − 1 , X l ) with l being the rule length.
This syntactic structure naturally captures composition, and can easily express other common logic rules such as symmetric or inverse rules.
For example, let r − 1 denote the inverse relation of relation r , then each symmetric rule can be expressed as ∀{ X , Y } r ( X, Y ) ← r − 1 ( X, Y ) .
In RNNLogic, we treat a set of logic rules which could explain a query as a latent variable we have to infer.
To do this, we introduce a rule generator and a reasoning predictor using logic rules.
Given a query, the rule generator employs a recurrent neural network to generate a set of logic rules, which are given to the reasoning predictor for prediction.
We optimize RNNLogic with an EM-based algorithm.
In each iteration, we start with updating the reasoning predictor to try out some logic rules generated by the rule generator.
Then in the E-step, we identify a set of high-quality rules from all generated rules via posterior inference, with the prior from the rule generator and likelihood from the reasoning predictor.
Finally in the M-step, the rule generator is updated with the identiﬁed high-quality rules.
3. P ROBABILISTIC F ORMALIZATION
We start by formalizing knowledge graph reasoning in a probabilistic way, where a set of logic rules z is treated as a latent variable.
The target distribution p ( a |G , q ) is jointly modeled by a rule generator and a reasoning predictor.
The rule generator p θ deﬁnes a prior over a set of latent rules z conditioned on a query q , while the reasoning predictor p w gives the likelihood of the answer a conditioned on latent rules z , the query q , and the knowledge graph G .
Thus p ( a |G , q ) can be computed as below:
[Equation]
The goal is to jointly train the rule generator and reasoning predictor to maximize the likelihood of training data.
Formally, the objective function is presented as below:
[Equation]
3. P ARAMETERIZATION
Rule Generator.
The rule generator deﬁnes the distribution p θ ( z | q ) .
For a query q , the rule generator aims at generating a set of latent logic rules z for answering the query.
Formally, given a query q = ( h, r , ?) , we generate compositional logic rules by only considering the query relation r without the query entity h , which allows the generated rules to generalize across entities.
For each compositional rule in the abbreviation form r ← r 1 ∧· · ·∧ r l , it can be viewed as a sequence of relations [ r , r 1 , r 2 · · · r l , r END ] , where r is the query relation or the rule head, { r i } li = are the body of the rule, and r END is a special relation indicating the end of the relation sequence.
Such relation sequences can be effectively modeled by recurrent neural networks (Hochreiter &
Schmidhuber, 1997), and thus we introduce RNN θ to parameterize the rule generator.
Given a query relation r , RNN θ sequentially generates each relation in the body of a rule, until it reaches the ending relation r END .
In this process, the probabilities of generated rules are simultaneously computed.
With such rule probabilities, we deﬁne the distribution over a set of rules z as a multinomial distribution:
[Equation]
where Mu stands for multinomial distributions, N is a hyperparameter for the size of the set z , and
RNN θ ( ·| r ) deﬁnes a distribution over compositional rules with rule head being r .
The generative process of a rule set z is quite intuitive, where we simply generate N rules with RNN θ to form z .
Reasoning Predictor with Logic Rules.
The reasoning predictor deﬁnes p w ( a |G , q , z ) .
For a query q , the predictor uses a set of rules z to reason on a knowledge graph G and predict the answer a .
Following stochastic logic programming (Cussens, 2000), a principled reasoning framework, we use a log-linear model for reasoning.
For each query q
= ( h, r , ?) , a compositional rule is able to ﬁnd different grounding paths on graph G , leading to different candidate answers.
For example, given query ( Alice , hobby , ?) , a rule hobby ← friend ∧ hobby can have two groundings, hobby ( Alice , Sing ) ← friend ( Alice , Bob )
∧ hobby ( Bob , Sing ) and hobby ( Alice , Ski ) ← friend ( Alice , Charlie )
∧ hobby
( Charlie , Ski ) , yielding two candidate answers Sing and Ski .
Let A be the set of candidate answers which can be discovered by any logic rule in the set z .
For each candidate answer e ∈ A , we compute the following scalar score w ( e ) for that candidate:
[Equation]
where P ( h, rule , e ) is the set of grounding paths which start at h and end at e following a rule (e.g.,
Alice friend [Equation] Bob hobby −−−−→ Sing ).
ψ w ( rule ) and φ w ( path ) are scalar weights of each rule and path .
Intuitively, the score of each candidate answer e is the sum of scores contributed by each rule, i.e., score w ( e | rule ) .
To get score w ( e | rule ) , we sum over every grounding path found in the graph G .
For the scalar weight ψ w ( rule ) of each rule , it is a learnable parameter to optimize.
For the score φ w ( path ) of each speciﬁc path , we explore two methods for parameterization.
One method always sets φ w ( path )
= 1 .
However, this method cannot distinguish between different relational paths.
To address the problem, for the other method, we follow an embedding algorithm RotatE
(Sun et al., 2019) to introduce an embedding for each entity and model each relation as a rotation operator on entity embeddings.
Then for each grounding path of rule starting from h to e , if we rotate the embedding of h according to the rotation operators deﬁned by the body relations of rule , we should expect to obtain an embedding close to the embedding of e .
Thus we compute the similarity between the derived embedding and the embedding of e as φ w ( path ) , which can be viewed as a measure of the soundness and consistency of each path .
For example, given a path Alice friend −−−−→ Bob hobby −−−−→ Sing , we rotate Alice ’s embedding with the operators deﬁned by friend and hobby .
Afterwards we compute the similarity of the derived embedding and embedding of Sing as φ w ( path ) .
Such a method allows us to compute a speciﬁc φ w ( path ) for each path , which further leads to more precise scores for different candidate answers.
See App.
C for more details of the parameterization method.
Once we have the score for every candidate answer, we can further deﬁne the probability that the answer a of the query q is entity e by using a softmax function as follows:
[Equation]
3. O PTIMIZATION
Next, we introduce how we optimize the reasoning predictor and rule generator to maximize the objective in Eq.
At each training iteration, we ﬁrst update the reasoning predictor p w according to some rules generated by the generator, and then update the rule generator p θ with an EM algorithm.
In the E-step, a set of high-quality rules are identiﬁed from all generated rules via posterior inference, with the rule generator as the prior and the reasoning predictor as the likelihood.
In the M-step, the rule generator is then updated to be consistent with the high-quality rules selected in the E-step.
Formally, at each training iteration, we start with maximizing the objective O ( θ, w ) in Eq.
(2) with respect to the reasoning predictor p w .
To do that, we notice that there is an expectation operation with respect to p θ ( z | q ) for each training instance ( G , q , a ) .
By drawing a sample ˆ z ∼ p θ ( z | q ) for query q , we can approximate the objective function of w at each training instance ( G , q , a ) as below:
[Equation]
Basically, we sample a set of rules ˆ z from the generator and feed ˆ z into the reasoning predictor.
Then the parameter w of the reasoning predictor is updated to maximize the log-likelihood of the answer a .
With the updated reasoning predictor, we then update the rule generator p θ to maximize the objective
O ( θ, w ) .
In general, this can be achieved by REINFORCE (Williams, 1992) or the reparameterization trick (Jang et al., 2017; Maddison et al., 2017), but they are less effective in our problem due to the large number of logic rules.
Therefore, an EM framework is developed to optimize the rule generator.
E-step.
Recall that when optimizing the reasoning predictor, we draw a set of rules ˆ z for each data instance ( G , q , a ) , and let the reasoning predictor use ˆ z to predict a .
For each data instance, the E-step aims to identify a set of K high-quality rules
z I from all generated rules ˆ z , i.e., z I ⊂ ˆ z ,
| z I | =
K .
Formally, this is achieved by considering the posterior probability of each subset of logic rules
z I , i.e., p θ,w ( z I |G , q , a ) ∝ p w ( a |G , q , z I ) p θ ( z I | q ) , with prior of z I from the rule generator p θ and likelihood from the reasoning predictor p w .
The posterior combines knowledge from both the rule generator and reasoning predictor, so the likely set of high-quality rules can be obtained by sampling from the posterior.
However, sampling from the posterior is nontrivial due to its intractable partition function, so we approximate the posterior using a more tractable form with the proposition below:
Proposition 1 Consider a data instance ( G , q , a ) with q = ( h, r, ?) and a = t .
For a set of rules ˆ z generated by the rule generator p θ , we can compute the following score H for each rule ∈
ˆ z :
[Equation]
where A is the set of all candidate answers discovered by rules in ˆ z , score w ( e | rule ) is the score that each rule contributes to entity e as deﬁned by Eq.
(4), RNN θ ( rule | r ) is the prior probability of rule computed by the generator.
Suppose s = max e ∈A | score w ( e ) | < 1 .
Then for a subset of rules z I ⊂ ˆ z with | z
I | =
K , the log-probability log p θ,w ( z I |G , q , a ) can be approximated as follows:
[Equation]
where const is a constant term that is independent from z I , γ ( z I ) =
log( K ! / (cid:81) rule ∈
ˆ z n rule !) , with K being the given size of set
z I and n rule being the number of times each rule appears in z I .
We prove the proposition in App. A.1.
In practice, we can apply weight decay to the weight of logic rules in Eq.
(4), and thereby reduce s = max e ∈A | score w ( e ) | to get a more precise approximation.
The above proposition allows us to utilize ( (cid:80) rule ∈
z I H ( rule )+ γ ( z I )+ const ) to approximate the logposterior log p θ,w ( z I |G , q , a ) , yielding a distribution q
( z I )
∝ exp( (cid:80) rule ∈
z I H ( rule ) + γ ( z I )) as a good approximation of the posterior.
It turns out that the derived q ( z I ) is a multinomial distribution, and thus sampling from q ( z I ) is more tractable.
Speciﬁcally, a sample ˆ
z I from q ( z I ) can be formed with K logic rules which are independently sampled from ˆ z , where the probability of sampling each rule is computed as exp( H ( rule )) /
( (cid:80) rule (cid:48) ∈
ˆ z exp( H ( rule (cid:48) ))) .
We provide the proof in the App.
A.2.
Intuitively, we could view H ( rule ) of each rule as an assessment of the rule quality, which considers two factors.
The ﬁrst factor is based on the reasoning predictor p w , and it is computed as the score that a rule contributes to the correct answer t minus the mean score that this rule contributes to other candidate answers.
If a rule gives higher score to the true answer and lower score to other candidate answers, then the rule is more likely to be important.
The second factor is based on the rule generator p θ , where we compute the prior probability for each rule and use the probability for regularization.
Empirically, we ﬁnd that picking K rules with highest H ( rule ) to form ˆ z I works better than sampling from the posterior.
Similar observations have been made on the node classiﬁcation task (Qu et al., 2019).
In fact, ˆ z I formed by the top- K rules is an MAP estimation of the posterior, and thus the variant of picking top-
K rules yields a hard-assignment EM algorithm (Koller & Friedman, 2009).
Despite the reduced theoretical guarantees, we use this variant in practice for its good performance.
M-step.
Once we obtain a set of high-quality logic rules ˆ
z I for each data instance ( G , q , a ) in the
E-step, we further leverage those rules to update the parameters θ of the rule generator in the M-step.
Speciﬁcally, for each data instance ( G , q , a ) , we treat the corresponding rule set ˆ
z I as part of the (now complete) training data, and update the rule generator by maximizing the log-likelihood of ˆ z I :
[Equation]
With the above objective, the feedback from the reasoning predictor can be effectively distilled into the rule generator.
In this way, the rule generator will learn to only generate high-quality rules for the reasoning predictor to explore, which reduces the search space and yields better empirical results.
For more detailed analysis of the EM algorithm for optimizing O ( θ, w ) , please refer to App.
B.
Algorithm 1 Workﬂow of RNNLogic while not converge do
For each instance, use the rule generator p θ to generate a set of rules ˆ z ( | ˆ z | = N ).
For each instance, update the reasoning predictor p w based on generated rules ˆ z and Eq.
(cid:0) E-step: For each instance, identify K high-quality rules
ˆ z
I from ˆ z according to H ( rule ) in Eq.
(cid:0) M-step: For each instance, update the rule generator p θ according to the identiﬁed rules and Eq.
end while
During testing, for each query, use p θ to generate N rules and feed them into p w for prediction.
E XPERIMENT 4. E XPERIMENT S ETTINGS
Datasets.
We choose four datasets for evaluation, including FB15k-237 (Toutanova & Chen, 2015),
WN18RR (Dettmers et al., 2018), Kinship and UMLS (Kok & Domingos, 2007).
For Kinship and
UMLS, there are no standard data splits, so we randomly sample 30% of all the triplets for training, 20% for validation, and the rest 50% for testing.
The detailed statistics are summarized in the App.
D.
Compared Algorithms.
We compare the following algorithms in experiment: (cid:0)
Rule learning methods.
For traditional statistical relational learning methods, we choose Markov logic networks (Richardson & Domingos, 2006), boosted relational dependency networks (Natarajan et al., 2010) and path ranking (Lao & Cohen, 2010).
We also consider neural logic programming methods, including NeuralLP (Yang et al., 2017), DRUM (Sadeghian et al., 2019) and NLIL (Yang &
Song, 2020).
In addition, we compare against CTP (Minervini et al., 2020), a differentiable method based on neural theorem provers.
Besides, we consider three reinforcement learning methods, which
[Table] [Table] are MINERVA (Das et al., 2018), MultiHopKG (Lin et al., 2018) and M-Walk (Shen et al., 2018).
(cid:0)
Other methods.
We also compare with some embedding methods, including TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), ComplEx-N3 (Lacroix et al., 2018), ConvE (Dettmers et al., 2018), TuckER (Balazevic et al., 2019) and RotatE (Sun et al., 2019).
(cid:0)
RNNLogic.
For RNNLogic, we consider two model variants.
The ﬁrst variant assigns a constant score to different grounding paths in the reasoning predictor, i.e., φ w ( path )
= 1 in Eq.
(4), and we denote this variant as w/o emb. .
The second variant leverages entity embeddings and relation embeddings to compute the path score φ w ( path ) , and we denote the variant as with emb. .
Evaluation Metrics.
During evaluation, for each test triplet ( h, r , t ) , we build two queries ( h, r , ?)
and ( t, r − 1 , ?)
with answers t and h .
For each query, we compute a probability for each entity, and compute the rank of the correct answer.
Given the ranks from all queries, we report the Mean Rank (MR), Mean Reciprocal Rank (MRR) and Hit@ k (H@ k ) under the ﬁltered setting (Bordes et al., 2013), which is used by most existing studies.
Note that there can be a case where an algorithm assigns the same probability to the correct answer and a few other entities.
For such a case, many methods compute the rank of the correct answer as ( m + 1) where m is the number of entities receiving higher probabilities than the correct answer.
This setup can be problematic according to Sun et al.
For fair comparison, in that case we compute the expectation of each evaluation metric over all the random shufﬂes of entities which receive the same probability as the correct answer.
For example, if there are n entities which have the same probability as the correct answer in the above case, then we treat the rank of the correct answer as ( m + ( n + 1) / 2) when computing Mean Rank.
Besides, we notice that in MINERVA (Das et al., 2018) and MultiHopKG (Lin et al., 2018), they only consider queries in the form of ( h, r , ?) , which is different from our default setting.
To make fair comparison with these methods, we also apply RNNLogic to this setting and report the performance.
Experimental Setup of RNNLogic.
For each training triplet ( h, r , t ) , we add an inverse triplet ( t, r − 1 , h ) into the training set, yielding an augmented set of training triplets T .
We use a closedworld assumption for model training, which assumes that any triplet outside T is incorrect.
To build a training instance from p data , we ﬁrst randomly sample a triplet ( h, r , t ) from T , and then form an instance as ( G = T \ { ( h, r , t ) } , q = ( h, r , ?) , a = t ) .
Basically, we use the sampled triplet ( h, r , t ) to construct the query and answer, and use the rest of triplets in T to form the background knowledge graph G .
During testing, the background knowledge graph G is formed with all the triplets in T .
For the rule generator, the maximum length of generated rules is set to 4 for FB15k-237, 5 for WN18RR, and 3 for the rest, which are selected on validation data.
See App.
D for the detailed setting.
[Table]
4. R ESULTS
Comparison against Existing Methods.
We present the results on the FB15k-237 and WN18RR datasets in Tab. 1 and Tab. 2.
The results on the Kinship and UMLS datasets are shown in Tab. 3.
We ﬁrst compare RNNLogic with rule learning methods.
RNNLogic achieves much better results than statistical relational learning methods (MLN, Boosted RDN, PathRank) and neural differentiable methods (NeuralLP, DRUM, NLIL, CTP).
This is because the rule generator and reasoning predictor of RNNLogic can collaborate with each other to reduce search space and learn better rules.
RNNLogic also outperforms reinforcement learning methods (MINERVA, MultiHopKG, M-Walk).
The reason is that RNNLogic is optimized with an EM-based framework, in which the reasoning predictor provides more useful feedback to the rule generator, and thus addresses the challenge of sparse reward.
We then compare RNNLogic against state-of-the-art embedding-based methods.
For RNNLogic with embeddings in the reasoning predictor ( with emb. ), it outperforms most compared methods in most cases, and the reason is that RNNLogic is able to use logic rules to enhance reasoning performance.
For RNNLogic without embedding ( w/o emb. ), it achieves comparable results to embedding-based methods, especially on WN18RR, Kinship and UMLS where the training triplets are quite limited.
Quality of Learned Logic Rules.
Next, we study the quality of rules learned by different methods for reasoning.
For each trained model, we let it generate I rules with highest qualities per query relation, and use them to train a predictor w/o emb.
as in Eq.
(5) for reasoning.
For RNNLogic, the quality of each rule is measured by its prior probability from the rule generator, and we use beam search to infer top-
I rules.
The results at different I are in Fig.
2, where RNNLogic achieves much better results.
Even with only 10 rules per relation, RNNLogic still achieves competitive results.
Performance w.r.t.
the Number of Training Triplets.
To better evaluate different methods under cases where training triplets are very limited, in this section we reduce the amount of training data on
Kinship and UMLS to see how the performance varies.
The results are presented in Fig.
We see that RNNLogic w/o emb.
achieves the best results.
Besides, the improvement over RotatE is more signiﬁcant as we reduce training triplets, showing that RNNLogic is more robust to data sparsity.
[Figure]
[Figure]
Performance w.r.t.
Embedding Dimension.
RNNLogic with emb.
uses entity and relation embeddings to improve the reasoning predictor.
Next, we study its performance with different embedding dimensions.
The results are presented in Fig.
3, where we compare against RotatE (Sun et al., 2019).
We see that RNNLogic signiﬁcantly outperforms RotatE at every embedding dimension.
The improvement is mainly from the use of logic rules, showing that our learned rules are indeed helpful.
Comparison of Optimization Algorithms.
RNNLogic uses an EM algorithm to optimize the rule generator.
In practice, the generator can also be optimized with REINFORCE (Williams, 1992) (see
App. F for details).
We empirically compare the two algorithms in the w/o emb.
case.
The results on
Kinship and UMLS are presented in Tab. 4.
We see EM consistently outperforms REINFORCE. [Figure] [Figure] [Figure]
Case Study of Generated Logic Rules.
Finally, we show some logic rules generated by RNNLogic on the FB15k-237 dataset in Tab. 5.
We can see that these logic rules are meaningful and diverse.
The ﬁrst rule is a subrelation rule.
The third and ﬁfth rules are two-hop compositional rules.
The rest of logic rules have even more complicated forms.
This case study shows that RNNLogic can indeed learn useful and diverse rules for reasoning.
For more generated logic rules, please refer to App.
E.
[Table]
C ONCLUSION
This paper studies learning logic rules for knowledge graph reasoning, and an approach called
RNNLogic is proposed.
RNNLogic treats a set of logic rules as a latent variable, and a rule generator as well as a reasoning predictor with logic rules are jointly learned.
We develop an EM-based algorithm for optimization.
Extensive expemriments prove the effectiveness of RNNLogic.
In the future, we plan to study generating more complicated logic rules rather than only compositional rules.
Besides, we plan to extend RNNLogic to other reasoning problems, such as question answering.
A CKNOWLEDGMENTS
This project is supported by the Natural Sciences and Engineering Research Council (NSERC)
Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft
Research and Mila, Samsung Electronics Co., Ldt., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund and a NRC Collaborative R&D Project (AI4D-CORE-06). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3583139727. A P ROOFS
This section presents the proofs of some propositions used in the optimization algorithm of RNNLogic.
Recall that in the E-step of the optimization algorithm, we aim to sample from the posterior distribution over rule sets.
However, directly sampling from the posterior distribution is intractable due to the intractable partition function.
Therefore, we introduce Proposition 1, which gives an approximation distribution with more tractable form for the posterior distribution.
With this approximation, sampling becomes much easier.
In Section A.1, we present the proof of Proposition 1.
In Section A.2, we show how to perform sampling based on the approximation of the posterior distribution.
A. P ROOF OF P ROPOSITION
Next, we prove proposition 1, which is used to approximate the true posterior probability in the
E-step of optimization.
We ﬁrst restate the proposition as follows:
Proposition Consider a data instance ( G , q , a ) with q = ( h, r, ?) and a = t .
For a set of rules ˆ z generated by the rule generator p θ , we can compute the following score H for each rule ∈
ˆ z :
[Equation]
where A is the set of all candidate answers discovered by rules in ˆ z , score w ( e | rule ) is the score that each rule contributes to entity e as deﬁned by Eq.
(4), RNN θ ( rule | r ) is the prior probability of rule computed by the generator.
Suppose s = max e ∈A | score w ( e ) | < 1 .
Then for a subset of rules z I ⊂ ˆ z with | z
I | =
K , the log-probability log p θ,w ( z I |G , q , a ) could be approximated as follows:
[Equation]
where const is a constant term that is independent from z I , γ ( z I ) =
log( K ! / (cid:81) rule ∈
ˆ z n rule !) , with K being the given size of set
z I and n rule being the number of times each rule appears in z I .
Proof: We ﬁrst rewrite the posterior probability as follows:
[Equation]
where const is a constant term which does not depend on the choice of z I , and RNN θ ( ·| r ) deﬁnes a probability distribution over all the composition-based logic rules.
The probability mass function of the above multinomial distribution Mu ( z I | K, RNN θ ( ·| r )) can be written as below:
[Equation]
where K is the size of set
z I and n rule is the number of times a rule appears in z I .
Letting γ ( z I ) =
log( K ! / (cid:81) rule ∈
ˆ z n rule !) , then the posterior probability can then be rewritten as:
[Equation]
The above term log (cid:80) e ∈A exp( score w ( e )) makes the posterior distribution hard to deal with, and thus we approximate it using Lemma 1, which we prove at the end of this section.
Lemma 1 Let e ∈
A be a ﬁnite set of entities, let | score w ( e ) | ≤ s < 1 , and let score w be a function from entities to real numbers.
Then the following inequalities hold:
[Equation]
Hence, using the lemma we can get the following upper bound of the posterior probability:
[Equation]
and also the following lower bound of the posterior probability: log p
[Equation]
[Equation]
G
[Equation]
q
[Equation]
a )
[Equation]
rule
[Equation]
s 4 ) , where const is a constant term which does not depend on z I .
By combining the lower and the upper bound, we get:
[Equation]
Thus, it only remains to prove Lemma 1 to complete the proof.
We use Theorem 1 from (Simic, 2008) as a starting point:
Theorem 1 Suppose that ˜ x
= { x i } ni =1 represents a ﬁnite sequence of real numbers belonging to a ﬁxed closed interval I =
[ a, b ] , a < b .
If f is a convex function on I , then we have that:
[Equation]
As ( − log) is convex and exp( score w ( e )) ∈
[exp( − s ) , exp( s )] , Theorem 1 gives us that:
[Equation] [Equation]
After some simpliﬁcation, we get:
[Equation]
where the last inequality is based on Taylor’s series log(1 + e x ) = log 2 + 12 x + 18 x 2 + O ( x 4 ) with | x | < 1 .
On the other hand, according to the well-known Jensen’s inequality
[Equation]
we have:
[Equation]
which implies:
[Equation]
By combining Eq.
(10) and Eq.
[Equation]
we obtain:
[Equation]
This completes the proof.
(cid:3) .
A. S AMPLING B ASED ON THE A PPROXIMATION OF THE T RUE P OSTERIOR
Based on Proposition 1, the log-posterior probability log p θ,w ( z I |G , q , a ) could be approximated by ( (cid:80) rule ∈
z I H ( rule )+ γ ( z I )+ const ) , with const being a term that does not depend on z
I .
This implies that we could construct a distribution q ( z I )
∝ exp( (cid:80) rule ∈
z I H ( rule ) + γ ( z I )) to approximate the true posterior, and draw samples from q as approximation to the real samples from the posterior.
It turns out that the distribution q ( z I ) is a multinomial distribution.
To see that, we rewrite q ( z I ) as:
[Equation]
where n rule is the number of times a rule appears in the set
z I , q r is a distribution over all the generated logic rules ˆ z with q r ( rule ) =
exp( H ( rule )) / (cid:80) rule (cid:48) ∈
ˆ z exp( H ( rule (cid:48) )) , Z and Z (cid:48) are normalization terms.
By summing over z I on both sides of the above equation, we obtain Z (cid:48)
= 1 , and hence: q ( z I ) =
Mu ( z I | K, q r ) .
To sample from such a multinomial distribution, we could simply sample K rules independently from the distribution q r , and form a sample ˆ
z I with these K rules.
In practice, we observe that the hard-assignment EM algorithm (Koller & Friedman, 2009) works better than the standard EM algorithm despite the reduced theoretical guarantees.
In the hardassignment EM algorithm, we need to draw a sample ˆ
z
I with the maximum posterior probability.
Based on the above approximation q ( z I ) of the true posterior distribution p θ,w ( z I |G , q , a ) , we could simply construct such a sample ˆ
z I with K rules which have the maximum probability under the distribution q r .
By deﬁnition, we have q r ( rule ) ∝ exp
( H ( rule )) , and hence drawing K rules with maximum probability under q r is equivalent to choosing K rules with the maximum H values.
B M ORE A NALYSIS OF THE EM A LGORITHM
In RNNLogic, we use an EM algorithm to optimize the rule generator.
In this section, we show why this EM algorithm is able to maximize the objective function of the rule generator.
Recall that for a ﬁxed reasoning predictor p w , we aim to update p θ to maximize the log-likelihood function log p w,θ ( a |G , q ) for each data instance ( G , q , a ) .
Directly optimizing log p w,θ ( a |G , q ) is difﬁcult due to the latent logic rules, and therefore we consider the following evidence lower bound of the log-likelihood function:
[Equation]
where q ( z I ) is a variational distribution, and the equation holds when q ( z I )
= p θ,w ( z I |G , q , a ) .
With this lower bound, we can optimize the log-likelihood function log p w,θ ( a |G , q ) with an E-step and an M-step.
In the E-step, we optimize q ( z I ) to maximize L ELBO ( q, p θ ) , which is equivalent to minimizing KL ( q ( z I )
||
p θ,w ( z I |G , q , a )) .
By doing so, we are able to tighten the lower bound.
Then in the M-step, we further optimize θ to maximize L ELBO ( q, p θ ) .
Next, we introduce the details.
E-
step.
In the E-step, our goal is to update q to minimize KL ( q ( z I )
||
p θ,w ( z I |G , q , a )) .
However, there are a huge number of possible logic rules, and hence optimizing q ( z I ) on every possible rule set
z
I is intractable.
To solve the problem, recall that we generate a set of logic rules
ˆ z when optimizing the reasoning predictor, and here we add a constraint to q based on ˆ z .
Speciﬁcally, we constrain the sample space of q ( z I ) to be all subsets of ˆ z with size being K , i.e., z I ⊂ ˆ z and | z I | =
K .
In other words, we require (cid:80) z
I ⊂ ˆ z , | z I | =
K q ( z I ) = 1 .
With such a constraint, we can further use propositionto construct the variational distribution q to approximate p θ,w ( z I |G , q , a ) , as what is described in the model section.
M-step.
In the M-step, our goal is to update p θ to maximize the lower bound L ELBO ( q, p θ ) .
To do that, we notice that there is an expectation operation with respect to q ( z I ) in L ELBO ( q, p θ ) .
By drawing a sample from q ( z I ) , L ELBO ( q, p θ ) can be estimated as follows:
[Equation]
where ˆ z I ∼ q ( z I ) is a sample drawn from the variational distribution.
By ignoring the terms which are irrelevant to p θ , we obtain the following objective function for θ :
[Equation]
which is the same as the objective function described in the model section.
As a result, by performing the E-step and the M-step described in the model section, we are able to update p θ to increase the lower bound L ELBO ( q, p θ ) , and thereby push up the log-likelihood function log p w,θ ( a |G , q ) .
Therefore, we see that the EM algorithm can indeed maximize log p w,θ ( a |G , q ) with respect to p θ .
C D ETAILS ABOUT P ARAMETERIZATION AND I MPLEMENTATION
Section 3.2 of the paper introduces the high-level idea of the reasoning predictor with logic rules and the rule generator.
Due to the limited space, some details of the models are not covered.
In this section, we explain the details of the reasoning predictor and the rule generator.
C. R EASONING P REDICTOR WITH L OGIC R ULES
We start with the reasoning predictor with logic rules.
Recall that for each query, our reasoning predictor leverages a set of logic rules z to give each candidate answer a score, which is further used to predict the correct answer from all candidates.
Speciﬁcally, let A denote the set of all the candidate answers discovered by logic rules in set z .
For each candidate answer e ∈ A , we deﬁne the following function score w to compute a score:
[Equation]
where P ( h, rule , e ) is the set of grounding paths which start at h and end at e following a rule (e.g.,
Alice friend −−−−→ Bob hobby −−−−→ Sing ).
ψ w ( rule ) and φ w ( path ) are scalar weights of each rule and path .
For the scalar weight ψ w ( rule ) of a rule , we initialize ψ w ( rule ) as follows:
[Equation]
where |P ( h, rule , t ) | is the number of relational paths starting from head entity h , following the relations in rule and ending at tail entity t .
The form is very similar to the deﬁnition of H values for logic rules, and the value can effectively measure the contribution of a rule to the correct answers.
We also try randomly initializing rule weights or initializing them as 0, which yield similar results.
For the scalar score φ w ( path ) of a path , we either ﬁx it to 1, or compute it by introducing entity and relation embeddings.
In the second case, we introduce an embedding for each entity and relation in the complex space.
Formally, the embedding of an entity e is denoted as x e , and the embedding of a relation r is denoted as x r .
For a grounding path path =
e 0 r 1 −→ e 1 r 2 −→ e 2 · · · e l − 1 r l −→ e l , we follow the idea in RotatE (Sun et al., 2019) and compute φ w ( path ) in the following way:
[Equation]
where σ ( x ) = 1 1+ e − x is the sigmoid function, d is a distance function between two complex vectors, δ is a hyperparameter, and ◦ is the Hadmard product in complex spaces, which could be viewed as a rotation operator.
Intuitively, for the embedding x e 0 of entity e 0 , we rotate x e 0 by using the rotation operators deﬁned by { r k } lk =1 , yielding ( x e 0 ◦ x r 1 ◦ x r 2 ◦ · · · ◦ x r l ) .
Then we compute the distance between the new embedding and the embedding x e l of entity e l , and further convert the distance to a value between 0 and 1 by using the sigmoid function and a hyperparameter δ .
C. R ULE G ENERATOR
This paper focuses on compositional rules, which have the abbreviation form r ← r 1 ∧ · · · ∧ r l and thus could be viewed a sequence of relations [ r , r 1 , r 2 · · · r l , r END ] , where r is the query relation or the head of the rule, { r i } li =1 are the body of the rule, and r END is a special relation indicating the end of the relation sequence.
We introduce a rule generator RNN θ parameterized with an LSTM (Hochreiter & Schmidhuber, 1997) to model such sequences.
Given the current relation sequence [ r , r 1 , r 2 · · · r i ] , RNN θ aims to generate the next relation r i +1 and meanwhile output the probability of r i +1 .
The detailed computational process towards the goal is summarized as follows:
• Initialize the hidden state of the RNN θ as follows
[Equation]
where v r is a parameter vector associated with the query relation or the head relation r , f is a linear transformation.
• Sequentially compute the hidden state at different positions by using the LSTM gate:
[Equation]
where v r t is a parameter vector associated with the relation r t , [ v r , v r t ] is the concatenation of v r and v r t , g is a linear transformation.
• Generate r t +1 and its probability based on h t
+1 and the following vector: softmax ( o ( h t +1 )) .
Suppose the set of relations is denoted as R .
We ﬁrst transform h t +1 to a |R| -dimensional vector by using a linear transformation o , and then apply softmax function to the |R| -dimensional vector to get the probability of each relation.
Finally, we generate r t +1 according to the probability vector.
D E XPERIMENTAL S ETUP D. D ATASET S TATISTICS
The statistics of datasets are summarised in Table 6.
[Table]
D. E XPERIMENTAL S ETUP OF RNNL OGIC
Next, we explain the detailed experimental setup of RNNLogic.
We try different conﬁgurations of hyperparameters on the validation set, and the optimal conﬁguration is then used for testing.
We report the optimal hyperparameter conﬁguration as below.
Data Preprocessing.
For each training triplet ( h, r , t ) , we add an inverse triplet ( t, r − 1 , h ) into the training set, yielding an augmented set of training triplets T .
To build a training instance from p data , we ﬁrst randomly sample a triplet ( h, r , t ) from T , and then form an instance as ( G =
T \ { ( h, r , t ) } , q = ( h, r , ?) , a = t ) .
Basically, we use the sampled triplet ( h, r , t ) to construct the query and answer, and use the rest of triplets in T to form the background knowledge graph G .
During testing, the background knowledge graph G is constructed by using all the triplets in T .
Reasoning Predictor.
For the reasoning predictor in with embedding cases, the embedding dimension is set to 500 for FB15k-237, 200 for WN18RR, 2000 for Kinship and 1000 for UMLS.
We pre-train these embeddings with RotatE (Sun et al., 2019).
The hyperparameter δ for computing φ w ( path ) in Equation (17) is set to 9 for FB15k-237, 6 for WN18RR, 0.25 for Kinship and 3 for UMLS.
We use the Adam (Kingma & Ba, 2014) optimizer with an initial learning rate being 5 × 10 − 5 , and we decrease the learning rate in a cosine shape.
Rule Generator.
For the rule generator, the maximum length of generated rules is set to 4 for
FB15k-237, 5 for WN18RR, and 3 for Kinship and UMLS.
These numbers are chosen according to model performace on the validation data.
The size of input and hidden states in RNN θ are set toand 256.
The learning rate is set to 1 × 10 − 3 and monotonically decreased in a cosine shape.
Beam search is used to generate rules with high probabilities, so that we focus on exploiting these logic rules which the rule generator is conﬁdent about.
Besides, we pre-train the rule generator by using sampled relational paths on the background knowledge graph formed with training triplets, which prevents the rule generator from exploring meaningless logic rules in the beginning of training.
EM Optimization.
During optimization, we sample 1000 rules from the rule generator for each data instance.
In the E-step, for each data instance, we identify 300 rules as high-quality logic rules.
Evaluation.
In testing, for each query q
=
( h, r, ?) , we use the rule generator p θ to generatelogic rules, and let the reasoning predictor p w use the generated logic rules to predict the answer a .
E C ASE S TUDY
We present more rules learned by RNNLogic on FB15k-237 dataset and UMLS dataset in Tableat the last page.
In this table, h r −→ t means triplet ( h, r , t ) and h r ←− t means triplet ( h, r − 1 , t ) , or equivalently ( t, r , h ) .
F C ONNECTION WITH REINFORCE
In terms of optimizing the rule generator, our EM algorithm has some connections with the REINFORCE algorithm (Williams, 1992).
Formally, for a training instance ( G , q , a ) , REINFORCE computes the derivative with respect to the parameters θ of the rule generator as follows:
[Equation]
where ˆ z is a sample from the rule generator, i.e., ˆ z ∼ p θ ( z | q ) .
R is a reward from the reasoning predictor based on the prediction result on the instance.
For example, we could treat the probability of the correct answer computed by the reasoning predictor as reward, i.e., R = p w ( a |G , q , ˆ z ) .
In contrast, our EM optimization algorithm optimizes the rule generator with the objective function deﬁned in Equation (2), yielding the derivative ∇ θ log p θ (ˆ z I | q ) .
Comparing the derivative to that in
REINFORCE (Eq. (18)), we see that EM only maximizes the log-probability for rules selected in the
E-step, while REINFORCE maximizes the log-probability for all generated rules weighted by the scalar reward R .
Hence the two approaches coincide if R is set to 1 for the selected rules from the approximate posterior and 0 otherwise.
In general, ﬁnding an effective reward function to provide feedback is nontrivial, and we empirically compare these two optimization algorithms in experiments.
[Table]
V Occurs in ←−−−−−− Y ← X Precedes −−−−−−→ U Occurs in ←−−−−−− V Degree of ←−−−−−− Y X Affects
←−−−−− Y ← X Result of −−−−−−→ U Occurs in −−−−−−→ V Precedes −−−−−−→
Y ← X Precedes ←−−−−−− U Produces −−−−−−→ V Occurs in ←−−−−−− Y ← X Prevents ←−−−−−− U Disrupts −−−−−−→
V Co-occurs with −−−−−−−−−−→ Y ← X Result of ←−−−−−−
U Complicates −−−−−−−−→ V Precedes −−−−−−→ Y ← X Assesses Effect of ←−−−−−−−−−−−−− U Method of −−−−−−→ V Complicates −−−−−−−−→
Y ← X Process of −−−−−−−→ U Interacts with −−−−−−−−−−→ V Causes −−−−→ Y ← X Assesses Effect of ←−−−−−−−−−−−−− U Result of ←−−−−−− V Precedes −−−−−−→ Y