Original: N EURAL
Revised: N EURAL

Original: C OMPOSITIONAL
Revised: C OMPOSITIONAL

Original: R ULE
Revised: R ULE

Original: L EARNING
Revised: L EARNING

Original: FOR
Revised: FOR

Original: K NOWLEDGE G RAPH R EASONING
Revised: K NOWLEDGE G RAPH R EASONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning logic rules is critical to improve reasoning in KGs.
Revised: Learning logic rules is critical to improve reasoning in KGs.

Original: This is due to their ability to provide logical interpretable explanations when used for predictions, as well as their ability to generalize to other tasks, domains, and data.
Revised: This is due to their ability to provide logical interpretable explanations when used for predictions, as well as their ability to generalize to other tasks, domains, and data.

Original: While recent methods have been proposed to learn logic rules, the majority of these methods are either restricted by their computational complexity and cannot handle the large search space of large-scale KGs or show poor generalization when exposed to data outside the training set.
Revised: While recent methods have been proposed to learn logic rules, the majority of these methods are either restricted by their computational complexity and cannot handle the large search space of large-scale KGs or show poor generalization when exposed to data outside the training set.

Original: In this paper we propose an end-to-end neural model for learning compositional logic rules called NCRL .
Revised: In this paper, we propose an end-to-end neural model for learning compositional logic rules called NCRL .

Original: NCRL detects the best compositional structure of a rule body and breaks it into small compositions in order to infer the rule head.
Revised: NCRL detects the best compositional structure of a rule body and breaks it into small compositions in order to infer the rule head.

Original: By recurrently merging compositions in the rule body with a recurrent attention unit, NCRL finally predicts a single rule head.
Revised: By recurrently merging compositions in the rule body with a recurrent attention unit, NCRL finally predicts a single rule head.

Original: Experimental results show that NCRL learns high-quality rules, as well as being generalizable.
Revised: Experimental results show that NCRL learns high-quality rules, as well as being generalizable.

Original: Specifically, we show that NCRL is scalable, efficient, and yields state-of-the-art results for link prediction on large-scale KGs.
Revised: Specifically, we show that NCRL is scalable, efficient, and yields state-of-the-art results for link prediction on large-scale KGs.

Original: Moreover, we test NCRL for systematic generalization by learning to reason on small-scale observed graphs and evaluating on larger unseen ones
Revised: Moreover, we test NCRL for systematic generalization by learning to reason on small-scale observed graphs and evaluating on larger unseen ones

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Knowledge Graphs (KGs) provide a structured representation of real-world facts (Ji et al., 2021), and they are remarkably useful in various applications (Graupmann et al., 2005;
Revised: Knowledge Graphs (KGs) provide a structured representation of real-world facts (Ji et al., 2021), and they are remarkably useful in various applications (Graupmann et al., 2005;

Original: Lukovnikov et al., 2017; Xiong et al., 2017; Yih et al., 2015).
Revised: Lukovnikov et al., 2017; Xiong et al., 2017; Yih et al., 2015).

Original: Since KGs are usually incomplete, knowledge graph reasoning is a crucial problem on KGs, where the goal is to infer the missing knowledge using the observed facts.
Revised: Since KGs are usually incomplete, knowledge graph reasoning is a crucial problem on KGs, where the goal is to infer the missing knowledge using the observed facts.

Original: This paper investigates how to learn logic rules for KG reasoning.
Revised: This paper investigates how to learn logic rules for KG reasoning.

Original: Learning logic rules is critical for reasoning tasks in KGs and has received recent attention.
Revised: Learning logic rules is critical for reasoning tasks in KGs and has received recent attention.

Original: This is due to their ability to: (1) provide interpretable explanations when used for prediction, and (2) generalize to new tasks, domains, and data (Qu et al., 2020;
Revised: This is due to their ability to: (1) provide interpretable explanations when used for prediction, and (2) generalize to new tasks, domains, and data (Qu et al., 2020;

Original: Lu et al., 2022; Cheng et al., 2022).
Revised: Lu et al., 2022; Cheng et al., 2022).

Original: For example, in
Revised: For example, in

Original: Figure 1, the learned rules can be used to infer new facts related to objects that are unobserved in the training stage.
Revised: Figure 1, the learned rules can be used to infer new facts related to objects that are unobserved in the training stage.

Original: Moreover, logic rules naturally have an interesting property - called compositionality : where the meaning of a whole logical expression is a function of the meanings of its parts and of the way they are combined (Hupkes et al., 2020).
Revised: Moreover, logic rules naturally have an interesting property - called compositionality : where the meaning of a whole logical expression is a function of the meanings of its parts and of the way they are combined (Hupkes et al., 2020).

Original: To concretely explain compositionality, let us consider the family relationships shown in Figure 2.
Revised: To concretely explain compositionality, let us consider the family relationships shown in Figure 2.

Original: In Fig. 2(a), we show that the rule (hasUncle ← hasMother ∧
Revised: In Fig. 2(a), we show that the rule (hasUncle ← hasMother ∧

Original: hasMother ∧ hasSon) forms a composition of smaller logical expressions, which can be expressed as a hierarchy, where predicates (i.e., relations) can be combined and replaced by another single predicate.
Revised: hasMother ∧ hasSon) forms a composition of smaller logical expressions, which can be expressed as a hierarchy, where predicates (i.e., relations) can be combined and replaced by another single predicate.

Original: For example, predicates hasMother and hasMother can be combined and replaced by predicate hasGrandma as shown in Figure 2(a).
Revised: For example, predicates hasMother and hasMother can be combined and replaced by predicate hasGrandma as shown in Figure 2(a).

Original: As such, by recursively combining predicates into a composition and reduce the composition into a single predicate, we can finally infer the rule head (i.e., hasUncle) from the rule body.
Revised: As such, by recursively combining predicates into a composition and reduce the composition into a single predicate, we can finally infer the rule head (i.e., hasUncle) from the rule body.

Original: While, there are various possible hierarchical trees to represent such rules, not all of them are valid given the observed relations in the KG.
Revised: While, there are various possible hierarchical trees to represent such rules, not all of them are valid given the observed relations in the KG.

Original: For example, in Figure 2(b), given a KG which only contains relations { hasMother, hasSon, hasGrandma, hasUncle } , it is possible to combine hasMother and hasSon first.
Revised: For example, in Figure 2(b), given a KG which only contains relations { hasMother, hasSon, hasGrandma, hasUncle } , it is possible to combine hasMother and hasSon first.

Original: However, there is no proper predicate to represent it in the KG.
Revised: However, there is no proper predicate to represent it in the KG.

Original: Therefore, learning a high-quality compositional structure for a given logical expression is critical for rule discovery, and it is the focus of our work.
Revised: Therefore, learning a high-quality compositional structure for a given logical expression is critical for rule discovery, and it is the focus of our work.

Original: a. Training Graph b. Test Graph

Original: [Figure]

Original: In this work, our objective is to learn rules that generalize to large-scale tasks and unseen graphs.
Revised: In this work, our objective is to learn rules that generalize to large-scale tasks and unseen graphs.

Original: Let us consider the example shown in Figure 1.
Revised: Let us consider the example shown in Figure 1.

Original: From the training KG, we can extract two underlying rules – rule 1 ⃝ :hasGrandma ( x, y )
Revised: From the training KG, we can extract two un- [Figure] [Figure] derlying rules – rule 1 ⃝ :

Original:  ← hasMother ( x, z ) ∧ hasMother ( z, y ) and rule 2 ⃝ : hasUncle ( x, y ) ← hasGrandma ( x, z ) ∧
Revised: hasGrandma ( x, y ) ← hasMother ( x, z ) ∧ hasMother ( z, y ) and rule 2 ⃝ : hasUncle ( x, y ) ← hasGrandma ( x, z ) ∧

Original: hasSon ( z, y ) .
Revised: hasSon ( z, y ) .

Original: We also observe that the necessary rule to infer the relation between Alice and Bob in the test KG is rule 3 ⃝ : hasUncle ( x, y ) ← hasMother ( x, z 1 ) ∧ hasMother ( z 1 , z 2 )
Revised: We also observe that the necessary rule to infer the relation between Alice and Bob in the test KG is rule 3 ⃝ : hasUncle ( x, y ) ← hasMother ( x, z 1 ) ∧ hasMother ( z 1 , z 2 )

Original: ∧ hasSon ( z 2 , y ) , which is not observed in the training graph.
Revised: ∧ hasSon ( z 2 , y ) , which is not observed in the training graph.

Original: However, using compositionality to combine rules 1 ⃝ and 2 ⃝ , we can successfully learn rule 3 ⃝ which is necessary for inferring the relation between Alice and Bob in the test KG.
Revised: However, using compositionality to combine rules 1 ⃝ and 2 ⃝ , we can successfully learn rule 3 ⃝ which is necessary for inferring the relation between Alice and Bob in the test KG.

Original: The successful prediction in the test KG shows the model’s ability for systematic generalization (i.e., learning to reason on smaller graphs and making predictions on unseen graphs) (Sinha et al., 2019).
Revised: The successful prediction in the test KG shows the model’s ability for systematic generalization (i.e., learning to reason on smaller graphs and making predictions on unseen graphs) (Sinha et al., 2019).

Original: [Figure]
Revised: [Figure]

Original: Although compositionality is crucial for learning logic rules, most of existing logic rule learning methods fail to exploit it.
Revised: Although compositionality is crucial for learning logic rules, most of existing logic rule learning methods fail to exploit it.

Original: In traditional AI, inductive Logic Programming (ILP) (Muggleton & De Raedt, 1994; Muggleton et al., 1990) is the most representative symbolic method.
Revised: In traditional AI, inductive Logic Programming (ILP) (Muggleton & De Raedt, 1994; Muggleton et al., 1990) is the most representative symbolic method.

Original: Given a collection of positive examples and negative examples, an ILP system aims to learn logic rules which are able to entail all the positive examples while excluding any of the negative examples.
Revised: Given a collection of positive examples and negative examples, an ILP system aims to learn logic rules which are able to entail all the positive examples while excluding any of the negative examples.

Original: However it is difficult for ILP to scale beyond small rule sets due to their restricted computational complexity to handle the large search space of compositional rules.
Revised: However it is difficult for ILP to scale beyond small rule sets due to their restricted computational complexity to handle the large search space of compositional rules.

Original: There are also some recent neural-symbolic methods that extend ILP, e.g., neural logic programming methods (Yang et al., 2017; Sadeghian et al., 2019) and principled probabilistic methods (Qu et al., 2020).
Revised: There are also some recent neural-symbolic methods that extend ILP, e.g., neural logic programming methods (Yang et al., 2017; Sadeghian et al., 2019) and principled probabilistic methods (Qu et al., 2020).

Original: Neural logic programming simultaneously learns logic rules and their weights in a differentiable way.
Revised: Neural logic programming simultaneously learns logic rules and their weights in a differentiable way.

Original: Alternatively, principled probabilistic methods separate rule generation and rule weight learning by introducing a rule generator and a reasoning predictor.
Revised: Alternatively, principled probabilistic methods separate rule generation and rule weight learning by introducing a rule generator and a reasoning predictor.

Original: However, most of these approaches are particularly designed for the KG completion task.
Revised: However, most of these approaches are particularly designed for the KG completion task.

Original: Moreover, since they require enumeration of rules given a maximum rule length T , the complexity of these methods grows exponentially as max rule length increases, which severely limits their systematic generalization capability.
Revised: Moreover, since they require enumeration of rules given a maximum rule length T , the complexity of these methods grows exponentially as max rule length increases, which severely limits their systematic generalization capability.

Original: To overcome these issues, several works such as conditional theorem provers (CTPs) (Minervini et al., 2020), recurrent relational reasoning (R5)
Revised: To overcome these issues, several works such as conditional theorem provers (CTPs) (Minervini et al., 2020), recurrent relational reasoning (R5)

Original: (Lu et al., 2022) focused on the model’s systematicity instead.
Revised: (Lu et al., 2022) focused on the model’s systematicity instead.

Original: CTPs learn an adaptive strategy for selecting subsets of rules to consider at each step of the reasoning via gradient-based optimization while R5 performs rule extraction and logical reasoning with deep reinforcement learning equipped with a dynamic rule memory.
Revised: CTPs learn an adaptive strategy for selecting subsets of rules to consider at each step of the reasoning via gradient-based optimization while R5 performs rule extraction and logical reasoning with deep reinforcement learning equipped with a dynamic rule memory.

Original: Despite their strong generalizability to larger unseen graphs beyond the training sets (Sinha et al., 2019), they cannot handle KG completion tasks for large-scale KGs due to their high computational complexity.
Revised: Despite their strong generalizability to larger unseen graphs beyond the training sets (Sinha et al., 2019), they cannot handle KG completion tasks for large-scale KGs due to their high computational complexity.

Original: In this paper, we propose an end-to-end neural model to learn compositional logic rules for KG reasoning.
Revised: In this paper, we propose an end-to-end neural model to learn compositional logic rules for KG reasoning.

Original: Our proposed NCRL method is scalable and yields state-of-the-art (SOTA) results for link prediction on large-scale KGs.
Revised: Our proposed NCRL method is scalable and yields state-of-the-art (SOTA) results for link prediction on large-scale KGs.

Original: NCRL shows strong systematic generalization when tested on larger unseen graphs beyond the training sets.
Revised: NCRL shows strong systematic generalization when tested on larger unseen graphs beyond the training sets.

Original: NCRL views a logic rule as a composition of predicates and learns a hierarchical tree to express the rule composition.
Revised: NCRL views a logic rule as a composition of predicates and learns a hierarchical tree to express the rule composition.

Original: More specifically, NCRL breaks the rule body into small atomic compositions in order to infer the rule head.
Revised: More specifically, NCRL breaks the rule body into small atomic compositions in order to infer the rule head.

Original: By recurrently merging compositions in the rule body with a recurrent attention unit, NCRL finally predicts a single rule head.
Revised: By recurrently merging compositions in the rule body with a recurrent attention unit, NCRL finally predicts a single rule head.

Original: The main contributions of this paper are summarized as follows:
Revised: The main contributions of this paper are summarized as follows:

Original:  • We formulate the rule learning problem from a new perspective and define the score of a logical rule based on the semantic consistency between rule body and rule head.
Revised: [Figure] • We formulate the rule learning problem from a new perspective and define the score of a logical rule based on the semantic consistency between rule body and rule head.

Original: • NCRL presents an end-to-end neural approach to exploit the compositionality of a logical rule in a recursive way to improve models’ systematic generalizability.
Revised: • NCRL presents an end-to-end neural approach to exploit the compositionality of a logical rule in a recursive way to improve models’ systematic generalizability.

Original: • NCRL is scalable and yields state-of-the-art (SOTA) results for link prediction on large-scale KGs, and demonstrates strong systematic generalization to larger unseen graphs beyond training sets.
Revised: • NCRL is scalable and yields state-of-the-art (SOTA) results for link prediction on large-scale KGs, and demonstrates strong systematic generalization to larger unseen graphs beyond training sets.

Original:  N OTATION & P ROBLEM D EFINITION
Revised:  N OTATION & P ROBLEM D EFINITION

Original: Knowledge Graph.
Revised: Knowledge Graph.

Original: A KG, denoted by G = { E, R, O } , consists of a set of entities E , a set of relations R and a set of observed facts O .
Revised: A KG, denoted by G = { E, R, O } , consists of a set of entities E , a set of relations R and a set of observed facts O .

Original: Each fact in O is represented by a triple ( e i , r k , e j ) .
Revised: Each fact in O is represented by a triple ( e i , r k , e j ) .

Original: Horn Rule.
Revised: Horn Rule.

Original: Horn rules, as a special case of first order logic rules, are composed of a body of conjunctive predicates (i.e., relations are called also predicates) and a single head predicate.
Revised: Horn rules, as a special case of first order logic rules, are composed of a body of conjunctive predicates (i.e., relations are called also predicates) and a single head predicate.

Original: In this paper, we are interested in mining chain-like compositional Horn rules 1 in the following form.
Revised: In this paper, we are interested in mining chain-like compositional Horn rules 1 in the following form.

Original: [Equation]
Revised: [Equation]

Original: (1) where s ( r h , r b ) ∈ [0 , 1] is the confidence score associated with the rule , and r h ( x, y ) is called rule head and r b 1 ( x, z 1 ) ∧ · · · ∧ r b n ( z n − 1 , y ) is called rule body .
Revised: where s ( r h , r b ) ∈ [0 , 1] is the confidence score associated with the rule , and r h ( x, y ) is called rule head and r b 1 ( x, z 1 ) ∧ · · · ∧ r b n ( z n − 1 , y ) is called rule body .

Original: Combining rule head and rule body, we denote a Horn rule as ( r h , r b ) where r b =
Revised: Combining rule head and rule body, we denote a Horn rule as ( r h , r b ) where r b =

Original: [ r b 1 , . .
Revised: [ r b 1 , . .

Original: r b n ] .
Revised: r b n ] .

Original: Logic Rule Learning.
Revised: Logic Rule Learning.

Original: Logic rule learning aims to learn a confidence score s ( r h , r b ) for each rule ( r h , r b ) in rule space to measure its plausibility.
Revised: Logic rule learning aims to learn a confidence score s ( r h , r b ) for each rule ( r h , r b ) in rule space to measure its plausibility.

Original: During rule extraction, top k rules with highest scores will be selected as the learned rules.
Revised: During rule extraction, top k rules with highest scores will be selected as the learned rules.

Original:  N EURAL C OMPOSITIONAL R ULE L EARNING (NCRL)
Revised:  N EURAL C OMPOSITIONAL R ULE L EARNING (NCRL

Original: In this section, we introduce our NCRL to learn compositional logic rules.
Revised: In this section, we introduce our NCRL to learn compositional logic rules.

Original: Instead of using the frequency of rule instances to measure the plausibility of logical rules, we define the score of a logical rule as the probability that the rule body can be replaced by the rule head based on their semantic consistency.
Revised: Instead of using the frequency of rule instances to measure the plausibility of logical rules, we define the score of a logical rule as the probability that the rule body can be replaced by the rule head based on their semantic consistency.

Revised: The semantic consistency between a rule body and a rule head means that the body implies the head with a high probability.

Original: An overview of NCRL is shown in Figure 3.
Revised: An overview of NCRL is shown in Figure 3.

Original: NCRL starts by sampling a set of paths from a given KG, and further splitting each path into short compositions using a sliding window.
Revised: NCRL starts by sampling a set of paths from a given KG, and further splitting each path into short compositions using a sliding window.

Original: Then, NCRL uses a reasoning agent to reason over all the compositions to select one composition.
Revised: Then, NCRL uses a reasoning agent to reason over all the compositions to select one composition.

Original: NCRL uses a recurrent attention unit to transform the selected composition into a single relation represented as a weighted combination of existing relations.
Revised: NCRL uses a recurrent attention unit to transform the selected composition into a single relation represented as a weighted combination of existing relations.

Original: By recurrently merging compositions in the path, NCRL finally predicts the rule head.
Revised: By recurrently merging compositions in the path, NCRL finally predicts the rule head.

Original: Algorithm 1 outlines the learning procedure of NCRL .
Revised: Algorithm 1 outlines the learning procedure of NCRL .

Original: A detailed example to illustrate Figure 3 is given in Appendix A.1.
Revised: A detailed example to illustrate Figure 3 is given in Appendix A.1.

Original: 3. L OGIC R ULE L EARNING W ITH R ECURRENT A TTENTION U NIT
Revised: 3. L OGIC R ULE L EARNING W ITH R ECURRENT A TTENTION U NIT

Original: As discussed in Section 1, while the rule body can be viewed as a sequence, it naturally exhibits a rich hierarchical structure.
Revised: As discussed in Section 1, while the rule body can be viewed as a sequence, it naturally exhibits a rich hierarchical structure.

Original: The semantics of the rule body is highly dependent on its hierarchical structure, which cannot be exploited by most of the existing rule learning methods.
Revised: The semantics of the rule body is highly dependent on its hierarchical structure, which cannot be exploited by most of the existing rule learning methods.

Original: To explicitly allow our model to capture the hierarchical nature of the rule body, we need to learn how the relations in the rule body are combined as well as the principle to reduce each composition in the hierarchical tree into a single predicate.
Revised: To explicitly allow our model to capture the hierarchical nature of the rule body, we need to learn how the relations in the rule body are combined as well as the principle to reduce each composition in the hierarchical tree into a single predicate.

Original: 3.1. H IERARCHICAL S TRUCTURE L EARNING
Revised: 3.1. H IERARCHICAL S TRUCTURE L EARNING

Original: Hierarchical structure of logic rules is learned in an iterative way.
Revised: Hierarchical structure of logic rules is learned in an iterative way.

Original: At each step, NCRL selects only one composition from the rule body and replaces the selected composition by another single predicate based on the recurrent attention unit to reduce the rule body.
Revised: At each step, NCRL selects only one composition from the rule body and replaces the selected composition by another single predicate based on the recurrent attention unit to reduce the rule body.

Original: Although rule body is hierarchical,[Figure] when operations are very local (i,e., leaf-level composition), a composition is strictly sequential.
Revised: Although rule body is hierarchical, when operations are very local (i,e., leaf-level composition), a composition is strictly sequential.

Original: To identify a composition from a sampled path, we use a sliding window with different lengths to decompose the sampled paths into compositions of different sizes.
Revised: To identify a composition from a sampled path, we use a sliding window with different lengths to decompose the sampled paths into compositions of different sizes.

Original: In our implementation, we vary the size of the sliding window among { 2 , 3 } .
Revised: In our implementation, we vary the size of the sliding window among { 2 , 3 } .

Original: Given a fixed window size s , sliding windows are generated by a size s window which slides through the rule body r b =
Revised: Given a fixed window size s , sliding windows are generated by a size s window which slides through the rule body r b =

Original: [ r b 1 , . .
Revised: [ r b 1 , . .

Original: r b n ] .
Revised: r b n ] .

Original: Sliding Window Encoder.
Revised: Sliding Window Encoder.

Original: When operations are over a local sliding window (i.e., composition), the relations within a sliding window should strictly follow a chain structure.
Revised: When operations are over a local sliding window (i.e., composition), the relations within a sliding window should strictly follow a chain structure.

Original: Therefore, we utilize a  RNN (Schuster & Paliwal, 1997)  to encode a sliding window.
Revised: Sequence models can be utilized to encode a sliding window. Considering the tradeoff between model complexity and performance, we chose RNN (Schuster & Paliwal, 1997) over other sequence models to encode the sequence.

Original: For example, taking i -th sliding window whose size is 2 (i.e., w i =
Revised: For example, taking i -th sliding window whose size is 2 (i.e., w i =

Original: [ r b i , r b i [Equation] 1 ] ) as the input, RNN outputs: 
Revised: [ r b i , r b i +1 ] ) as the input, RNN outputs: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where h i ∈ R
Revised: ) where h i ∈ R

Original: d is a hidden-state corresponding to predicate r b i in w i .
Revised: d is a hidden-state corresponding to predicate r b i in w i .

Original: The final hidden-state h
Revised: The final hidden-state h

Original: i +1 is used as the representation of sliding window w i .
Revised: i +1 is used as the representation of sliding window w i .

Original: Thus, we have w i
Revised: Thus, we have w i

Original: =
Revised: =

Original: h
Revised: h

Original: i +1 .
Revised: i +1 .

Original: Composition Selection.
Revised: Composition Selection.

Original: w i is useful to estimate how likely the relations in i -th window appear together.
Revised: w i is useful to estimate how likely the relations in i -th window appear together.

Original: If these relations always appear together, they have higher probability to form a meaningful composition.
Revised: If these relations always appear together, they have higher probability to form a meaningful composition.

Original: To incorporate this observation into our model, we select the sliding window by computing:
Revised: To incorporate this observation into our model, we select the sliding window by computing:

Original: [Equation]
Revised: [Equation]

Original: where f is a fully connected neural network.
Revised: where f is a fully connected neural network.

Original: It learns the probability of i -th window to be a meaningful composition from its representation w i .
Revised: It learns the probability of i -th window to be a meaningful composition from its representation w i .

Original: w i with highest µ i will be selected as the input to the recurrent attention unit.
Revised: w i with highest µ i will be selected as the input to the recurrent attention unit.

Original: 3.1. R ECURRENT A TTENTION U NIT
Revised: 3.1. R ECURRENT A TTENTION U NIT

Original: Note that rule induction following its underlying hierarchical structure is a recurrent process.
Revised: Note that rule induction following its underlying hierarchical structure is a recurrent process.

Original: Therefore, we propose a novel recurrent attention unit to recurrently reduce the selected composition into a single predicate until it outputs a final relation.
Revised: Therefore, we propose a novel recurrent attention unit to recurrently reduce the selected composition into a single predicate until it outputs a final relation.

Original: Attention-based Induction.
Revised: Attention-based Induction.

Original: The goal of a recurrent attention unit is to reduce the selected composition into a single predicate, which can be modeled as matching the composition with another single predicate based on their semantic consistency.
Revised: The goal of a recurrent attention unit is to reduce the selected composition into a single predicate, which can be modeled as matching the composition with another single predicate based on their semantic consistency.

Original: Since attention mechanisms yield impressive results in Transformer models by capturing the semantic correlation between every pair of tokens in natural language sentence (Vaswani et al., 2017), we propose to utilize attention to reduce the selected composition w i .
Revised: Since attention mechanisms yield impressive results in Transformer models by capturing the semantic correlation between every pair of tokens in natural language sentence (Vaswani et al., 2017), we propose to utilize attention to reduce the selected composition w i .

Original: Note that we may not always find an existing relation to replace the selected composition.
Revised: Note that we may not always find an existing relation to replace the selected composition.

Original: For example, given the composition [ hasBrother , hasWife ] , none of the existing relations can be used to represent it.
Revised: For example, given the composition [ hasBrother , hasWife ] , none of the existing relations can be used to represent it.

Original: As such, in order to accommodate unseen relations, we incorporate a “null” predicate into potential rule heads and denote it as r 0 .
Revised: As such, in order to accommodate unseen relations, we incorporate a “null” predicate into potential rule heads and denote it as r 0 .

Revised: The embedding corresponding to r 0 is set as the representation of the selected composition w i .

Revised: In this way, when there is no direct link closing a sampled path (which means we do not have the ground truth about the rule head), we use the representation of the selected composition to represent itself rather than replace it with an existing relation.

Original: Let H ∈ R  | R | +1 × 
Revised: Let H ∈ R ( | R | +1) × d , where H 0 =

Original: d be the matrix of the concatenation of all head relations.
Revised: w i be the matrix of the concatenation of all head relations.

Original: By taking w i as a query and H as the content, the scaled dot-product attention θ can be computed to estimate the semantic consistency between the selected composition and its potential heads:
Revised: By taking w i as a query and H as the content, the scaled dot-product attention θ can be computed to estimate the semantic consistency between the selected composition and its potential heads:

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K ∈ R
Revised: where W Q , W K ∈ R

Original: d × d are learnable parameters that project the inputs into the space of query and key.
Revised: d × d are learnable parameters that project the inputs into the space of query and key.

Original: θ ∈ R | R | +1 is the learned attention, in which θ i measures p ( r j | w i ) - the probability that the selected composition can be replaced by the predicate r j based on their semantic consistency.
Revised: θ ∈ R | R | +1 is the learned attention, in which θ i measures p ( r j | w i ) - the probability that the selected composition can be replaced by the predicate r j based on their semantic consistency.

Original: Given θ , we are able to compute a new representation for the selected composition as a weighted combination of all head relations  each weighted by its attention weight.
Revised: Given θ , we are able to compute a new representation for the selected composition as a weighted combination of all head relations (i.e., values) each weighted by its attention weight.

Original: [Equation]
Revised: [Equation]

Original:  where (cid:99) w i ∈
Revised: 5) where (cid:99) w i ∈

Original: R
Revised: R

Original: d is the new representation of the selected composition.
Revised: d is the new representation of the selected composition.

Original: We project the key and value to the same space by requiring W V = W K .
Revised: We project the key and value to the same space by requiring W V = W K because the keys and the values are both embedding of relations in KG.

Original: As shown in Figure 3, we can reduce the long rule body [ r b 1 , r b 2 . .
Revised: As shown in Figure 3, we can reduce the long rule body [ r b 1 , r b 2 . .

Original: . , r b n ] by recursively applying the attention unit to replace its composition ( r b i , r b i +1 ) with a single predicate.
Revised: . , r b n ] by recursively applying the attention unit to replace its composition ( r b i , r b i +1 ) with a single predicate.

Original: In the final step of the prediction, the attention θ computed following Eq.
Revised: In the final step of the prediction, the attention θ computed following Eq.

Original: 4 collects the probability that the rule body can be replaced by each of the head relations.
Revised: 4 collects the probability that the rule body can be replaced by each of the head relations.

Original: 3. T RAINING AND R ULE E XTRACTION
Revised: 3. T RAINING AND R ULE E XTRACTION

Original: NCRL is trained in an end-to-end fashion.
Revised: NCRL is trained in an end-to-end fashion.

Original: It starts by sampling paths from an input KG, and predicts the relation which directly closes the sampled paths based on learned rules.
Revised: It starts by sampling paths from an input KG, and predicts the relation which directly closes the sampled paths based on learned rules.

Original: Path Sampling.
Revised: Path Sampling.

Original: We utilize a random walk (Spitzer, 2013) sampler to sample paths that connect two entities from the KG.
Revised: We utilize a random walk (Spitzer, 2013) sampler to sample paths that connect two entities from the KG.

Original: Formally, given a source entity x 0 , we simulate a random walk of max length n .
Revised: Formally, given a source entity x 0 , we simulate a random walk of max length n .

Original: Let x i denote the i -th node in the walk, which are generated by the following distribution:
Revised: Let x i denote the i -th node in the walk, which are generated by the following distribution:

Original: [Equation]
Revised: [Equation]

Original: where |N ( e j ) | is the neighborhood size of entity e j .
Revised: where |N ( e j ) | is the neighborhood size of entity e j .

Original: Different from random walk, each time after we sample the next entity x
Revised: Different from random walk, each time after we sample the next entity x

Original: i , we add all the edges which can directly connect x 0 and x i in KG.
Revised: i , we add all the edges which can directly connect x 0 and x i in KG.

Original: We denote the path connecting two nodes x 0 and x n as p , where p =
Revised: We denote the path connecting two nodes x 0 and x n as p , where p =

Original: [ r b 1 , . .
Revised: [ r b 1 , . .

Original: . , r b n ] , indicating x 0 r 1 −→ . . .
Revised: . , r b n ] , indicating x 0 r 1 −→ . . .

Original: r n −→ x n .
Revised: r n −→ x n .

Original: We also denote the relation that directly connects x 0 and x n as r h .
Revised: We also denote the relation that directly connects x 0 and x n as r h .

Original: If none of the relations directly connects the x 0 and x n , we set r h as “null”.
Revised: If none of the relations directly connects the x 0 and x n , we set r h as “null”.

Original: Algorithm Input: Observed triples in KG O Output: Relation embeddings whose −
Revised: We control the ratio of non-closed paths during path sampling to ensure a majority of sampled paths are associated with an observed head relation.

Original: Objective Function.
Revised: Objective Function.

Original: Our goal is to maximize the likelihood of the observed relation r h , which directly closes the sampled path p .
Revised: Our goal is to maximize the likelihood of the observed relation r h , which directly closes the sampled path p .

Original: The attention θ collects the predicted probability for p being closed by each of the head relations.
Revised: The attention θ collects the predicted probability for p being closed by each of the head relations.

Original: We formulate the objective using the cross-entropy loss as:
Revised: We formulate the objective using the cross-entropy loss as:

Original: [Equation]
Revised: [Equation]

Original: where P denotes a set of sampled paths from a given KG, y r h ∈ { 0 , 1 } | R | +1 is the one-hot encoded vector such that only the r h -th entry is 1, and θ p ∈ R | R | +1 is the learned attention for the sampled path p .
Revised: where P denotes a set of sampled paths from a given KG, y r h ∈ { 0 , 1 } | R | +1 is the one-hot encoded vector such that only the r h -th entry is 1, and θ p ∈ R | R | +1 is the learned attention for the sampled path p .

Original: In particular, θ p 0 represents the probability that the sampled path cannot be closed by any existing relations in KG.
Revised: In particular, θ p 0 represents the probability that the sampled path cannot be closed by any existing relations in KG.

Original: Rule Extraction.
Revised: Rule Extraction.

Original: To recover logical rules, we calculate the score s ( r h , r b ) for each rule ( r h , r b ) in rule space based on the learned model.
Revised: To recover logical rules, we calculate the score s ( r h , r b ) for each rule ( r h , r b ) in rule space based on the learned model.

Original: Given a candidate rule ( r h , r b ) , we reduce the rule body r b into a single head r h by recursively merge compositions in path r b .
Revised: Given a candidate rule ( r h , r b ) , we reduce the rule body r b into a single head r h by recursively merge compositions in path r b .

Original: At the final step of the prediction, we learn the attention θ =
Revised: At the final step of the prediction, we learn the attention θ =

Original: [ θ 0 , . .
Revised: [ θ 0 , . .

Original: . , θ | R | ] , where θ k is the score of rule ( r k , r b ) .
Revised: . , θ | R | ] , where θ k is the score of rule ( r k , r b ) .

Original: Top k rules with highest score will be selected as learned rules.
Revised: Top k rules with highest score will be selected as learned rules.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Logic rules are valuable for various downstream tasks, such as (1) KG completion task , which aims to infer the missing entity given the query ( h, r, ?) or (? , r, t ) ; (2) A more challenging inductive relational reasoning task , which tests the systematic generalization capability of the model by inferring the missing relation between two entities (i.e., ( h, ? , t ) ) with more hops than the training data.
Revised: Logic rules are valuable for various downstream tasks, such as (1) KG completion task , which aims to infer the missing entity given the query ( h, r, ?) or (? , r, t ) ; (2) A more challenging inductive relational reasoning task , which tests the systematic generalization capability of the model by inferring the missing relation between two entities (i.e., ( h, ? , t ) ) with more hops than the training data.

Original: A majority of existing methods can handle only one of these two tasks (e.g., RNNLogic is designed for KG completion task while R5 is designed for inductive relational reasoning task).
Revised: A majority of existing methods can handle only one of these two tasks (e.g., RNNLogic is designed for KG completion task while R5 is designed for inductive relational reasoning task).

Original: In this section, we show that our method is superior to existing SOTA algorithms on both tasks.
Revised: In this section, we show that our method is superior to existing SOTA algorithms on both tasks.

Original: In addition, we also empirically assess the interpretability of the learned rules.
Revised: In addition, we also empirically assess the interpretability of the learned rules.

Original: 4. K NOWLEDGE G RAPH C OMPLETION
Revised: 4. K NOWLEDGE G RAPH C OMPLETION

Original: KG completion is a classic task widely used by logic rule learning methods such as Neural-LP
Revised: KG completion is a classic task widely used by logic rule learning methods such as Neural-LP

Original: (Yang et al., 2017), DRUM (Sadeghian et al., 2019) and RNNLogic (Qu et al., 2020) to evaluate the quality of learned rules.
Revised: (Yang et al., 2017), DRUM (Sadeghian et al., 2019) and RNNLogic (Qu et al., 2020) to evaluate the quality of learned rules.

Original: An existing algorithm called forward chaining (Salvat & Mugnier, 1996) can used to predict missing facts from logic rules.
Revised: An existing algorithm called forward chaining (Salvat & Mugnier, 1996) can used to predict missing facts from logic rules.

Original: Datasets.
Revised: Datasets.

Original: We use six widely used benchmark datasets to evaluate our NCRL in comparison to stateof-the-art methods from knowledge graph embedding and rule learning methods.
Revised: We use six widely used benchmark datasets to evaluate our NCRL in comparison to stateof-the-art methods from knowledge graph embedding and rule learning methods.

Original: Specifically, we use the Family (Hinton et al., 1986), UMLS (Kok & Domingos, 2007), Kinship (Kok & Domingos, 2007), WN18RR
Revised: Specifically, we use the Family (Hinton et al., 1986), UMLS (Kok & Domingos, 2007), Kinship (Kok & Domingos, 2007), WN18RR

Original: (Dettmers et al., 2018), FB15K-

Revised: (Dettmers et al., 2018), FB15K-

Original: (Toutanova & Chen, 2015), YAGO310 (Suchanek et al., 2007) datasets.
Revised: (Toutanova & Chen, 2015), YAGO310 (Suchanek et al., 2007) datasets.

Original: The statistics of the datasets are given in Appendix A.3.1.
Revised: The statistics of the datasets are given in Appendix A.3.1.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We mask the head or tail entity of each test triple, and require each method to predict the masked entity.
Revised: We mask the head or tail entity of each test triple, and require each method to predict the masked entity.

Original: During evaluation, we use the filtered setting (Bordes et al., 2013) and three evaluation metrics, i.e., Hit@1, Hit@10 and MRR.
Revised: During evaluation, we use the filtered setting (Bordes et al., 2013) and three evaluation metrics, i.e., Hit@1, Hit@10 and MRR.

Original: Comparing with Other Methods.
Revised: Comparing with Other Methods.

Original: We evaluate our method against SOTA methods, including: (1) traditional KG embedding (KGE) methods (e.g., TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ConvE (Dettmers et al., 2018), ComplEx (Trouillon et al., 2016) and RotatE (Sun et al., 2019)); (2) logic rule learning methods (e.g., Neural-LP
Revised: We evaluate our method against SOTA methods, including: (1) traditional KG embedding (KGE) methods (e.g., TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ConvE (Dettmers et al., 2018), ComplEx (Trouillon et al., 2016) and RotatE (Sun et al., 2019)); (2) logic rule learning methods (e.g., Neural-LP

Original: (Yang et al., 2017), DRUM (Sadeghian et al., 2019), RNNLogic (Qu et al., 2020) and RLogic (Cheng et al., 2022)).
Revised: (Yang et al., 2017), DRUM (Sadeghian et al., 2019), RNNLogic (Qu et al., 2020) and RLogic (Cheng et al., 2022)).

Original: The systematic generalizable methods (e.g., CTPs and R5) cannot handle KG completion tasks due to their high complexity.
Revised: The systematic generalizable methods (e.g., CTPs and R5) cannot handle KG completion tasks due to their high complexity.

Original: Results.
Revised: Results.

Original: The comparison results are presented in Table 1.
Revised: The comparison results are presented in Table 1.

Original: We observe that: (1) Although NCRL is not designed for KG completion task, compared with traditional KGE models, it achieves comparable result on all datasets, especially on Family, UMLS and WN18RR datasets; (2) NCRL consistently outperforms all other rule learning methods with significant performance gain in most cases.
Revised: We observe that: (1) Although NCRL is not designed for KG completion task, compared with traditional KGE models, it achieves comparable result on all datasets, especially on Family, UMLS and WN18RR datasets; (2) NCRL consistently outperforms all other rule learning methods with significant performance gain in most cases.

Original: 4.1. A BLATION S TUDY
Revised: 4.1. A BLATION S TUDY

Original: Performance w.r.t.
Revised: Performance w.r.t.

Original: Data Sparsity.
Revised: Data Sparsity.

Original: We construct sparse KG by randomly removing θ triples from original dataset.
Revised: We construct sparse KG by randomly removing a triples from original dataset.

Original: Following this approach, we vary the sparsity ratio θ among { 0 . 33 , 0 . 66 , 1 } and report performance on different methods over KG completion task on Kinship dataset.
Revised: Following this approach, we vary the sparsity ratio a among { 0 . 33 , 0 . 66 , 1 } and report performance on different methods over KG completion task on Kinship dataset.

Original: As presented in Fig.
Revised: As presented in Fig.

Original: 4, the performance of NCRL does not vary a lot with different sparsity ratio θ , which is appealing in practice.
Revised: 4, the performance of NCRL does not vary a lot with different sparsity ratio a , which is appealing in practice.

Original: More analysis on other datasets are given in Appendix A.3.2.
Revised: More analysis on other datasets are given in Appendix A.3.2.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Table]
Revised: [Figure] [Figure]

Original: KG completion performance w.r.t.
Revised: KG completion performance w.r.t.

Original: the Number of Learned Rules.
Revised: the Number of Learned Rules.

Original: To investigate how the number of learned rules affect the KG completion performance, we generate k rules with highest qualities per query relation
Revised: To investigate how the number of learned rules affect the KG completion performance [Figure] we generate k rules with highest qualities [Table]

Revised: per query relation

Original: [Table]
Revised: [Table]

Original: and use them to predict missing links.
Revised: and use them to predict missing links.

Original: We vary k among { 10 , 20 , 40 , 60 , 80 , 100 } .
Revised: We vary k among { 10 , 20 , 40 , 60 , 80 , 100 } .

Original: The results on Kinship are given in Fig.
Revised: The results on Kinship are given in Fig.

Original: We observed that even with only 10 rules per relation,
Revised: We observed that even with only 10 rules per relation,

Original: NCRL still givens competitive results.
Revised: NCRL still givens competitive results.

Original: More analysis on other datasets are given in Appendix A.3.2.
Revised: More analysis on other datasets are given in Appendix A.3.2.

Original: Performance w.r.t.
Revised: Performance w.r.t.

Original: Embedding Dimension.
Revised: Embedding Dimension.

Original: To investigate how the dimension of relation embeddings affect the KG completion performance, we vary the dimension of relation embeddings among { 10 , 100 , 200 , 500 , 1000 , 2000 } and present the results on Kinship in Fig.
Revised: To investigate how the dimension of relation embeddings affect the KG completion performance, we vary the dimension of relation embeddings among { 10 , 100 , 200 , 500 , 1000 , 2000 } and present the results on Kinship in Fig.

Original: 6, comparing against
Revised: 6, comparing against

Original: RLogic (Cheng et al., 2022).
Revised: RLogic (Cheng et al., 2022).

Original: We see that embedding dimension has a significant impact on KG completion performance.
Revised: We see that embedding dimension has a significant impact on KG completion performance.

Original: The best performance is achieved at d = 1000 .
Revised: The best performance is achieved at d = 1000 .

Original: 4. T RAINING E FFICIENCY
Revised: 4. T RAINING E FFICIENCY

Original: [Table]
Revised: [Table]

Original: To demonstrate the scalability of NCRL , we give the training time of NCRL against other logic rule learning methods on three benchmark datasets in Table 2.
Revised: To demonstrate the scalability of NCRL , we give the training time of NCRL against other logic rule learning methods on three benchmark datasets in Table 2.

Original: We observe that: (1) Neural-LP and DRUM do not perform well in terms of efficiency as they apply a sequence of large matrix multiplications for logic reasoning.
Revised: We observe that: (1) Neural-LP and DRUM do not perform well in terms of efficiency as they apply a sequence of large matrix multiplications for logic reasoning.

Original: They cannot handle YAGO3-10 dataset due to the memory issue; (2)
Revised: They cannot handle YAGO3-10 dataset due to the memory issue; (2)

Original: It is also challenging for RNNLogic to scale to large scale KGs as it relies on all ground rules to evaluate the generated rules in each iteration.
Revised: It is also challenging for RNNLogic to scale to large scale KGs as it relies on all ground rules to evaluate the generated rules in each iteration.

Original: It is difficult for it to handle KG with hundreds of relations (e.g., FB15K-237) nor KG with million entities (e.g., YAGO3-10); (3) our NCRL is on average 100 x faster than state-of-the-art baseline methods.
Revised: It is difficult for it to handle KG with hundreds of relations (e.g., FB15K-237) nor KG with million entities (e.g., YAGO3-10); (3) our NCRL is on average 100 x faster than state-of-the-art baseline methods.

Original: 4. S YSTEMATIC G ENERALIZATION
Revised: 4. S YSTEMATIC G ENERALIZATION

Original: We test NCRL for systematic generalization to demonstrate the ability of NCRL to perform reasoning over graphs with more hops than the training data, where the model is trained on smaller graphs and tested on larger unseen ones.
Revised: We test NCRL for systematic generalization to demonstrate the ability of NCRL to perform reasoning over graphs with more hops than the training data, where the model is trained on smaller graphs and tested on larger unseen ones.

Original: The goal of this experiment is to infer the relation between nodel pair query.
Revised: The goal of this experiment is to infer the relation between node pair query.

Original: We use two benchmark datasets: (1) CLUTRR (Sinha et al., 2019) which is a dataset for inductive relational reasoning over family relations, and (2) GraphLog (Sinha et al., 2020) is a benchmark suite for rule induction and it consists of logical worlds and each world contains graphs 
Revised: We use two benchmark datasets: (1) CLUTRR (Sinha et al., 2019) which is a dataset for inductive relational reasoning over family relations, and (2) GraphLog (Sinha et al., 2020) is a benchmark suite for rule induction and it consists of logical worlds and each world contains graphs generated under different set of rules.

Original: [Table]
Revised: Note that most of existing rule learning methods lack systematic generalization. CTPs (Minervini et al., 2020), R5 (Lu et al., 2022) and RLogic (Cheng et al.,

Original: [Table]
Revised: [Table]

Original: isLocatedIn ( x, y ) ← isLocatedIn ( x, z ) ∧ isLocatedIn ( z, y ) isLocatedIn ( x, y ) ← hasAcademicAdvisor ( x, z 1 ) ∧ isLocatedIn ( z 1 , z 2 ) ∧ isLocatedIn ( z 2 , y ) isAffiliatedTo ( x, y ) ← isKnownFor ( x, z ) ∧ isAffiliatedTo ( z, y ) isAffiliatedTo ( x, y ) ← isKnownFor ( x, z 1 ) isAffiliatedTo ( z 1 , z 2 ) ∧ isLeaderOf ( z 2 , y )
Revised: [Table]

Original: playsFor ( x, y ) ← isKnownFor ( x, z ) ∧ isAffiliatedTo ( z, y ) playsFor ( x, y ) ← isKnownFor ( x, z 1 ) playsFor ( z 1 , z 2 ) ∧ owns ( z 2 , y ) influences ( x, y ) ← isPoliticianOf ( x, z ) ∧ influences ( z, y ) influences ( x, y ) ← isPoliticianOf ( x, z 1 ) ∧ influences ( z 1 , z 2 ) ∧ influences ( z 2 , y ) generated under different set of rules. Note that most of existing rule learning methods lack systematic generalization.
Revised: [Table]

Original: CTPs (Minervini et al., 2020), R5 (Lu et al., 2022) and RLogic (Cheng et al., 2022) are the only comparable rule learning methods for this task.
Revised: 2022) are the only comparable rule learning methods for this task.

Original: The detailed statistics and the description of the datasets is summarized in Appendix A.4.1.
Revised: The detailed statistics and the description of the datasets is summarized in Appendix A.4. [Table]

Original: Systematic Generalization on CLUTRR.
Revised: Systematic Generalization on CLUTRR.

Original: Table 3 shows the results of NCRL against SOTA algorithms.
Revised: Table 3 shows the results of NCRL against SOTA algorithms.

Original: The detailed information about the SOTA algorithms are given in Appendix A.4.2.
Revised: The detailed information about the SOTA algorithms are given in Appendix A.4.2.

Original: We observe that the performances of sequential models and embedding-based models drop severely when the path length grows longer while NCRL still predicts successfully on longer paths without significant performance degradation.
Revised: We observe that the performances of sequential models and embedding-based models drop severely when the path length grows longer while NCRL still predicts successfully on longer paths without significant performance degradation.

Original: Comparing with systematic generalizable rule learning methods,
Revised: Comparing with systematic generalizable rule learning methods,

Original: NCRL has better generalization capability than CTPs especially when the paths grows longer.
Revised: NCRL has better generalization capability than CTPs especially when the paths grows longer.

Original: Even though R5 gives invincible results over CLUTRR dataset, NCRL shows comparable performance.
Revised: Even though R5 gives invincible results over CLUTRR dataset, NCRL shows comparable performance.

Original: Systematic Generalization on GraphLog.
Revised: Systematic Generalization on GraphLog.

Original: Table 4 shows the results on 4 selected worlds.
Revised: Table 4 shows the results on 4 selected worlds.

Original: We observed that NCRL consistently outperforms other rule learning baselines over all 4 worlds.
Revised: We observed that NCRL consistently outperforms other rule learning baselines over all 4 worlds.

Original: 4. C ASE S TUDY OF G ENERATED L OGIC R ULES
Revised: 4. C ASE S TUDY OF G ENERATED L OGIC R ULES

Original: Finally, we show a case study of logic rules that are generated by NCRL on the YAGO3-10 dataset in Table 5.
Revised: Finally, we show a case study of logic rules that are generated by NCRL on the YAGO3-10 dataset in Table 5.

Original: We can see that these logic rules are meaningful and diverse.
Revised: We can see that these logic rules are meaningful and diverse.

Original: Two rules with different lengths are presented for each head predicate.
Revised: Two rules with different lengths are presented for each head predicate.

Original: We highlight the composition and predicate which share the same semantic meaning with boldface.
Revised: We highlight the composition and predicate which share the same semantic meaning with boldface.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Inductive Logic Programming.
Revised: Inductive Logic Programming.

Original: Mining Horn clauses has been extensively studied in the Inductive
Revised: Mining Horn clauses has been extensively studied in the Inductive

Original: Logic Programming (ILP) (Muggleton & De Raedt, 1994; Muggleton et al., 1990; Muggleton, 1992;
Revised: Logic Programming (ILP) (Muggleton & De Raedt, 1994; Muggleton et al., 1990; Muggleton, 1992;

Original: Nienhuys-Cheng & De Wolf, 1997; Quinlan, 1990) .
Revised: Nienhuys-Cheng & De Wolf, 1997; Quinlan, 1990) .

Original: Given a set of positive examples, as well as a set of negative examples, an ILP system aims to learn logic rules which are able to entail all the positive examples while exclude any of the negative examples.
Revised: Given a set of positive examples, as well as a set of negative examples, an ILP system aims to learn logic rules which are able to entail all the positive examples while exclude any of the negative examples.

Original: Although ILP has shown its power in many areas (Tsunoyama et al., 2008; Zelle & Mooney, 1993), scalability is a central challenge for ILP methods as they involves several steps that are NP-hard.
Revised: Although ILP has shown its power in many areas (Tsunoyama et al., 2008; Zelle & Mooney, 1993), scalability is a central challenge for ILP methods as they involves several steps that are NP-hard.

Revised: Recently, several differentiable ILP methods such as Neural Theorem Provers (Rockt¨aschel & Riedel, 2017; Campero et al., 2018) are proposed to enable a continuous relaxation of the logical reasoning process via gradient descent.

Revised: Different from our method, they require pre-defined hand-designed and task-specific templates to narrow down the rule space.

Revised: Recently, a neural-symbolic approach was proposed for hierarchical rule induction (HRI) (Glanois et al., 2022).

Revised: HRI extends logical rule induction (Campero et al., 2018) to a hierarchical structure with a pre-defined set of meta-rules templates, and was shown to be able to tackle many of the classical inductive logic programming (ILP) benchmark tasks.

Revised: Unlike our approach, HRI requires a pre-defined set of meta-rule templates, which is unclear how it could generalize to tasks beyond classical ILP and it remains as one of their major future work directions to extend the meta-rule templates to broaden expressivity to other tasks beyond ILP.

Original: Neural-Symbolic Methods.
Revised: Neural-Symbolic Methods.

Original: Very recently, several methods are proposed to extend the idea of
Revised: Very recently, several methods are proposed to extend the idea of

Original: ILP by simultaneously learning logic rules and the weights in a differentiable way.
Revised: ILP by simultaneously learning logic rules and the weights in a differentiable way.

Original: Most of them are based on neural logic programming.
Revised: Most of them are based on neural logic programming.

Original: For example, Neural-LP (Yang et al., 2017) enables logical reasoning via sequences of differentiable tensor multiplication.
Revised: For example, Neural-LP (Yang et al., 2017) enables logical reasoning via sequences of differentiable tensor multiplication.

Original: An neural controller system based on attention is used to learn the score of a specific logic.
Revised: An neural controller system based on attention is used to learn the score of a specific logic.

Original: However, Neural-LP could learn higher score of a meaningless rule because it shares an atom with a useful rule.
Revised: However, Neural-LP could learn higher score of a meaningless rule because it shares an atom with a useful rule.

Original: To address this problem, RNNs are utilized in DRUM (Sadeghian et al., 2019) to prune the potential incorrect rule bodies.
Revised: To address this problem, RNNs are utilized in DRUM (Sadeghian et al., 2019) to prune the potential incorrect rule bodies.

Original: In addition,
Revised: In addition,

Original: Neural-LP can learn only chain-like Horn rules while NLIL (Yang & Song, 2019) extends NeuralLP to learn Horn rules in more general form.
Revised: Neural-LP can learn only chain-like Horn rules while NLIL (Yang & Song, 2019) extends NeuralLP to learn Horn rules in more general form.

Original: Because neural logic programming approaches involve large matrix multiplication and simultaneously learn logic rules and their weights, which is nontrivial in terms of optimization, they cannot handle large KGs, such as YAGO3-10.
Revised: Because neural logic programming approaches involve large matrix multiplication and simultaneously learn logic rules and their weights, which is nontrivial in terms of optimization, they cannot handle large KGs, such as YAGO3-10.

Original: To address this issue,
Revised: To address this issue,

Original: RNNLogic (Qu et al., 2020)) is proposed to separate rule generation and rule weight learning by introducing a rule generator and a reasoning predictor respectively.
Revised: RNNLogic (Qu et al., 2020)) is proposed to separate rule generation and rule weight learning by introducing a rule generator and a reasoning predictor respectively.

Original: Although the introduction of the rule generator reduces the search space, it is still challenging for RNNLogic to scale to KGs with hundreds of relations (e.g., FB15K-237) or millions of entities (e.g., YAGO3-10).
Revised: Although the introduction of the rule generator reduces the search space, it is still challenging for RNNLogic to scale to KGs with hundreds of relations (e.g., FB15K-237) or millions of entities (e.g., YAGO3-10).

Original: Systematic Generalizable Methods.
Revised: Systematic Generalizable Methods.

Original: All the above methods cannot generalize to larger graph beyond training sets.
Revised: All the above methods cannot generalize to larger graph beyond training sets.

Original: To improve models’ systematicity, Conditional Theorem Provers (CTPs) is proposed to learn an optimal rule selection strategy via gradient-based optimisation.
Revised: To improve models’ systematicity, Conditional Theorem Provers (CTPs) is proposed to learn an optimal rule selection strategy via gradient-based optimisation.

Original: For each sub-goal, a select module produces a smaller set of rules, which is then used during the proving mechanism.
Revised: For each sub-goal, a select module produces a smaller set of rules, which is then used during the proving mechanism.

Original: However, since the length of the learned rules influences the number of parameters of the model, it limits the capability of CTPs to handle the complicated rules whose depth is large.
Revised: However, since the length of the learned rules influences the number of parameters of the model, it limits the capability of CTPs to handle the complicated rules whose depth is large.

Original: In addition, due to its high computational complexity, CTPs cannot handle KG completion tasks for large-scale KGs.
Revised: In addition, due to its high computational complexity, CTPs cannot handle KG completion tasks for large-scale KGs.

Original: Another reinforcement learning based method - R5 (Lu et al., 2022) is proposed to provide recurrent relational reasoning solution to learn compositional rules.
Revised: Another reinforcement learning based method - R5 (Lu et al., 2022) is proposed to provide recurrent relational reasoning solution to learn compositional rules.

Original: However, R5 cannot generalize to KG completion task due to the lack of scalability.
Revised: However, R5 cannot generalize to KG completion task due to the lack of scalability.

Original: It requires pre-sampling for the paths that entails the query.
Revised: It requires pre-sampling for the paths that entails the query.

Original: Considering that all the triples in a KG share the same training graph, even a relatively small scale KG contains a huge number of paths.
Revised: Considering that all the triples in a KG share the same training graph, even a relatively small scale KG contains a huge number of paths.

Original: Thus, it is impractical to applied R5 to even small scale
Revised: Thus, it is impractical to applied R5 to even small scale

Original: KG for rule learning.
Revised: KG for rule learning.

Original: In addition, R5 employs a hard decision mechanism for merging a relation pair into a single relation, which makes it challenging to handle the widely existing uncertainty in
Revised: In addition, R5 employs a hard decision mechanism for merging a relation pair into a single relation, which makes it challenging to handle the widely existing uncertainty in

Original: KGs.
Revised: KGs.

Original: For example, given the rule body hasAunt ( x, z ) ∧ hasSister
Revised: For example, given the rule body hasAunt ( x, z ) ∧ hasSister

Original: ( z, y ) , both hasMother ( x, y ) and hasAunt ( x, y ) can be derived as the rule head.
Revised: ( z, y ) , both hasMother ( x, y ) and hasAunt ( x, y ) can be derived as the rule head.

Original: The inaccurate merging of a relation pair may result in the error propagation when generalize to longer paths.
Revised: The inaccurate merging of a relation pair may result in the error propagation when generalize to longer paths.

Original: Although RLogic (Cheng et al., 2022) are generalizable across multiple tasks, including KG completion and inductive relation reasoning.
Revised: Although RLogic (Cheng et al., 2022) are generalizable across multiple tasks, including KG completion and inductive relation reasoning.

Original: However, the performance is not satisfying compared to our proposed NCRL .
Revised: However, the performance is not satisfying compared to our proposed NCRL .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose NCRL , an end-to-end neural model for learning compositional logic rules.
Revised: In this paper, we propose NCRL , an end-to-end neural model for learning compositional logic rules.

Original: NCRL treats logic rules as a hierarchical tree, and breaks the rule body into small atomic compositions in order to infer the head rule.
Revised: NCRL treats logic rules as a hierarchical tree, and breaks the rule body into small atomic compositions in order to infer the head rule.

Original: By recurrently merging compositions in the rule body with a recurrent attention unit, NCRL finally predicts a single rule head.

Original: Experimental results show that NCRL is scalable, efficient, and yields SOTA results for link prediction on large-scale KGs.
Revised: Experimental results show that NCRL is scalable, efficient, and yields SOTA results for link prediction on large-scale KGs.

Original: Moreover, we test NCRL for systematic generalization for inductive relational reasoning, by learning to reason on small observed graphs and evaluate on larger unseen graphs.

Original: NCRL demonstartes strong generalization across tasks compared to SOTA rule learning methods.