Original: R5: R ULE D ISCOVERY WITH R EINFORCED AND R ECURRENT R ELATIONAL R EASONING
Revised: R5: R ULE D ISCOVERY WITH R EINFORCED AND R ECURRENT R ELATIONAL R EASONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Systematicity, i.e., the ability to recombine known parts and rules to form new sequences while reasoning over relational data, is critical to machine intelligence.
Revised: Systematicity, i.e., the ability to recombine known parts and rules to form new sequences while reasoning over relational data, is critical to machine intelligence.

Original: A model with strong systematicity is able to train on small-scale tasks and generalize to large-scale tasks.
Revised: A model with strong systematicity is able to train on small-scale tasks and generalize to large-scale tasks.

Original: In this paper, we propose R5, a relational reasoning framework based on reinforcement learning that reasons over relational graph data and explicitly mines underlying compositional logical rules from observations.
Revised: In this paper, we propose R5, a relational reasoning framework based on reinforcement learning that reasons over relational graph data and explicitly mines underlying compositional logical rules from observations.

Original: R5 has strong systematicity and being robust to noisy data.
Revised: R5 has strong systematicity and being robust to noisy data.

Original: It consists of a policy value network equipped with Monte Carlo Tree Search to perform recurrent relational prediction and a backtrack rewriting mechanism for rule mining.
Revised: It consists of a policy value network equipped with Monte Carlo Tree Search to perform recurrent relational prediction and a backtrack rewriting mechanism for rule mining.

Original: By alternately applying the two components, R5 progressively learns a set of explicit rules from data and performs explainable and generalizable relation prediction.
Revised: By alternately applying the two components, R5 progressively learns a set of explicit rules from data and performs explainable and generalizable relation prediction.

Original: We conduct extensive evaluations on multiple datasets.
Revised: We conduct extensive evaluations on multiple datasets.

Original: Experimental results show that R5 outperforms various embedding-based and rule induction baselines on relation prediction tasks while achieving a high recall rate in discovering ground truth rules.
Revised: Experimental results show that R5 outperforms various embedding-based and rule induction baselines on relation prediction tasks while achieving a high recall rate in discovering ground truth rules.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While deep learning has achieved great success in various applications, it was pointed out that there is a debate over the problem of systematicity in connectionist models (Fodor & Pylyshyn, 1988; Fodor & McLaughlin, 1990; Hadley, 1994; Jansen & Watter, 2012;
Revised: While deep learning has achieved great success in various applications, it was pointed out that there is a debate over the problem of systematicity in connectionist models (Fodor & Pylyshyn, 1988; Fodor & McLaughlin, 1990; Hadley, 1994; Jansen & Watter, 2012;

Original: Dong et al., 2018).
Revised: Dong et al., 2018).

Original: To concretely explain systematicity (Hupkes et al., 2020), let us consider the kinship problem shown in Figure 1.
Revised: To concretely explain systematicity (Hupkes et al., 2020), let us consider the kinship problem shown in Figure 1.

Original: By training on the small-scale observed examples of family relationships where the rule paths are short (in Figure 1a&b, both of the rule paths have the length of 2), our objective is to extract the underlying rules as general principles and generalize to large-scale tasks where the rule paths are long (in Figure 1c, the rule path has the length of 3).
Revised: By training on the small-scale observed examples of family relationships where the rule paths are short (in Figure 1a&b, both of the rule paths have the length of 2), our objective is to extract the underlying rules as general principles and generalize to large-scale tasks where the rule paths are long (in Figure 1c, the rule path has the length of 3).

Original: The rules in the training examples are a) Mother ( X, Y )
Revised: The rules in the training examples are a) Mother ( X, Y )

Original: ←−
Revised: ←−

Original: Mother ( X, Z ) , Sister ( Z, Y ) , b) Grandma ( X, Y ) ←−
Revised: Mother ( X, Z ) , Sister ( Z, Y ) , b) Grandma ( X, Y ) ←−

Original: Mother ( X, Z ) , Father ( Z, Y ) .
Revised: Mother ( X, Z ) , Father ( Z, Y ) .

Original: And the necessary rule in order to conclude the relation between
Revised: And the necessary rule in order to conclude the relation between

Original: Mary and Ann is c)
Revised: Mary and Ann is c)

Original: Grandma ( X, Y )
Revised: Grandma ( X, Y )

Original: ←−
Revised: ←−

Original: Mother ( X, U ) , Sister ( U, Z ) , Father ( Z, Y ) , which does not explicitly show up in the training examples.
Revised: Mother ( X, U ) , Sister ( U, Z ) , Father ( Z, Y ) , which does not explicitly show up in the training examples.

Original: Successfully giving prediction in c) shows the ability of systematicity, i.e., model’s ability to recombine known parts and rules to form new sequences.
Revised: Successfully giving prediction in c) shows the ability of systematicity, i.e., model’s ability to recombine known parts and rules to form new sequences.

Original: Some attempts on the graph relational data grounded in Graph Neural Networks (Schlichtkrull et al., 2018; Sinha et al., 2020; Pan et al., 2020) rely on learning embedding vectors of entities and relations, and have shown some capability of systematicity, However, since the rules are implicitly encapsulated in the neural networks, these models are lack of interpretability.
Revised: Some attempts on the graph relational data grounded in Graph Neural Networks (Schlichtkrull et al., 2018; Sinha et al., 2020; Pan et al., 2020) rely on learning embedding vectors of entities and relations, and have shown some capability of systematicity, However, since the rules are implicitly encapsulated in the neural networks, these models are lack of interpretability.

Original: Inductive Logic Programming (ILP) (Muggleton, 1992; Muggleton et al., 1996; Getoor, 2000;  Evans & Grefenstette, 2018) is a subﬁeld of symbolic rule learning, which naturally learns symbolic rules and provides interpretable explanations for labels prediction as well as being able to generalize to other tasks.
Revised: Inductive Logic Programming (ILP) (Muggleton, 1992; Muggleton et al., 1996; Getoor, 2000; Yang et al., 2017; Evans & Grefenstette, 2018) is a subfield of symbolic rule learning, which naturally learns symbolic rules and provides interpretable explanations for labels prediction as well as being able to generalize to other tasks.

Original: A traditional ILP system learns a set of rules from a collection of positive and negative examples, which entails all the positives and none of the negatives.
Revised: A traditional ILP system learns a set of rules from a collection of positive and negative examples, which entails all the positives and none of the negatives.

Original: However, these methods face the challenge that the search space of the compositional rules is exponentially large, making it hard to scale beyond small rule sets.
Revised: However, these methods face the challenge that the search space of the compositional rules is exponentially large, making it hard to scale beyond small rule sets.

Original: Extending the idea of ILP, Neural-Symbolic Learning (Garcez et al., 2015; Besold et al., 2017) seeks to integrate principles from neural networks learning and logical reasoning, among which
Revised: Extending the idea of ILP, Neural-Symbolic Learning (Garcez et al., 2015; Besold et al., 2017) seeks to integrate principles from neural networks learning and logical reasoning, among which

Original: Neural Logic Machines (Dong et al., 2018; Zimmer et al., 2021) and Theorem Provers (Rockt¨aschel & Riedel, 2017; Minervini et al., 2020a;b) aim to overcome the problems of systematicity.
Revised: Neural Logic Machines (Dong et al., 2018; Zimmer et al., 2021) and Theorem Provers (Rockt¨aschel & Riedel, 2017; Minervini et al., 2020a;b) aim to overcome the problems of systematicity.

Original: On one
Revised: On one

Original: [Figure]
Revised: [Figure]

Original: + ➔ a. Training 
Revised: + ➔ a. Training graphb.

Original: [Figure] b. Training [Figure]
Revised: Training graphc.

Original: c. Inductive relation prediction
Revised: Inductive relation prediction

Original: [Figure]
Revised: [Figure]

Original: Grandma

Original: [Figure]

Original: Brother

Original: Husband

Original: [Figure]

Original: Sister

Original: M o t h e r

Original: X
Revised: X

Original: [Figure]
Revised: [Figure]

Original: C [Figure] Kate hand, the Neural Logic Machines approximate the logic predicates using a probabilistic tensor representation, and the conclusion tensors are generated by applying the neural operators over the premise tensors.
Revised: hand, the Neural Logic Machines approximate the logic predicates using a probabilistic tensor representation, and the conclusion tensors are generated by applying the neural operators over the premise tensors.

Original: These approaches have done a great job in reasoning the decision-making tasks including sorting tasks, ﬁnding the shortest paths, and so on.
Revised: These approaches have done a great job in reasoning the decision-making tasks including sorting tasks, finding the shortest paths, and so on.

Original: However, in terms of the relational reasoning tasks on graphs, Neural Logic Machines reason only the single relation prediction tasks, i.e. determining whether a single relation exists between all the queried entities or entity pairs.
Revised: However, in terms of the relational reasoning tasks on graphs, Neural Logic Machines reason only the single relation prediction tasks, i.e. determining whether a single relation exists between all the queried entities or entity pairs.

Original: On the other hand, Theorem Provers jointly learn representations and rules from data via backpropagation, given a pre-deﬁned task-speciﬁc rule template and a set of possible rules following the template.
Revised: On the other hand, Theorem Provers jointly learn representations and rules from data via backpropagation, given a pre-defined task-specific rule template and a set of possible rules following the template.

Original: Theorem
Revised: Theorem

Original: Provers are able to solve and reason the relation prediction tasks instead of only determining the existence of a single relation.
Revised: Provers are able to solve and reason the relation prediction tasks instead of only determining the existence of a single relation.

Original: However, the performance is not satisfying.
Revised: However, the performance is not satisfying.

Original: In this paper, we focus on model’s systematicity for relational reasoning over graph relational prediction tasks and present a new reasoning framework named R5 , i.e., R ule discovery with R einforced and R ecurrent R elational R easoning, for rule induction and reasoning on relational data with strong systematicity.
Revised: In this paper, we focus on model’s systematicity for relational reasoning over graph relational prediction tasks and present a new reasoning framework named R5 , i.e., R ule discovery with R einforced and R ecurrent R elational R easoning, for rule induction and reasoning on relational data with strong systematicity.

Original: R5 formulates the problem of relational reasoning from the perspective of sequential decision-making, and performs rule extraction and logical reasoning with deep reinforcement learning equipped with a dynamic rule memory.
Revised: R5 formulates the problem of relational reasoning from the perspective of sequential decision-making, and performs rule extraction and logical reasoning with deep reinforcement learning equipped with a dynamic rule memory.

Original: More concretely, R5 learns the short deﬁnite clauses that are in the form u ←− p i ∧ p j .
Revised: More concretely, R5 learns the short definite clauses that are in the form u ←− p i ∧ p j .

Original: Since long Horn clauses (Horn, 1951) can be decomposed into short
Revised: Since long Horn clauses (Horn, 1951) can be decomposed into short

Original: Horn clauses, a long deﬁnite clause outcome ←− p 

Revised: Horn clauses, a long definite clause outcome ←− p

Original: ∧ p 1 ...p i ∧ p j ...p k used in prediction tasks is represented with a sequence of short deﬁnite clauses while performing decision-making, i.e. p i ∧ p
Revised: ∧ p 1 ...p i ∧ p j ...p k used in prediction tasks is represented with a sequence of short definite clauses while performing decision-making, i.e. p i ∧ p

Original: j is replaced by u .
Revised: j is replaced by u .

Original: Speciﬁcally, we make the following contributions:
Revised: Specifically, we make the following contributions:

Original: First, R5 performs explicit reasoning for relation prediction via composition and achieves explainability.
Revised: First, R5 performs explicit reasoning for relation prediction via composition and achieves explainability.

Original: Instead of learning embeddings for entities and relations, (Hamilton et al., 2017;
Revised: Instead of learning embeddings for entities and relations, (Hamilton et al., 2017;

Original: Schlichtkrull et al., 2018; Wang et al., 2019; Pan et al., 2020) and performing implicit reasoning, we perform explicit relational reasoning by modeling it as sequential decision-making.
Revised: Schlichtkrull et al., 2018; Wang et al., 2019; Pan et al., 2020) and performing implicit reasoning, we perform explicit relational reasoning by modeling it as sequential decision-making.

Original: Speciﬁcally, given a query (usually consists of two queried entities) and the related relationship graph, the agent recurrently selects a relation pair r i , r j from the input graph to combine into a compound relation r k and updates the graph, until the target relation for the query is reached.
Revised: Specifically, given a query (usually consists of two queried entities) and the related relationship graph, the agent recurrently selects a relation pair r i , r j from the input graph to combine into a compound relation r k and updates the graph, until the target relation for the query is reached.

Original: Trained by reinforcement learning, the agent learns to take actions, i.e., which pair of relations to combine, given the state representations of all possible pairs of relations.
Revised: Trained by reinforcement learning, the agent learns to take actions, i.e., which pair of relations to combine, given the state representations of all possible pairs of relations.

Original: Second, we propose a dynamic rule memory module to maintain and score candidate rules during the training of the reasoning agent.
Revised: Second, we propose a dynamic rule memory module to maintain and score candidate rules during the training of the reasoning agent.

Original: Each item in the rule memory is a candidate rule in the format of r k ←− r i , r j .
Revised: Each item in the rule memory is a candidate rule in the format of r k ←− r i , r j .

Original: r i , r j serve as the key, and r k serves as the value.
Revised: r i , r j serve as the key, and r k serves as the value.

Original: In each step of decisionmaking, the agent queries the rule memory to re-use already stored rules for reasoning, i.e., deducing compound relations, or insert new candidate rules into it if no existing rule matches the agent’s action.
Revised: In each step of decisionmaking, the agent queries the rule memory to re-use already stored rules for reasoning, i.e., deducing compound relations, or insert new candidate rules into it if no existing rule matches the agent’s action.

Original: Each rule is associated with a score that indicates the conﬁdence for the rule.
Revised: Each rule is associated with a score that indicates the confidence for the rule.

Original: The rules and their scores in the memory are dynamically updated during training.
Revised: The rules and their scores in the memory are dynamically updated during training.

Original: Finally, a set of candidate rules with scores above a threshold are kept.
Revised: Finally, a set of candidate rules with scores above a threshold are kept.

Original: By recurrently applying the learned rules, R5 demonstrates strong compositional generalization ability (i.e. systematicity) in relational reasoning tasks.
Revised: By recurrently applying the learned rules, R5 demonstrates strong compositional generalization ability (i.e. systematicity) in relational reasoning tasks.

Original: Third, rather than only taking the observed relations into account, we introduce extra hidden (or unknown) relations into the reasoning process.
Revised: Third, rather than only taking the observed relations into account, we introduce extra hidden (or invented) relations into the reasoning process.

Original: For example, R5 may learn to combine r 1 and r 2 by a rule r ← r 1 , r 2 , where r is an intentionally introduced unknown relation.
Revised: For example, R5 may learn to combine r 1 and r 2 by a rule r ← r 1 , r 2 , where r is an intentionally introduced unknown relation.

Original: The unknown relations can appear on both the left or right side of a rule clause.
Revised: The invented relations can appear on both the left or right side of a rule clause.

Original: Such a design enables our model to learn intermediate relations that do not explicitly appear in the training data to model underlying complex relations between entities, which are sometimes necessary to discover complex rules in the relational data.
Revised: Such a design enables our model to learn intermediate relations that do not explicitly appear in the training data to model underlying complex relations between entities, which are sometimes necessary to discover complex rules in the relational data.

Original: Furthermore, we design a backtrack rewriting mechanism that replaces an unknown relation with an observed one when R5 ﬁnds they are actually equivalent.
Revised: Furthermore, we design a backtrack rewriting mechanism that replaces an invented relation with an observed one when R5 finds they are actually equivalent.

Original: Our goal is to explain the observed data with rules that are as simple as possible.
Revised: Our goal is to explain the observed data with rules that are as simple as possible.

Original: Backtrack rewriting merges redundant relations and rules 
Revised: Backtrack rewriting merges redundant relations and rules to enforce the principle of Occam’s razor, which prevents overfitting and improves the robustness of our approach.

Original: [Table]
Revised: [Table]

Original: [Figure] [Table][Figure]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: loJikGMR1KHnFGjZX8Xsr79X6l6tbcOcgq8QpShQLNfuWrN0hYFqM0TFCtu56bmiCnynAmcFruZRpTysZ0iF1LJY1RB

Original: [Figure]
Revised: [Table]

Original: to enforce the principle of Occam’s razor, which prevents overﬁtting and improves the robustness of our approach.

Original:  We perform extensive evaluations based on two public relation prediction datasets, CLUTRR (Sinha et al., 2019) and GraphLog (Sinha et al., 2020), and compare R5 with a variety of baseline methods.
Revised: [Figure] We perform extensive evaluations based on two public relation prediction datasets, CLUTRR (Sinha et al., 2019) and GraphLog (Sinha et al., 2020), and compare R5 with a variety of baseline methods.

Original: The experimental results demonstrate that our approach signiﬁcantly outperforms state-of-the-art methods in terms of relation prediction accuracy and recall rate in rule discovery.
Revised: The experimental results demonstrate that our approach significantly outperforms state-of-the-art methods in terms of relation prediction accuracy and recall rate in rule discovery.

Original: Moreover, R exhibits a strong ability of compositional generalization and robustness to data noise.
Revised: Moreover, R exhibits a strong ability of compositional generalization and robustness to data noise.

Original:  P ROPOSED A PPROACH
Revised:  P ROPOSED A PPROACH

Original: [Table]
Revised: [Table]

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we introduce our Rule Discovery with Reinforced and Recurrent Relational Reasoning framework, namely R5, to solve the inductive relation prediction problem.
Revised: we introduce our Rule Discovery with Reinforced and Recurrent Relational Reasoning framework, namely R5, to solve the inductive relation prediction problem.

Revised: We first formally define the relation prediction problem discussed in this paper.

Revised: Let p data ( G , q , a ) be a training data distribution, where G is the set of training graphs, q = ( X, Y ) is a query

Original: Figureshows an overview of R5.
Revised: [Table] and a = r is the answer. The graphs consist of nodes in a set N , which are connected by relations in a set R . X, Y ∈ N are nodes, and r ∈ R is a relation.

Revised: Given

Revised: G and the query q , the goal is to predict the correct answer a .

Revised: Relational prediction task is actually program induction, which is a hard problem that has been studied for many years in the area of Inductive Logic Programming and Statistical

Revised: Relational Learning, especially for large-scale tasks in noisy domains.

Revised: Figure 2 shows an overview of R5.

Original: Our framework takes a relational graph as input, and outputs the relationship between two queried entities based on extracted rules.
Revised: Our framework takes a relational graph as input, and outputs the relationship between two queried entities based on extracted rules.

Original: R5 ﬁrst transforms a relation graph into a set of paths connecting the queried node (entity) pair.
Revised: R5 first transforms a relation graph into a set of paths connecting the queried node (entity) pair.

Original: After that, it recurrently applies learned rules to merge a relation pair in the paths to form a compound relation, until it outputs a ﬁnal relation between the two queried nodes.
Revised: After that, it recurrently applies learned rules to merge a relation pair in the paths to form a compound relation, until it outputs a final relation between the two queried nodes.

Original: The reasoning agent is trained with deep reinforcement learning and MCTS, while a dynamic rule memory module is utilized to extract rules from observations during training.
Revised: The reasoning agent is trained with deep reinforcement learning and MCTS, while a dynamic rule memory module is utilized to extract rules from observations during training.

Original: The notations we used in this paper are summarized in Table 1.
Revised: The notations we used in this paper are summarized in Table 1.

Original: Note that an action a :=
Revised: Note that an action a :=

Original: a H ← ( a B, 0 , a B, 1 ) and a rule rl := rl H ← ( rl B, 0 , rl B, 1 ) share the same structure, where different a and rl are all relations.
Revised: a H ← ( a B, 0 , a B, 1 ) and a rule rl := rl H ← ( rl B, 0 , rl B, 1 ) share the same structure, where different a and rl are all relations.

Original: a H and a B represent the head and body in a rule, respectively.
Revised: a H and a B represent the head and body in a rule, respectively.

Original: Next we introduce R5 in detail.
Revised: Next we introduce R5 in detail.

Original: 2. R ECURRENT R ELATIONAL R EASONING
Revised: 2. R ECURRENT R ELATIONAL R EASONING

Original: Path sampling.
Revised: Path sampling.

Original: In order to predict the relation between two entities in graphs, we start with datapreprocessing by enumerating or sampling paths that connect the two entities from the graph.
Revised: To predict the relation between two entities in graphs, we preprocess the data by enumerating or sampling paths that connect the two entities from the graph.

Revised: Similar to Mooney (1992), R5 is based on the assumption that, in most relational domains, important concepts can be represented by a number of fixed paths among the constants defining the a positive instance.

Original: When the total number of paths is small, we enumerate all the paths.
Revised: When the total number of paths is small, we enumerate all the paths.

Original: Otherwise, we randomly sample maximum L paths.
Revised: Otherwise, we randomly sample maximum L paths.

Original: Each path purely consists of relations.
Revised: Each path purely consists of relations.

Original: For instance, in Figure 2, given the input graph, we can get 4 paths between query nodes X and Y , including r 1 - r 2 - r 5 - r 3 - r 7 - r 6 and so on.
Revised: For instance, in Figure 2, given the input graph, we can get 4 paths between query nodes X and Y , including r 1 - r 2 - r 5 - r 3 - r 7 - r 6 and so on.

Original: Reasoning as sequential decision-making.
Revised: Reasoning as sequential decision-making.

Original: Our method solves the problem of relational reasoning in terms of sequential decision-making.
Revised: Our method solves the problem of relational reasoning in terms of sequential decision-making.

Original: Speciﬁcally, we train a reasoning agent based on a policy value network combined with MCTS (Silver et al., 2017a) to recurrently reason over the extracted relation paths.
Revised: Specifically, we train a reasoning agent based on a policy value network combined with MCTS (Silver et al., 2017a) to recurrently reason over the extracted relation paths.

Original: The policy value network f θ ( s ) is a neural network with parameters θ .
Revised: The policy value network f θ ( s ) is a neural network with parameters θ .

Original: It takes the current state s as input, and outputs the action probability distribution ρ and a state value ν .
Revised: It takes the current state s as input, and outputs the action probability distribution ρ and a state value ν .

Original: MCTS utilizes the policy network f θ ( s ) to guide its simulations, and outputs a vector π representing the improved search probabilities of the available actions.
Revised: MCTS utilizes the policy network f θ ( s ) to guide its simulations, and outputs a vector π representing the improved search probabilities of the available actions.

Original: The policy network ( ρ , ν ) =
Revised: The policy network ( ρ , ν ) =

Original: f θ ( s ) is then trained to minimize the error between the predicted state value ν
Revised: f θ ( s ) is then trained to minimize the error between the predicted state value ν

Original: and the reward z received after an episode, as well as maximize the similarity between two probability distributions ρ and π .
Revised: and the reward z received after an episode, as well as maximize the similarity between two probability distributions ρ and π .

Original: The loss function l is l
Revised: The loss function l is l

Original: =
Revised: =

Original: ( z − ν ) 2 − π (cid:124) log ρ + a (cid:107) θ (cid:107) 2 , where a is a hyper-parameter.
Revised: ( z − ν ) 2 − π ⊺ log ρ + a ∥ θ ∥ 2 , where a is a hyper-parameter.

Original: Action.
Revised: Action.

Original: At each step of an episode, the MCTS outputs an action ( a B, 0 , a B, 1 ) , which is a relation pair denoted as a B .
Revised: At each step of an episode, the MCTS outputs an action ( a B, 0 , a B, 1 ) , which is a relation pair denoted as a B .

Original: Furthermore, by looking up the dynamic rule memory, it obtains a : a H ← ( a B, 0 , a B, 1 ) that contains a head relation a H to be deducted to, which means that the relation pair ( a B, 0 , a B, 1 ) in the path will be substitute with a H .
Revised: Furthermore, by looking up the dynamic rule memory, it obtains a : a H ← ( a B, 0 , a B, 1 ) that contains a head relation a H to be deducted to, which means that the relation pair ( a B, 0 , a B, 1 ) in the path will be substitute with a H .

Original: For example, in Figure 2 at step 1 , MCTS outputs an action a : r 3 ← ( r 1 , r 2 ) , and the path r 1 - r 2 - r 5 - r 3 - r 7 - r 6 is transformed into r 3 - r 5 - r 3 - r 7 - r 6 .
Revised: For example, in Figure 2 at step 1 , MCTS outputs an action a : r 3 ← ( r 1 , r 2 ) , and the path r 1 - r 2 - r 5 - r 3 - r 7 - r 6 is transformed into r 3 - r 5 - r 3 - r 7 - r 6 .

Original: By recurrently applying different actions to a path between the query nodes, it will be transformed into a single relation at the end of an episode.
Revised: By recurrently applying different actions to a path between the query nodes, it will be transformed into a single relation at the end of an episode.

Original: State.
Revised: State.

Original: Instead of encoding the walking paths between the query nodes as other RL-based rule induction methods (Shen et al., 2018; Das et al., 2017; Xiong et al., 2017), we make use of the features of the possible relations pairs at the current state.
Revised: Instead of encoding the walking paths between the query nodes as other RL-based rule induction methods (Shen et al., 2018; Das et al., 2017; Xiong et al., 2017), we make use of the features of the possible relations pairs at the current state.

Original: As shown in Figure 2, we deﬁne the state s ∈ R ( m + n ) × ( m + n ) × k as the representation of all the possible relations pairs among the current paths, where m is the number of observed relations, n is the number of unknown relations we assumed, and k is the dimension of features of each relation pair.
Revised: As shown in Figure 2, we define the state s ∈ R ( m + n ) × ( m + n ) × k as the representation of all the possible relations pairs among the current paths, where m is the number of observed relations, n is the number of invented relations we assumed, and k is the dimension of features of each relation pair.

Original: As previously discussed, a relation pair may be deducted to an unknown relation r ∗ ∈ N .
Revised: As previously discussed, a relation pair may be deducted to an invented relation r ∗ ∈ N .

Original: Besides, r ∗ can be paired with another relation r , and maybe further deducted to another unknown relation r (cid:48)∗ ∈ N .
Revised: Besides, r ∗ can be paired with another relation r , and maybe further deducted to another invented relation r ′∗ ∈ N .

Original: Our state space design enables us to represent such complicated rules with the unknown relations that serve as intermediate predicates, which is essential to model complex relational data.
Revised: Our state space design enables us to represent such complicated rules with the invented relations that serve as intermediate predicates, which is essential to model complex relational data.

Original: n unknown relations are allowed in our design, where n depends on the complexity of input.
Revised: n invented relations are allowed in our design, where n depends on the complexity of input.

Original: An example is presented in Appendix A.4.
Revised: An example is presented in Appendix A.4.

Original: Even if sometimes the input data can be explained by rules without unknown relation, the unknown relations in our state design can actually help to speed up model training.
Revised: Even if sometimes the input data can be explained by rules without invented relation, the invented relations in our state design can actually help to speed up model training.

Original: For example, when we observe that a relation pair r 1 and r 2 shows up frequently in the training data, we can sample an unknown relation r ∗ as the head, and form a candidate rule r ∗ ← ( r 1 , r 2 ) .
Revised: For example, when we observe that a relation pair r 1 and r 2 shows up frequently in the training data, we can sample an invented relation r ∗ as the head, and form a candidate rule r ∗ ← ( r 1 , r 2 ) .

Original: In this way, our model learns to merge r 1 and r 2 quickly without the need to ﬁgure out what exactly r ∗ is.
Revised: In this way, our model learns to merge r 1 and r 2 quickly without the need to figure out what exactly r ∗ is.

Original: The value of r ∗ will be inferred by our model with Backtrack Rewriting mechanism, which we will introduce in more detail later.
Revised: The value of r ∗ will be inferred by our model with Backtrack Rewriting mechanism, which we will introduce in more detail later.

Original: Without these unknown relations, a reasoning agent has to infer what exactly r ∗ is at an early state, otherwise it cannot proceed to merge the relation pairs in a path correctly.
Revised: Without these invented relations, a reasoning agent has to infer what exactly r ∗ is at an early state, otherwise it cannot proceed to merge the relation pairs in a path correctly.

Original: Thus, the unknown relations actually serve as a buffering mechanism to allow an agent to learn “what to merge” ﬁrst, rather than inferring the exact rule at an early stage.
Revised: Thus, the invented relations actually serve as a buffering mechanism to allow an agent to learn “what to merge” first, rather than inferring the exact rule at an early stage.

Original: The features we utilized to represent each relation pair can be summarized into two groups.
Revised: The features we utilized to represent each relation pair can be summarized into two groups.

Original: The ﬁrst group includes the general and statistical features of the pair, such as the number of occurrences, the index of the most viewed position among the paths, the number of occurrences at the most viewed position, and the types of the two relations consist the pair.
Revised: The first group includes the general and statistical features of the pair, such as the number of occurrences, the index of the most viewed position among the paths, the number of occurrences at the most viewed position, and the types of the two relations consist the pair.

Original: The other group includes the rulesrelated features, such as whether the relation pair is in the obtained rules, whether the relation pair consists of two known relations, whether the rule head is a known relation if the pair is in the rules memory, and the score of the corresponding rule if the pair is in the rules memory.
Revised: The other group includes the rulesrelated features, such as whether the relation pair is in the obtained rules, whether the relation pair consists of two known relations, whether the rule head is a known relation if the pair is in the rules memory, and the score of the corresponding rule if the pair is in the rules memory.

Original: Reward.
Revised: Reward.

Original: During an episode, actions are applied to a relation path recurrently, until the path is deducted to a single relation r , and a reward z T ∈ {− 1 , 0 , +1 } will be assigned to all the states in this episode.
Revised: During an episode, actions are applied to a relation path recurrently, until the path is deducted to a single relation r , and a reward z T ∈ {− 1 , 0 , +1 } will be assigned to all the states in this episode.

Original: If r is an known relation, but is not the target relation, z T = − 1 .
Revised: If r is an known relation, but is not the target relation, z T = − 1 .

Original: If r is an unknown relation, z T = 0 .
Revised: If r is an invented relation, z T = 0 .

Original: If r is the target relation, z T = 1 .
Revised: If r is the target relation, z T = 1 .

Original: 2. R ULE I NDUCTION WITH A D YNAMIC R ULE M EMORY
Revised: 2. R ULE I NDUCTION WITH A D YNAMIC R ULE M EMORY

Original: The above recurrent reasoning agent enables us to predict the relation between two queried nodes efﬁciently.
Revised: The above recurrent reasoning agent enables us to predict the relation between two queried nodes more efficiently than only using MCTS.

Original: To extract rules from observed data, we further design a dynamic rule memory module which stores and updates candidate rules and interact with the reasoning agent during training.
Revised: To extract rules from observed data, we further design a dynamic rule memory module which stores and updates candidate rules and interact with the reasoning agent during training.

Original: Our memory design can be interpreted as two hash tables with the same keys: one hash table D rl in the form of ( r B 0 , r B 1 ) :
Revised: Our memory design can be interpreted as two hash tables with the same keys: one hash table D rl in the form of ( r B 0 , r B 1 ) :

Original: r h is used to memorize the candidate rules, and another hash table D rls in the form of ( r B 0 , r B 1 ) : score to track the rule scores.
Revised: r h is used to memorize the candidate rules, and another hash table D rls in the form of ( r B 0 , r B 1 ) : score to track the rule scores.

Original: Note that in Figure 2, we union the two tables as they share the same keys.
Revised: Note that in Figure 2, we union the two tables as they share the same keys.

Original: A newly added rule will have an initial score of 0 .
Revised: A newly added rule will have an initial score of 0 .

Original: In Figure 2, a rule ( r 3 ) ← ( r 1 , r 2 ) was stored in the hash tables, where the rule body ( r 1 , r 2 ) is the key, and the rule head r 3 and score 4 .
Revised: In Figure 2, a rule ( r 3 ) ← ( r 1 , r 2 ) was stored in the hash tables, where the rule body ( r 1 , r 2 ) is the key, and the rule head r 3 and score 4 .

Original: 303 are the values in D rl and D rls , respectively.
Revised: 303 are the values in D rl and D rls , respectively.

Original: We utilize a buffer B unkn to store the n intermediate unknown relations as described in Sec. 2.1.
Revised: We utilize a buffer B unkn to store the n intermediate invented relations as described in Sec. 2.1.

Original: When an unknown relation type r ∈ N is used to complete a rule, it will be removed from B unkn .
Revised: When an invented relation type r ∈ N is used to complete a rule, it will be removed from B unkn .

Original: If r is freed from the rule memory D rl , then it will be added to B unkn again.
Revised: If r is freed from the rule memory D rl , then it will be added to B unkn again.

Original: Recall that at each time step the MCTS will return an action body a B .
Revised: Recall that at each time step the MCTS will return an action body a B .

Original: In order to complete the action, we get the action head a H by
Revised: In order to complete the action, we get the action head a H by

Original: [Equation]
Revised: [Equation]

Original: RandomSample
Revised: RandomSample

Original: [Equation]

Original: B unkn ) , otherwise .

Original: [Equation]
Revised: [Equation]

Original: Then, if a B / ∈ D rl , we update the memory D rl with D rl [ a B ] = a H , and compute the score c a by
Revised: Then, if a B / ∈ D rl , we update the memory D rl with D rl [ a B ] = a H , and compute the score c a by

Original: [Equation]
Revised: [Equation]

Original: where v 0 > v 1 > 0 > v 2 > v 3 > v 4 .
Revised: where v 0 > v 1 > 0 > v 2 > v 3 > v 4 .

Original: We can see that an action consisting of three known relations will receive the largest positive score, and an action with an unknown relation in the head and at least an unknown relation in the body will receive the smallest negative score.
Revised: We can see that an action consisting of three known relations will receive the largest positive score, and an action with an invented relation in the head and at least an invented relation in the body will receive the smallest negative score.

Original: This way we penalize the usage of unknown relations to prevent overﬁtting.
Revised: This way we penalize the usage of invented relations to prevent overfitting.

Original: [Table]
Revised: [Table]

Original: [ a TB ] 3: if r ∈ N then 4: for a B in D rl do 5: if D rl [ a B ] =

Original: r then 6: D rl [ a B ] = y 7: for a B in D rl do 8: a H = D rl [ a B ] 9: if a B, 0 = r and a B, 1 = r then 10: key =
Revised: After extracted candidate rules, at the end of each training episode, we further utilize Backtrack

Original: ( y, y ) 11: else if a B, 0 = r then 12: key = ( y, a B, 1 ) 13: else if a B, 1 = r then 14: key = ( a B, 0 , y ) 15: else 16: continue 17: if key / ∈ D rl or D rl [ key ] ∈ N then 18: delete a B from D rl 19: D rl [ key ] = a H 20: else 21: delete a B from D rl 22: remove r from B unkn After extracted candidate rules, at the end of each training episode, we further utilize Backtrack Rewriting to consolidate all the candidate rules and replace unknown relations in a rule with observed relations if we found they are equivalent.
Revised: Rewriting to consolidate all the candidate rules and replace invented relations in a rule with observed relations if we found they are equivalent.

Original: Algorithm 1 describes our Backtrack Rewriting mechanism for updating the rule memory D rl .
Revised: Algorithm 1 describes our Backtrack Rewriting mechanism for updating the rule memory D rl .

Original: Speciﬁcally, assume the action sequence we have taken in an episode is { a 0 , a 1 , ...a t , ...a T } , and the head relation of the last action
Revised: Specifically, assume the action sequence we have taken in an episode is { a 0 , a 1 , ...a t , ...a T } , and the head relation of the last action

Original: a
Revised: a

Original: T is r := a T H
Revised: T is r := a T H

Original: = D rl [ a T B ] .
Revised: = D rl [ a T B ] .

Original: As y is the target relation, we can backtrack r in D rl , and replace all its occurrences with y .
Revised: As y is the target relation, we can backtrack r in D rl , and replace all its occurrences with y .

Original: Before updating D rl [ a T B ] , we update the corresponding entries in D rls of the actions { a 0 , a 1 , ...a t , ...a T } in the episode by
Revised: Before updating D rl [ a T B ] , we update the corresponding entries in D rls of the actions { a 0 , a 1 , ...a t , ...a T } in the episode by

Original: D rls [ a tB ] =    c a t + v T pos , r = y, v T neg , r ∈ M, r (cid:54) =
Revised: [Equation]

Original: y, c a t , otherwise , (3) where r :
Revised: where r :

Original: = D rl [ a T B ] , and v T neg < 0 < v T pos are the episode end scores.
Revised: = D rl [ a T B ] , and v T neg < 0 < v T pos are the episode end scores.

Original: Next, we decay the scores D rls [ a B ] of all the rules a H ← a B over episodes by:
Revised: Next, we decay the scores D rls [ a B ] of all the rules a H ← a B over episodes by:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is a small positive decay factor.
Revised: where ϵ is a small positive decay factor.

Original: Thus, the score of a rule that is not accessed over training episodes will decrease.
Revised: Thus, the score of a rule that is not accessed over training episodes will decrease.

Original: Finally, when a rule score is less than a negative threshold σ , the corresponding rule will be omitted from the rule memory.
Revised: Finally, when a rule score is less than a negative threshold σ , the corresponding rule will be omitted from the rule memory.

Original: We recursively check the rule memory to remove all the candidate rules  that contain the omitted rule’s head a Hbad ∈ N .
Revised: We recursively check the rule memory to remove all the candidate rules [Table] [Table] that contain the omitted rule’s head a Hbad ∈ N .

Original: Thus, the bad rules will be gradually removed from the rule memory.
Revised: Thus, the bad rules will be gradually removed from the rule memory.

Original: B unkn and D rls will be updated accordingly.
Revised: B unkn and D rls will be updated accordingly.

Original: Besides, when the unknown [Table] [Table] relations buffer B unkn is empty, we will look up the rules scores hash table D rls to ﬁnd the rule a Hbad ← a Bbad that has the smallest rule score, where a Hbad ∈ N .
Revised: Besides, when the invented relations buffer B unkn is empty, we will look up the rules scores hash table D rls to find the rule a Hbad ← a Bbad that has the smallest rule score, where a Hbad ∈ N .

Original: Then all the corresponding rules in D rl that contain a Hbad will be removed, and a Hbad will be added back to B unkn .
Revised: Then all the corresponding rules in D rl that contain a H bad will be removed, and a H bad will be added back to B unkn .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate R5 on two datasets: CLUTRR (Sinha et al., 2019) and GraphLog (Sinha et al., 2020), which test the logical generalization capabilities.
Revised: We evaluate R5 on two datasets: CLUTRR (Sinha et al., 2019) and GraphLog (Sinha et al., 2020), which test the logical generalization capabilities.

Original: We compare with embedding-based and rule induction baselines for both CLUTRR and GraphLog.
Revised: We compare with embedding-based and rule induction baselines for both CLUTRR and GraphLog.

Original: 3. S YSTEMATICITY ON C LEAN D ATA
Revised: 3. S YSTEMATICITY ON C LEAN D ATA

Original: We ﬁrst evaluate R5 on the CLUTRR dataset and demonstrate its Systematicity.
Revised: We first evaluate R5 on the CLUTRR dataset and demonstrate its Systematicity.

Original: In other words, we test whether R5 can perform reasoning over graphs with more hops than the training data.
Revised: In other words, we test whether R5 can perform reasoning over graphs with more hops than the training data.

Original: Dataset and Experimental Setup.
Revised: Dataset and Experimental Setup.

Original: CLUTRR (Sinha et al., 2019) is a dataset for inductive reasoning over family relations.
Revised: CLUTRR (Sinha et al., 2019) is a dataset for inductive reasoning over family relations.

Original: Paragraphs describing family relationships are encoded as a large set of graphs, where family members are modeled as nodes, and the relations in between are modeled as edges.
Revised: Paragraphs describing family relationships are encoded as a large set of graphs, where family members are modeled as nodes, and the relations in between are modeled as edges.

Original: The goal is to infer the relation between two family members, which is not explicitly mentioned in the paragraph.
Revised: The goal is to infer the relation between two family members, which is not explicitly mentioned in the paragraph.

Original: The train set contains graphs with up to 3 or 4 edges, and the test set contains graphs with up to 10 edges.
Revised: The train set contains graphs with up to 3 or 4 edges, and the test set contains graphs with up to 10 edges.

Original: The train and test splits do not share the same set of entities.
Revised: The train and test splits do not share the same set of entities.

Original: As baselines, we evaluate several embedding-based models (Veli ˇ ckovi ´ c et al., 2017; Kipf & Welling, 2017; Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997; Chung et al., 2014; Kim, 2014;
Revised: As baselines, we evaluate several embedding-based models (Veliˇckovi´c et al., 2017; Kipf & Welling, 2017; Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997; Chung et al., 2014; Kim, 2014;

Original: Kim et al., 2015; Vaswani et al., 2017) and several neuro-symbolic models that consists the rule induction baselines, including GNTPs (Minervini et al., 2020a) and CTPs (Minervini et al., 2020b).
Revised: Kim et al., 2015; Vaswani et al., 2017) and several neuro-symbolic models that consists the rule induction baselines, including GNTPs (Minervini et al., 2020a) and CTPs (Minervini et al., 2020b).

Original: The results of the baselines are taken from Minervini et al.
Revised: The results of the baselines are taken from Minervini et al.

Original: (2020b).
Revised: (2020b).

Original: Due to limited space, for the baselines other than CTPs and GNTPs, we only present the results ﬁne-tuned on short stories.
Revised: Due to limited space, for the baselines other than CTPs and GNTPs, we only present the results fine-tuned on short stories.

Original: The comparisons with these baselines ﬁne-tuned on long stories are provided in the Appendix C, and none of them can beat R5.
Revised: The comparisons with these baselines fine-tuned on long stories are provided in the Appendix C, and none of them can beat R5.

Original: Results.
Revised: Results.

Original: Table 10, 11 present the prediction accuracy on two datasets pulished by Sinha et al.
Revised: Table 10, 11 present the prediction accuracy on two datasets pulished by Sinha et al.

Original: We call the two datasets CLUTRR k ∈{ 2 , 3 } and CLUTRR k ∈{ 2 , 3 , 4 } , as they train on graphs with { 2 , 3 } edges while test on graphs with { 4 , ..., 10 } edges, and train on graphs with { 2 , 3 , 4 } edges while test on graphs with { 5 , ..., 10 } edges, respectively.
Revised: We call the two datasets CLUTRR k ∈{ 2 , 3 } and CLUTRR k ∈{ 2 , 3 , 4 } , as they train on graphs with { 2 , 3 } edges while test on graphs with { 4 , ..., 10 } edges, and train on graphs with { 2 , 3 , 4 } edges while test on graphs with { 5 , ..., 10 } edges, respectively.

Original: As shown in both tables, R5 either outperforms all the baselines or reach the best results in both metrics.
Revised: As shown in both tables, R5 either outperforms all the baselines or reach the best results in both metrics.

Original: In particular, R5 has better generalization capability than CTPs.
Revised: In particular, R5 has better generalization capability than CTPs.

Original: For both datasets, the prediction accuracy of CTPs goes down when evaluated on longer stories.
Revised: For both datasets, the prediction accuracy of CTPs goes down when evaluated on longer stories.

Original: In contrast, R5 still predicts successfully on longer stories without signiﬁcant performance degradation.
Revised: In contrast, R5 still predicts successfully on longer stories without significant performance degradation.

Original: This demonstrates the strong compositional generalization ability of R5, in terms of systematicity, productivity, substitutivity, and localism (Hupkes et al., 2020).
Revised: This demonstrates the strong compositional generalization ability of R5, in terms of systematicity, productivity, substitutivity, and localism (Hupkes et al., 2020).

Original: 3. S YSTEMATICITY ON N OISY D ATA
Revised: 3. S YSTEMATICITY ON N OISY D ATA

Original: In Section 3.1, we test model performance with no bad cases in data.
Revised: In Section 3.1, we test model performance with no bad cases in data.

Original: A bad case is a logically wrong example.
Revised: A bad case is a logically wrong example.

Original: For example, the mother of a sister is a father is a bad case.
Revised: For example, the mother of a sister is a father is a bad case.

Original: Apart from the compositional
Revised: Apart from the compositional

Original: [Table]
Revised: [Table]

Original: generalization ability, it is also important to assess how effectively a model can discover all underlying rules.
Revised: generalization ability, it is also important to assess how effectively a model can discover all underlying rules.

Original: The existing relational reasoning works only show that they are able to improve relation prediction with some learned rules, without an evaluation on whether the extracted rules are exhaustive.
Revised: The existing relational reasoning works only show that they are able to improve relation prediction with some learned rules, without an evaluation on whether the extracted rules are exhaustive.

Original: GraphLog, the dataset we use in this subsection, provides the ground rules used to create each logical world, so we evaluate this ability by the recall rate of ground truth rules.
Revised: GraphLog, the dataset we use in this subsection, provides the ground rules used to create each logical world, so we evaluate this ability by the recall rate of ground truth rules.

Original: Dataset and Experimental Setup.
Revised: Dataset and Experimental Setup.

Original: Graphlog (Sinha et al., 2020) is a benchmark suite designed to evaluate systemeticity.
Revised: Graphlog (Sinha et al., 2020) is a benchmark suite designed to evaluate systemeticity.

Original: It contains logical worlds where each world contains graphs that are created using a different set of ground rules.
Revised: It contains logical worlds where each world contains graphs that are created using a different set of ground rules.

Original: Similar to CLUTRR, the goal is to infer the relation between a queried node pair.
Revised: Similar to CLUTRR, the goal is to infer the relation between a queried node pair.

Original: However, GraphLog is much more challenging than CLUTRR.
Revised: However, GraphLog is much more challenging than CLUTRR.

Original: Firstly, it contains more bad examples than CLUTRR does.
Revised: Firstly, it contains more bad examples than CLUTRR does.

Original: Secondly, the graphs in both train sets and test sets of
Revised: Secondly, the graphs in both train sets and test sets of

Original: Graphlog contain more edges; it is normal that there are more than 10 edges in a graph.
Revised: Graphlog contain more edges; it is normal that there are more than 10 edges in a graph.

Original: Thirdly, the underlying compositional rules may not be shared between a train set and its corresponding test set.
Revised: Thirdly, the underlying compositional rules may not be shared between a train set and its corresponding test set.

Original: The above facts make GraphLog a more challenging dataset for rule induction.
Revised: The above facts make GraphLog a more challenging dataset for rule induction.

Original: In the evaluations on GraphLog, we consider E-GAT (Sinha et al., 2020) and R-GCN (Schlichtkrull et al., 2018) as the embedding-based baselines, and CTPs (Minervini et al., 2020b) as the rule induction baseline.
Revised: In the evaluations on GraphLog, we consider E-GAT (Sinha et al., 2020) and R-GCN (Schlichtkrull et al., 2018) as the embedding-based baselines, and CTPs (Minervini et al., 2020b) as the rule induction baseline.

Original: For E-GAT and R-GCN, we use the results in (Sinha et al., 2020).
Revised: For E-GAT and R-GCN, we use the results in (Sinha et al., 2020).

Original: For CTPs, we use the implementation publicly provided by the author with the best conﬁgurations.
Revised: For CTPs, we use the implementation publicly provided by the author with the best configurations.

Original: In addition, we use the core generator provided by (Sinha et al., 2020), which is used to produce the GraphLog datasets, to generate three new worlds: Re 0, Re 1, Re 2.
Revised: In addition, we use the core generator provided by (Sinha et al., 2020), which is used to produce the GraphLog datasets, to generate three new worlds: Re 0, Re 1, Re 2.

Original: The new worlds have less training data and more distinct relations sequences.
Revised: The new worlds have less training data and more distinct relations sequences.

Original: We aim to test how the rule induction models perform when only a small amount of training samples are given.
Revised: We aim to test how the rule induction models perform when only a small amount of training samples are given.

Original: We divide the 57 worlds in Graphlog into two groups in terms of the average resolution length (ARL), where the resolution length is the number of nodes traversed in the correct path between the queried nodes.
Revised: We divide the 57 worlds in Graphlog into two groups in terms of the average resolution length (ARL), where the resolution length is the number of nodes traversed in the correct path between the queried nodes.

Original: Figure 1a,b are instances of resolution lengths of 2, and Figure 1c is an instance of resolution length of 3.
Revised: Figure 1a,b are instances of resolution lengths of 2, and Figure 1c is an instance of resolution length of 3.

Original: We then randomly pick 5 worlds with an ARL less than 4, and 5 other worlds with an ARL greater than 4 to conduct experiments on.
Revised: We then randomly pick 5 worlds with an ARL less than 4, and 5 other worlds with an ARL greater than 4 to conduct experiments on.

Original: Results.
Revised: Results.

Original: Table 4 shows the results on 10 selected worlds (datasets) published by (Sinha et al., 2020).
Revised: Table 4 shows the results on 10 selected worlds (datasets) published by (Sinha et al., 2020).

Original: When the ARL is larger, the dataset becomes noisier and contains more bad cases.
Revised: When the ARL is larger, the dataset becomes noisier and contains more bad cases.

Original: In terms of relation prediction accuracy, R5 signiﬁcantly outperforms both the embedding-based and rule induction baselines over all 10 datasets.
Revised: In terms of relation prediction accuracy, R5 significantly outperforms both the embedding-based and rule induction baselines over all 10 datasets.

Original: In terms of the ground rule recall rate, R5 also consistently outperforms CTP, and even reaches full recall rate on some datasets.
Revised: In terms of the ground rule recall rate, R5 also consistently outperforms CTP, and even reaches full recall rate on some datasets.

Original: At a closer inspection, the two GNN models perform poorly for ARL < 4 .
Revised: At a closer inspection, the two GNN models perform poorly for ARL < 4 .

Original: A dataset with a smaller
Revised: A dataset with a smaller

Original: ARL indicates the graphs are very simple, and models that represent the graph structure based on message-passing have not enough information to learn from.
Revised: ARL indicates the graphs are very simple, and models that represent the graph structure based on message-passing have not enough information to learn from.

Original: As for datasets with ARL > 4 , the graph structures become more complex, the GNNs are able to aggregate more information to each node from the neighbors.
Revised: As for datasets with ARL > 4 , the graph structures become more complex, the GNNs are able to aggregate more information to each node from the neighbors.

Original: Thus, the performance of GNNs improves on these datasets.
Revised: Thus, the performance of GNNs improves on these datasets.

Original: However,
Revised: However,

Original: CTP performs worse than GNNs in this case, due to two reasons.
Revised: CTP performs worse than GNNs in this case, due to two reasons.

Original: On one hand, the increasing amount of bad cases inﬂuences the rule recall rate, making CTP less conﬁdent on some correct rules.
Revised: On one hand, the increasing amount of bad cases influences the rule recall rate, making CTP less confident on some correct rules.

Original: On the other hand, it is hard for CTP to generalize to complex data.
Revised: On the other hand, it is hard for CTP to generalize to complex data.

Original: Table 5 presents the results for the three new datasets created that have fewer training samples.
Revised: Table 5 presents the results for the three new datasets created that have fewer training samples.

Original: In particular, resolution lengths of graphs in Re 0 can be up to 10, while resolution lengths of graphs in
Revised: In particular, resolution lengths of graphs in Re 0 can be up to 10, while resolution lengths of graphs in

Original: Re 1 can be up to 15.
Revised: Re 1 can be up to 15.

Original: Re 2 requires the models to train on graphs with resolution lengths of { 3 , ..., 5 } , and to test on graphs with resolution lengths of { 2 , ..., 7 } .
Revised: Re 2 requires the models to train on graphs with resolution lengths of { 3 , ..., 5 } , and to test on graphs with resolution lengths of { 2 , ..., 7 } .

Original: As shown in the table, R5 achieves better results than CTP when the amount of training facts is very small.
Revised: As shown in the table, R5 achieves better results than CTP when the amount of training facts is very small.

Original: And we can observe from Reand Re 1 that as the resolution length becomes longer, CTP has a sharp performance drop on the prediction accuracy, whereas R5 maintains relatively high accuracy and full ground rule recall rate. 
Revised: And we can observe from Reand Re 1 that as the resolution length becomes longer, CTP has a sharp performance drop on the prediction accuracy, whereas R5 maintains relatively high accuracy and full ground rule recall rate. [Table]

Original: In the train set of Re 2, there is no simple data.
Revised: In the train set of Re [Table] there is no simple data.

Original: Simple data refers to a sample whose resolution[Table] length is
Revised: Simple data refers to a sample whose resolution length is

Original: [Table]
Revised: [Table]

Original: and the ground rules can be naturally collected from such data.
Revised: and the ground rules can be naturally collected from such data.

Original: However, in Re [Table] as we train on samples of lengths { 3 , ..., 5 } and evaluate on samples of lengths { 2 , ..., 7 } , the models have to infer the ground rules.
Revised: However, in Re 2, as we train on samples of lengths { 3 , ..., 5 } and evaluate on samples of lengths { 2 , ..., 7 } , the models have to infer the ground rules.

Original: As is shown, R5 can still extract most of the ground rules while CTP fails [Table]
Revised: As is shown, R5 can still extract most of the ground rules while CTP fails.

Original: 3. A BLATION S TUDIES
Revised: 3. A BLATION S TUDIES

Original: [Table]
Revised: [Table]

Original: Policy Value Network.
Revised: Policy Value Network.

Original: As stated earlier, the policy value network is trained to sequentially decide which edge pair we should select to do a deduction.
Revised: As stated earlier, the policy value network is trained to sequentially decide which edge pair we should select to do a deduction.

Original: Without training this network, only
Revised: Without training this network, only

Original: MCTS will help us to make the decisions, whose action distribution is given by the numbers of visits at all possible nodes.
Revised: MCTS will help us to make the decisions, whose action distribution is given by the numbers of visits at all possible nodes.

Original: From Table 6 we acquire that although we are still able to recall the ground rules with rules extraction, the prediction accuracy drastically decreases since the paths are not deducted in the correct order.8 Training Epoch 0. T e s t A cc u r a c y
Revised: From Table 6 we acquire that although we are still able to recall the ground rules with rules extraction, the prediction accuracy drastically decreases since the paths are not deducted in the correct order.

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Pre-allocated Unknown Relations.
Revised: Pre-allocated Invented Relations.

Original: We preallocate n unknown relations to allow a relation pair to deduct to an intermediate predicate when it still is unclear whether the relation pair can give us a correct rule.
Revised: We preallocate n invented relations to allow a relation pair to deduct to an intermediate predicate when it still is unclear whether the relation pair can give us a correct rule.

Original: And the intermediate predicate will be tracked during the training until it is identiﬁed as a known relation.
Revised: And the intermediate predicate will be tracked during the training until it is identified as a known relation.

Original: Refer to Table 6, when we only allow a single unknown relation, the prediction accuracy will drop, which indicates that the extra hidden relations improve the model’s capability of learning complicated graphs.
Revised: Refer to Table 6, when we only allow a single invented relation, the prediction accuracy will drop, which indicates that the extra hidden relations improve the model’s capability of learning complicated graphs.

Original: And in Re 2, where the training samples’ resolution lengths are greater than 2, the results catastrophically decrease.
Revised: And in Re 2, where the training samples’ resolution lengths are greater than 2, the results catastrophically decrease.

Original: We evaluate the prediction accuracy and rules recall each training epoch.
Revised: We evaluate the prediction accuracy and rules recall each training epoch.

Original: As shown in Figure 3, the model loses the induction ability at the early stage with only one intermediate predicate allowed.
Revised: As shown in Figure 3, the model loses the induction ability at the early stage with only one intermediate predicate allowed.

Original: Once the ﬁrst ground rule is extracted, more will be gradually learned, and ﬁnally the R5 with a single hidden predicate reaches the same rules recall rate as R5 with 50 hidden predicates.
Revised: Once the first ground rule is extracted, more will be gradually learned, and finally the R5 with a single hidden predicate reaches the same rules recall rate as R5 with 50 hidden predicates.

Original: This ﬁgure proves that a number of pre-allocated unknown relations can not only help to learn complicated graphs, but also speed up the convergence.
Revised: This figure proves that a number of pre-allocated invented relations can not only help to learn complicated graphs, but also speed up the convergence.

Original:  R ELATED W ORKS AND D ISCUSSIONS
Revised:  R ELATED W ORKS AND D ISCUSSIONS

Original: Graph Neural Networks and General Relation Prediction Tasks
Revised: Graph Neural Networks and General Relation Prediction Tasks.

Original: Graph Convolutional Networks (GCNs) (Bruna et al., 2014; Li et al., 2016; Defferrard et al., 2016; Kipf & Welling, 2017) is a family of message-passing neural architectures designed for processing graph data.
Revised: Graph Convolutional Networks (GCNs) (Bruna et al., 2014; Li et al., 2016; Defferrard et al., 2016; Kipf & Welling, 2017) is a family of message-passing neural architectures designed for processing graph data.

Original: While most
Revised: While most

Original: GCNs operate only with the entities and binary links, some recent works (Schlichtkrull et al., 2018;
Revised: GCNs operate only with the entities and binary links, some recent works (Schlichtkrull et al., 2018;

Original: Sinha et al., 2020; Pan et al., 2020) are applicable on relational data where the relation types (i.e., edge types) between entities (i.e., vertices) matters.
Revised: Sinha et al., 2020; Pan et al., 2020) are applicable on relational data where the relation types (i.e., edge types) between entities (i.e., vertices) matters.

Original: The GCN-based models can capture lifted rules deﬁned on the sets of entities and have presented good results on the node classiﬁcation tasks (Sinha et al., 2020).
Revised: The GCN-based models can capture lifted rules defined on the sets of entities and have presented good results on the node classification tasks (Sinha et al., 2020).

Original: However, since the lifted rules are implicitly encapsulated in the neural networks, these models are lack of interpretability.
Revised: However, since the lifted rules are implicitly encapsulated in the neural networks, these models are lack of interpretability.

Original: ILP, Neural-Symbolic Learning and Systematicity.
Revised: ILP, Neural-Symbolic Learning and Systematicity.

Original: Inductive Logic Programming (ILP) (Muggleton, 1992; Muggleton et al., 1996; Getoor, 2000; Evans & Grefenstette, 2018) is a subﬁeld of symbolic rule learning, which learns logic rules derived from a limited and predeﬁned set of rule templates.
Revised: Inductive Logic Programming (ILP) (Muggleton, 1992; Muggleton et al., 1996; Getoor, 2000; Evans & Grefenstette, 2018) is a subfield of symbolic rule learning, which learns logic rules derived from a limited and predefined set of rule templates.

Original: The general idea behind a traditional ILP system is to learn a set of rules from a collection of examples, which entails all the positives and none of the negatives.
Revised: The general idea behind a traditional ILP system is to learn a set of rules from a collection of examples, which entails all the positives and none of the negatives.

Original: Such a system naturally provides interpretable explanations for labels prediction as well as being able to generalize to other tasks but is hard to scale up due to the exponentially increased search space of compositional rules.
Revised: Such a system naturally provides interpretable explanations for labels prediction as well as being able to generalize to other tasks but is hard to scale up due to the exponentially increased search space of compositional rules.

Original: In contrast, R5 preserves the capability of systematicity (Hupkes et al., 2020), learns only the short predicates in the format of u ← p i , p j , and the long predicates are represented by the composition of short ones.
Revised: In contrast, R5 preserves the capability of systematicity (Hupkes et al., 2020), learns only the short predicates in the format of u ← p i , p j , and the long predicates are represented by the composition of short ones.

Original: For instance, ˆ p ← p 0 , p i , p j can be decomposed into ˆ p ← p 0 , u and u ← p i , p j .
Revised: For instance, ˆ p ← p 0 , p i , p j can be decomposed into ˆ p ← p 0 , u and u ← p i , p j .

Original: Thus, there is no need to consider all the predicates following a set of rule templates.
Revised: Thus, there is no need to consider all the predicates following a set of rule templates.

Original: As shown in Section 2.1 and Figure 2, the parameter size of the policy and value network of R5 grows polynomially with respect to the number of allowed predicates/relations, compared with the exponential dependence in ∂ ILP (Evans & Grefenstette, 2018).
Revised: As shown in Section 2.1 and Figure 2, the parameter size of the policy and value network of R5 grows polynomially with respect to the number of allowed predicates/relations, compared with the exponential dependence in ∂ ILP (Evans & Grefenstette, 2018).

Original: Neural-Symbolic models (Garcez et al., 2015; Besold et al., 2017) seek to integrate principles from neural networks learning and logical reasoning, among which Neural Logic Machines (NLMs) (Dong et al., 2018;
Revised: Neural-Symbolic models (Garcez et al., 2015; Besold et al., 2017) seek to integrate principles from neural networks learning and logical reasoning, among which Neural Logic Machines (NLMs) (Dong et al., 2018;

Original: Zimmer et al., 2021) and Theorem Provers (Rockt¨aschel & Riedel, 2017; Minervini et al., 2020a;b) attempt to overcome the problems of systematicity.
Revised: Zimmer et al., 2021) and Theorem Provers (Rockt¨aschel & Riedel, 2017; Minervini et al., 2020a;b) attempt to overcome the problems of systematicity.

Original: Such systems successfully show the systematicity for up to 50 entities on some selected decision-making tasks.
Revised: The NLMs successfully show the systematicity for up to 50 entities on some selected decision-making tasks.

Original: However, in terms of relational reasoning on graphs, NLMs only consider the binary classiﬁcation tasks .
Revised: However, in terms of relational reasoning on graphs, NLMs only consider the binary classification tasks .

Original: Although the links in the graphs have various types, for each task, NLMs predict the existence of a single relation type instead of concluding which type the relation is when there is one.
Revised: Although the links in the graphs have various types, for each task, NLMs predict the existence of a single relation type instead of concluding which type the relation is when there is one.

Original: When performing general relation prediction, NLMs have to create a number of tasks where each task predicts for a single relation type, making the total number of tasks grows as the number of relation types increases.
Revised: When performing general relation prediction, NLMs have to create a number of tasks where each task predicts for a single relation type, making the total number of tasks grows as the number of relation types increases.

Original: As a result, NLMs are hard to scale up on relation prediction tasks.
Revised: As a result, NLMs are hard to scale up on relation prediction tasks.

Original: Theorem Provers, in particular the Conditional Theorem Provers (CTPs) (Minervini et al., 2020a;b), are capable of the relation prediction problem without the need to create a number of tasks.
Revised: Theorem Provers, in particular the Conditional Theorem Provers (CTPs) (Minervini et al., 2020a;b), are capable of the relation prediction problem without the need to create a number of tasks.

Original: CTPs use templates equipped with a variety of selection modules to reformulate a given goal into subgoals, and jointly learn representations and rules from data.
Revised: CTPs use templates equipped with a variety of selection modules to reformulate a given goal into subgoals, and jointly learn representations and rules from data.

Original: However, the depth of the templates (refer to hops in their code) is limited, which inﬂuences the number of parameters of the model.
Revised: However, the depth of the templates (refer to hops in their code) is limited, which influences the number of parameters of the model.

Original: In contrast, as explained in Section 2.1, R5 is able to handle the complicated rules where the allowed rule depth is much larger.
Revised: In contrast, as explained in Section 2.1, R5 is able to handle the complicated rules where the allowed rule depth is much larger.

Original: That is because the rule depth is only related to the number of steps in an episode of the sequential decision-making process of R5, which does not touch the number of model parameters.
Revised: That is because the rule depth is only related to the number of steps in an episode of the sequential decision-making process of R5, which does not touch the number of model parameters.

Original: We have also presented in Section 3.1 and 3.2 that R5 gives better relation prediction results on the systematicity tasks and shows better performance in terms of the rules recall.
Revised: We have also presented in Section 3.1 and 3.2 that R5 gives better relation prediction results on the systematicity tasks and shows better performance in terms of the rules recall.

Original: Relational Reasoning for Knowledge Base Completion.
Revised: Relational Reasoning for Knowledge Base Completion.

Original: Many recent works propose a variety of relational reasoning models for efﬁcient Knowledge Base Completion (KBC), which differs from what we are focusing on.
Revised: Many recent works propose a variety of relational reasoning models for efficient Knowledge Base Completion (KBC), which differs from what we are focusing on.

Original: These works include the embedding-based methods (Sun et al., 2019;
Revised: These works include the embedding-based methods (Sun et al., 2019;

Original: Trouillon et al., 2017; Dettmers et al., 2017; Bordes et al., 2013), GNN-like methods (Teru et al., 2020; Mai et al., 2020), Neural-Symbolic Methods (Rockt ¨ aschel & Riedel, 2017; Minervini et al., 2020a;b; Qu et al., 2021), and RL Path-ﬁnding Agents (Xiong et al., 2017; Chen et al., 2018; Das et al., 2017; Lin et al., 2018; Shen et al., 2018).
Revised: Trouillon et al., 2017; Dettmers et al., 2017; Bordes et al., 2013), GNN-like methods (Teru et al., 2020; Mai et al., 2020), Neural-Symbolic Methods (Rockt ¨ aschel & Riedel, 2017; Minervini et al., 2020a;b; Qu et al., 2021), and RL Path-finding Agents (Xiong et al., 2017; Chen et al., 2018; Das et al., 2017; Lin et al., 2018; Shen et al., 2018).

Original: Most of them are designed for the transductive aspects instead of systematicity.
Revised: Most of them are designed for the transductive aspects instead of systematicity.

Original: Moreover, the KBC tasks can be treated as entity sorting problems, where the objective is ranking the tail entities given a query ( h, r, ?) with head entity h and the relation r are given (Wang et al., 2021).
Revised: Moreover, the KBC tasks can be treated as entity sorting problems, where the objective is ranking the tail entities given a query ( h, r, ?) with head entity h and the relation r are given (Wang et al., 2021).

Original: KBC is different from the general relation prediction tasks investigated in this paper.
Revised: KBC is different from the general relation prediction tasks investigated in this paper.

Original: It is also non-trivial to extend our research to tackle KBC tasks.
Revised: It is also non-trivial to extend our research to tackle KBC tasks.

Original: Since the relation r is given in the query, rather than deciding the relation type between a head entity h and a candidate tail t * like what R5 might do, a better choice would be directly determining whether the single relation r exists between h and t * like what NLMs might be able to do.
Revised: Since the relation r is given in the query, rather than deciding the relation type between a head entity h and a candidate tail t * like what R5 might do, a better choice would be directly determining whether the single relation r exists between h and t * like what NLMs might be able to do.

Original: Therefore, we do not discuss R5’s performance on KBC tasks in this paper.
Revised: Therefore, we do not discuss R5’s performance on KBC tasks in this paper.

Original: Although non-trivial, surely it is worth trying to extend the main structure of R5 to KB.
Revised: Although non-trivial, surely it is worth trying to extend the main structure of R5 to KB.

Original: We leave this task to future works.
Revised: We leave this task to future works.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose R5, a framework that models relational reasoning as sequential decision-making.
Revised: We propose R5, a framework that models relational reasoning as sequential decision-making.

Original: Our framework performs explicit and explainable relational reasoning with a design of state and action spaces based on pairs of relations.
Revised: Our framework performs explicit and explainable relational reasoning with a design of state and action spaces based on pairs of relations.

Original: Besides, R5 performs rule induction with a dynamic rule memory module, which is updated during the training of an agent.
Revised: Besides, R5 performs rule induction with a dynamic rule memory module, which is updated during the training of an agent.

Original: We further incorporate hidden (unknown) relations in our state and action spaces to help accelerate model training as well as improve rule mining.
Revised: We further incorporate hidden (invented) relations in our state and action spaces to help accelerate model training as well as improve rule mining.

Original: R5 exhibits high accuracy for relation prediction and a high recall rate for rule discovery, as has been demonstrated by extensive experimental results.
Revised: R5 exhibits high accuracy for relation prediction and a high recall rate for rule discovery, as has been demonstrated by extensive experimental results.

Original: We also show that R5 has a strong ability of systematicity and is robust to data noise.
Revised: We also show that R5 has a strong ability of systematicity and is robust to data noise.

Original: S UPPLEMENTARY M ATERIALS A D ATASET S TATISTICS AND I MPLEMENTATION D ETAILS
Revised: S UPPLEMENTARY M ATERIALS A D ATASET S TATISTICS AND I MPLEMENTATION D ETAILS

Original: We have introduced R5 in Section 3.
Revised: We have introduced R5 in Section 3.

Original: Due to the limited space, some implementation details are not covered.
Revised: Due to the limited space, some implementation details are not covered.

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we explain the parameterization and implementation details of R
Revised: we explain the parameterization and implementation details of R

Original: [Table]
Revised: [Table]

Original: A. D ATASET S TATISTICS
Revised: A. D ATASET S TATISTICS

Original: The statistics of CLUTRR are summarized in Table 7, and the statistics of GraphLog are summarized in Table
Revised: The statistics of CLUTRR are summarized in Table 7, and the statistics of GraphLog are summarized in Table

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: We use the following hyperparameters for all the datasets: epochs = 10, learning rate = 0.01, n = 50, (cid:15) = 0.003, v 0 = 0.6, v 1 = 0.3, v 2 = -0.05, v 3 = -0.1, v 4 =
Revised: We use the following hyperparameters for all the datasets: epochs = 10, learning rate = 0.01, n = 50, ϵ = 0.003, v 0 = 0.6, v 1 = 0.3, v 2 = -0.05, v 3 = -0.1, v 4 =

Original: -0.3, v T neg
Revised: -0.3, v T neg

Original: = -1, σ = -1.2, and v T pos = 0.
Revised: = -1, σ = -1.2, and v T pos = 0.

Original: or 0.35 or 0.8 depending on the dataset we investigate.
Revised: or 0.35 or 0.8 depending on the dataset we investigate.

Original: For example, since there is no simple data in
Revised: For example, since there is no simple data in

Original: Re 2, we use v T pos = 0.8 instead to encourage R5 to memorize uncertain rules.
Revised: Re 2, we use v T pos = 0.8 instead to encourage R5 to memorize uncertain rules.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: We have mentioned the scoring scheme in Section 3.2.
Revised: We have mentioned the scoring scheme in Section 3.2.

Original: However, two optional hyperparameters are introduced, which can make sure the ground truth rules not be omitted due to large noise in datasets.
Revised: However, two optional hyperparameters are introduced, which can make sure the ground truth rules not be omitted due to large noise in datasets.

Original: Recall that simple data are samples with a resolution length of 2, and the ground truth rules can naturally be collected from such data.
Revised: Recall that simple data are samples with a resolution length of 2, and the ground truth rules can naturally be collected from such data.

Original: To deduct such samples, R5 only needs one step.
Revised: To deduct such samples, R5 only needs one step.

Original: We give an extra simple true reward v true to the action to consolidate the corresponding rule in the memory.
Revised: We give an extra simple true reward v true to the action to consolidate the corresponding rule in the memory.

Original: The other optional hyperparameter is a simple wrong allowance v wrong .
Revised: The other optional hyperparameter is a simple wrong allowance v wrong .

Original: It can be applied when the rule collected from a simple sample is not in accordance with the existing rule in the rule memory.
Revised: It can be applied when the rule collected from a simple sample is not in accordance with the existing rule in the rule memory.

Original: Besides, to speed up training, we sort all the training samples by their maximum resolution paths, and train in the ascending order of the maximum resolution path in the ﬁrst epoch.
Revised: Besides, to speed up training, we sort all the training samples by their maximum resolution paths, and train in the ascending order of the maximum resolution path in the first epoch.

Original: Thus, some ground truth rules can be quickly collected from simple data at an early stage.
Revised: Thus, some ground truth rules can be quickly collected from simple data at an early stage.

Original: X
Revised: X

Original: Y r2 r0 r r2 r0 r
Revised: Y r2 r0 r r2 r0 r

Original: Path 1 Path 2 r2 r0 r2 r2 r0 r
Revised: Path 1 Path 2 r2 r0 r2 r2 r0 r

Original: Input Graph
Revised: Input Graph

Original: Paths between query nodes
Revised: Paths between query nodes

Original: Relation pairs to state 2 features
Revised: Relation pairs to state 2 features

Original: Number of occurrence r0 r1 r2 u r r1 r2 u

Revised: Number of occurrence r0 r1 r2 u r r1 r2 u

Original: Whether in the obtained rules r0 r1 r2 u r r1 r2 u
Revised: Whether in the obtained rules r0 r1 r2 u r r1 r2 u

Original: Body 

Revised: Body

Original: Body 1 Head r r u u r r
Revised: Body 1 Head r r u u r r

Original: Score 4.303 -0.
Revised: Score 4.303 -0.

Original: Rules Hash Table
Revised: Rules Hash Table

Original: Path 1 Path 2 u r2 u r
Revised: Path 1 Path 2 u r2 u r

Original: Paths between query nodes
Revised: Paths between query nodes

Original: Relation pairs to state 2 features
Revised: Relation pairs to state 2 features

Original: Number of occurrence r0r1 r2 u r r1 r2 u 

Revised: Number of occurrence r0r1 r2 u r r1 r2 u

Original: Whether in the obtained rules r0 r1 r2 u r r1 r2 u
Revised: Whether in the obtained rules r0 r1 r2 u r r1 r2 u

Original: Action: (r2,r0) -> u
Revised: Action: (r2,r0) -> u

Original: A. A N E XAMPLE OF S TATE T RANSITION IN THE D ECISION -M AKING P ROCESS
Revised: A. A N E XAMPLE OF S TATE T RANSITION IN THE D ECISION -M AKING P ROCESS

Original: Figure 4 gives a trivial example to illustrate how the state is constructed and the state transition after applying an action.
Revised: Figure 4 gives a trivial example to illustrate how the state is constructed and the state transition after applying an action.

Original: As shown in Figure 2 and Section 2.1, the state represents ( m + n )( m + n ) relation pairs and k features.
Revised: As shown in Figure 2 and Section 2.1, the state represents ( m + n )( m + n ) relation pairs and k features.

Original: In this trivial example, we consider m = 3 , n = 1 and k = 2 for a reasoning task, which means there are m = 3 known relations, and n = 1 allocated unknown relation, thus in total 4 × 4 = 16 relation pairs.
Revised: In this trivial example, we consider m = 3 , n = 1 and k = 2 for a reasoning task, which means there are m = 3 known relations, and n = 1 allocated invented relation, thus in total 4 × 4 = 16 relation pairs.

Original: We also consider the two features to be the number of occurrences and whether the relation pair is in the obtained rules .
Revised: We also consider the two features to be the number of occurrences and whether the relation pair is in the obtained rules .

Original: Please be noted that we only compute the state features for the relation pairs in the paths.
Revised: Please be noted that we only compute the state features for the relation pairs in the paths.

Original: This is why u, r 1 → r 0 is in the rules hash table
Revised: This is why u, r 1 → r 0 is in the rules hash table

Original: but it is not counted when deciding whether a relation pair is in the obtained rules before taking the action.
Revised: but it is not counted when deciding whether a relation pair is in the obtained rules before taking the action.

Original: A. H ANDLING ˆ p ← p i AND H YPER -N ETWORKS
Revised: A. E XTENDING TO C OMPLEX D OMAINS

Revised: Handling ˆ p ← p i .

Original: In Section 2 we explained that R5 realizes Horn clauses by ﬁrstly ﬁnding paths based on the query, and then omitting the entities and represent the long clauses by the composition of short clauses u ← p i ∧ p j .
Revised: In Section 2 we explained that R5 realizes Horn clauses by firstly finding paths based on the query, and then omitting the entities and represent the long clauses by the composition of short clauses u ← p i ∧ p j .

Original: We want to make a note here R5 is also able to represent the very short clause ˆ p ← p i by adding a dummy relation r dm to the state described in Section 2.1, and the corresponding predicates will be p dm .
Revised: We want to make a note here R5 is also able to represent the very short clause ˆ p ← p i by adding a dummy relation r dm to the state described in Section 2.1, and the corresponding predicates will be p dm .

Original: Thus, ˆ p ← p i can be represented by ˆ p ← p i
Revised: Thus, ˆ p ← p i can be represented by ˆ p ← p i

Original: ∧ p dm .
Revised: ∧ p dm .

Original: This has been implemented in the code.
Revised: This has been implemented in the code.

Revised: Hyper-Networks.

Revised: In this paper, we assume all rules are using the dyadic chain meta-rule template (Muggleton et al., 2015).

Original: In addition, R5 has the potency to handle hyper-networks.
Revised: However, R5 has the potential to handle hyper-networks.

Original: A hyper-network is formed by hyperedges, where a hyperedge refers to the relationship among data points that could go beyond pairwise, i.e., three or more objects are involved in each relationship.
Revised: A hyper-network is formed by hyperedges, where a hyperedge refers to the relationship among data points that could go beyond pairwise, i.e., three or more objects are involved in each relationship.

Original: (Tu et al., 2018)
Revised: (Tu et al., 2018)

Original: Since R5 does not consider entities after paths are sampled, it naturally has the ability to handle hyperedges.
Revised: Since R5 does not consider entities after paths are sampled, it naturally has the ability to handle hyperedges.

Original: For example, given two hyperedges r 0 ( x 0 , x 1 , x 2 ) and r 1 ( x 1 , x 2 , x 7 ) , where x i are entities, part of the path can be r 0 , r 1 .
Revised: For example, given two hyperedges r 0 ( x 0 , x 1 , x 2 ) and r 1 ( x 1 , x 2 , x 7 ) , where x i are entities, part of the path can be r 0 , r 1 .

Original: However, hyperedges and hyper-networks is a different topic, which is not what we focus on in this paper.
Revised: However, hyperedges and hyper-networks is a different topic, which is not what we focus on in this paper.

Original: We left this topic as future work.
Revised: We left this topic as future work.

Original: B A UGMENTING R5 WITH R-GCN
Revised: B A UGMENTING R5 WITH R-GCN

Original: R5 makes predictions relying on the obtained rules.
Revised: R5 makes predictions relying on the obtained rules.

Original: Different from the embedding-based methods, which can always return a relation type with the least loss, it is possible that R5 cannot give a known relation as the prediction with its logical knowledge.
Revised: Different from the embedding-based methods, which can always return a relation type with the least loss, it is possible that R5 cannot give a known relation as the prediction with its logical knowledge.

Original: During testing, R5 may fail to ﬁnd the body of a deduction step in the memory, or ﬁnally deduce into an unknown relation.
Revised: During testing, R5 may fail to find the body of a deduction step in the memory, or finally deduce into an invented relation.

Original: Neither is able to give a valid prediction.
Revised: Neither is able to give a valid prediction.

Original: We deﬁne the invalid prediction ratio as the ratio that R5 fails to return a known relation.
Revised: We define the invalid prediction ratio as the ratio that R5 fails to return a known relation.

Original: [Table]
Revised: [Table]

Original: When a human is not able to make decisions based on logic, he may need the help of intuition .
Revised: When a human is not able to make decisions based on logic, he may need the help of intuition .

Original: Similarly, we combine R5 with R-GCN, a GNN-based model, which is not able to explicitly reason to make predictions (similar to a human’s intuition), to see if the results can be further improved when R5 is complemented by a GNN.
Revised: Similarly, we combine R5 with R-GCN, a GNN-based model, which is not able to explicitly reason to make predictions (similar to a human’s intuition), to see if the results can be further improved when R5 is complemented by a GNN.

Original: Besides, R-GCN takes node embeddings into account, which may help to improve the predictions that require node properties.
Revised: Besides, R-GCN takes node embeddings into account, which may help to improve the predictions that require node properties.

Original: The R-GCN model is trained with the following hyperparameters: a learning rate of 0.001, a node/relation embedding dimension of 200, 20 training epochs, 4 layers, and a dropout rate of 0.1.
Revised: The R-GCN model is trained with the following hyperparameters: a learning rate of 0.001, a node/relation embedding dimension of 200, 20 training epochs, 4 layers, and a dropout rate of 0.1.

Original: Table 9 shows the results of augmenting R5 using R-GCN.
Revised: Table 9 shows the results of augmenting R5 using R-GCN.

Original: And in the R5+R-GCN model, when R fails to give a valid prediction, we use R-GCN to predict the target relation.
Revised: And in the R5+R-GCN model, when R fails to give a valid prediction, we use R-GCN to predict the target relation.

Original: The invalid prediction ratios of R5 for Re 1, Re 2, and World 32 are 0.26, 0.434, and 0.085 respectively.
Revised: The invalid prediction ratios of R5 for Re 1, Re 2, and World 32 are 0.26, 0.434, and 0.085 respectively.

Original: As presented in the table, the prediction accuracy can further be improved by augmenting R5 with R-GCN, i.e., the accuracies improve by 0.05, 0.212, and 0.048 on the three datasets, respectively, indicating that the reasoning provided by R5 and GNN-based approaches are complementary to each other.
Revised: As presented in the table, the prediction accuracy can further be improved by augmenting R5 with R-GCN, i.e., the accuracies improve by 0.05, 0.212, and 0.048 on the three datasets, respectively, indicating that the reasoning provided by R5 and GNN-based approaches are complementary to each other.

Original: C R ESULTS ON CLUTRR WITH H YPERPARAMETERS F INE - TUNED ON L ONG S TORIES
Revised: C R ESULTS ON CLUTRR WITH H YPERPARAMETERS F INE - TUNED ON L ONG S TORIES

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D C ASE S TUDIES
Revised: D C ASE S TUDIES

Original: Table 12 and Table 13 present some examples of the recurrent relational reasoning process by R5 on CLUTRR and GraphLog datasets.
Revised: Table 12 and Table 13 present some examples of the recurrent relational reasoning process by R on CLUTRR and GraphLog datasets.

Original: For veriﬁcation, GraphLog provides the ground truth rules sets used to generate each dataset, and in Tables 13 and 14 we mark such ground truth rules in bold.
Revised: For verification, GraphLog provides the ground truth rules sets used to generate each dataset, and in Tables 13 and 14 we mark such ground truth rules in bold.

Original: As presented, R5 is able to discover the ground truth rules, and learn long compositional rules.
Revised: As presented, R5 is able to discover the ground truth rules, and learn long compositional rules.

Original: In the second instance in Table 13, we have an unknown relation unknown 44 .
Revised: In the second instance in Table 13, we have an invented relation unknown 44 .

Original: With this intermediate relation introduced, we can get a clause with three atoms in its body, i.e., r 6 ← (r 9,r 3,r 9) .
Revised: With this intermediate relation introduced, we can get a clause with three atoms in its body, i.e., r 6 ← (r 9,r 3,r 9) .

Original: Table 14 shows some examples that R5 falsely predicts the queried relations.
Revised: Table 14 shows some examples that R5 falsely predicts the queried relations.

Original: In the ﬁrst example, both rules used are in the set of ground truth rules, leading to a prediction of r 3 between the queried pair, which is different from the target relation r 13 provided in the test set.
Revised: In the first example, both rules used are in the set of ground truth rules, leading to a prediction of r 3 between the queried pair, which is different from the target relation r 13 provided in the test set.

Original: Furthermore, we ﬁnd that there is no ground truth rule in this world to allow the prediction of r 13 by human deduction.
Revised: Furthermore, we find that there is no ground truth rule in this world to allow the prediction of r 13 by human deduction.

Original: Neither can R5 deduct r 13 .
Revised: Neither can R5 deduct r 13 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In the second example, all the three rules learned by R5 are ground truth rules, enabling us to predict r 18 from the resolution path [r 8,r 15,r 8,r 4] .
Revised: In the second example, all the three rules learned by R5 are ground truth rules, enabling us to predict r 18 from the resolution path [r 8,r 15,r 8,r 4] .

Original: Yet, r 18 , although correct, does not exactly match the target relation r 5 provided by the test set.
Revised: Yet, r 18 , although correct, does not exactly match the target relation r 5 provided by the test set.

Original: We note that there is not any ground truth rule provided by the dataset whose head is r 5 .
Revised: We note that there is not any ground truth rule provided by the dataset whose head is r 5 .

Original: However, R5 has also learnt an additional rule from the training samples headed to r 5 :
Revised: However, R5 has also learnt an additional rule from the training samples headed to r 5 :

Original: [Equation]
Revised: [Equation]

Original: which is not in the set of ground truth rules.
Revised: which is not in the set of ground truth rules.

Original: With Rule (5), we have another sequence of deductions that can lead to r 5 from the resolution path [r 8,r 15,r 8,r 4] :
Revised: With Rule (5), we have another sequence of deductions that can lead to r 5 from the resolution path [r 8,r 15,r 8,r 4] :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst two rules in bold are in the ground truth rules, while the third is not.
Revised: where the first two rules in bold are in the ground truth rules, while the third is not.

Original: Since there are two valid relation types r 5 and r 18 between the queried node pair, R5 may not be able to tell which one should be selected without knowledge of the entities.
Revised: Since there are two valid relation types r 5 and r 18 between the queried node pair, R5 may not be able to tell which one should be selected without knowledge of the entities.

Original: This is an example where the entity properties may help with logical inference.
Revised: This is an example where the entity properties may help with logical inference.

Original: E D ISCUSSIONS ON S OME KB REASONING WORKS .
Revised: E D ISCUSSIONS ON S OME KB R EASONING W ORKS .

Original: As discussed in Section 4, KB reasoning is not what we focus on.
Revised: As discussed in Section 4, KB reasoning is not what we focus on.

Original: However, R5 has some strength compared with some of the existing KB relational learning works.
Revised: However, R5 has some strength compared with some of the existing KB relational learning works.

Original: Yang et al.
Revised: Yang et al.

Original: (2017) learns the rules in the form query(Y,X) ← R n (Y,Z n ) ∧ · · · ∧ R 1 ( Z 1 , X ) , where each predicate strictly contains 2 entities.
Revised: (2017) learns the rules in the form query(Y,X) ← R n (Y,Z n ) ∧ · · · ∧ R 1 ( Z 1 , X ) , where each predicate strictly contains 2 entities.

Original: Whereas, R5 is able to consider predicates with any number of entities as explained in
Revised: Whereas, R5 is able to consider predicates with any number of entities as explained in

Original: Appendix A.5.
Revised: Appendix A.5.

Original: Moreover, many RL Path-ﬁnding Agents (Xiong et al., 2017; Chen et al., 2018; Das et al., 2017; Lin et al., 2018; Shen et al., 2018) are proposed for KB reasoning.
Revised: Moreover, many RL Path-finding Agents (Xiong et al., 2017; Chen et al., 2018; Das et al., 2017; Lin et al., 2018; Shen et al., 2018) are proposed for KB reasoning.

Original: The major challenges for these works include the sparse rewards.
Revised: The major challenges for these works include the sparse rewards.

Original: For example, at each step of decision-making, Shen et al.
Revised: For example, at each step of decision-making, Shen et al.

Original: (2018) selects neighbouring edges at the current node, and follows the edges to the next node.
Revised: (2018) selects neighbouring edges at the current node, and follows the edges to the next node.

Original: Termination occurs when the agent gives a “STOP” action instead of selecting the next neighbouring edge.
Revised: Termination occurs when the agent gives a “STOP” action instead of selecting the next neighbouring edge.

Original: The termination criterion is unclear, which is one of the important reasons for the sparse reward.
Revised: The termination criterion is unclear, which is one of the important reasons for the sparse reward.

Original: In contrast, the MCTS in R5 guarantees that only the feasible relation pairs can be selected.
Revised: In contrast, the MCTS in R5 guarantees that only the feasible relation pairs can be selected.

Original: Termination occurs when reduced to a single relation.
Revised: Termination occurs when reduced to a single relation.

Original: The criterion is clear.
Revised: The criterion is clear.

Original: However, R5 faces some challenges to generalize to KB reasoning.
Revised: However, R5 faces some challenges to generalize to KB reasoning.

Original: Apart from the ones mentioned in Section 4, currently, R5 does not generate probabilities or scores for a prediction, which is not capable of KB Completion since KB Completion is actually a set of ranking tasks.
Revised: Apart from the ones mentioned in Section 4, currently, R5 does not generate probabilities or scores for a prediction, which is not capable of KB Completion since KB Completion is actually a set of ranking tasks.

Original: Besides, R requires pre-sampling for the paths that entails the query.
Revised: Besides, R requires pre-sampling for the paths that entails the query.

Original: Since all the triples in a Knowledge Graph share the same training graph , which is usually very large, it will take a long time to enumerate all the paths.
Revised: Since all the triples in a Knowledge Graph share the same training graph , which is usually very large, it will take a long time to enumerate all the paths.

Original: If we choose not to perform enumeration and sample some paths instead, the sampling method is also a point that needs to be investigated.
Revised: If we choose not to perform enumeration and sample some paths instead, the sampling method is also a topic that needs to be investigated.

Original: Due to these challenges, we leave the application to KB as future work.
Revised: Due to these challenges, we leave the application to KB as future work.

Revised: F C OMPARISON WITH N EURAL -LP AND RNNL OGIC

Revised: Neural-LP

Revised: (Yang et al., 2017) is a neural logic programming method and RNNLogic (Qu et al., 2021) is a principled probabilistic method, where both of the two approaches are designed for Knowledge

Revised: Base Completion (KBC) problems.

Revised: These methods requires enumeration of all possible rules given a max rule length T .

Revised: Thus, the complexity of these models grows exponentially as max rule length increases, which is an outstanding disadvantage for systematicity problems.

Revised: [Table]

Revised: We tweaked the code of RNNLogic, which was originally designed for entity ranking, to predict and rank relation r given ( h, ? , t ) on GraphLog.

Revised: The results are shown in Table 15.

Revised: One issue we observed was that we were only able to run RNNLogic with max rule length T up to 7 on a powerful server, beyond which there was out-of-memory issue.

Revised: (For instance, setting T = 6 for RNNLogic on GraphLog Re 0 requires more than 130 G memory.)

Revised: We report its best result in Table 15, which is achieved when T = 5 .

Revised: It is shown that R5 substantially outperforms RNNLogic, which is in fact not designed for this task.

Revised: Similarly we revised NeurlLP to make it work for the relation prediction prediction task.

Revised: But unfortunately, it didn’t work well with a relation prediction accuracy of less than 10%.

Revised: G E XTRA F EATURES TO R ECORD THE M AX /M IN N UMBER OF O CCURRENCES AMONG P ATHS .

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: H N UMBER OF P RE -A LLOCATED I NVENTED R ELATIONS

Revised: [Figure]

Revised: [Figure]

Revised: In this paper, we have introduced n pre-allocated invented relations to capture the intermediate short rules during the decision-making procedure.

Revised: For example, r 1 , r 2 , r 3 → r can be decomposed into ( r 1 , r 2 → invented 0 ) , ( invented 0 , r 3 → r ) .

Revised: The details can be found in Section 2.2.

Revised: To choose n , we conduct experiments on two GraphLog datasets, World 17 and Re 0, as n varies.

Revised: We use the parameters listed in Appendix A.2, except that 5 training epochs are used instead of 10 to speed up training.

Revised: As presented in Figure 5 and Figure 6, R5 is able to get a near optimal performance even when n is small.

Revised: But as n becomes larger, there is a general trend that the performance can be more stable.

Revised: Intuitively, a small n makes the dynamic memory replace the invented (unknown) relations more frequently as new invented relations appear, slowing down the convergence (as we discussed in Section 3.3).

Revised: Therefore, in our experiments, as computational resources permit, we set n to befor all the datasets.

Revised: However, if a small n is required in practice, we can also use a validation set to select some n that is sufficient for each specific dataset.