Original: Explainable Pathﬁnding for Inscrutable Planners with Inductive Logic Programming
Revised: Explainable Pathﬁnding for Inscrutable Planners with Inductive Logic Programming

Original: Abstract
Revised: Abstract

Original: The complexity of the solutions that artiﬁcial intelli- gence can learn to solve problems currently surpasses its ability to explain these solutions.
Revised: The complexity of the solutions that artiﬁcial intelli- gence can learn to solve problems currently surpasses its ability to explain these solutions.

Original: In many domains, explainable solutions are a necessary condition while optimality is not.
Revised: In many domains, explainable solutions are a necessary condition while optimality is not.

Original: Therefore, we seek to constrain solu- tions to the space of solutions that can be explained to a human.
Revised: Therefore, we seek to constrain solu- tions to the space of solutions that can be explained to a human.

Original: To do this, we build on inductive logic pro- gramming (ILP) techniques that allow us to deﬁne robust background knowledge and inductive biases.
Revised: To do this, we build on inductive logic pro- gramming (ILP) techniques that allow us to deﬁne robust background knowledge and inductive biases.

Original: By combining ILP with a given inscrutable planner, we are able to construct an explainable graph representing so- lutions to all states in the state space.
Revised: By combining ILP with a given inscrutable planner, we are able to construct an explainable graph representing so- lutions to all states in the state space.

Original: This graph can then be summarized using a variety of methods such as hierarchical representations and simple if/else rules.
Revised: This graph can then be summarized using a variety of methods such as hierarchical representations and simple if/else rules.

Original: We test our approach on Towers of Hanoi and discuss future work for applications to the Rubik’s cube.
Revised: We test our approach on Towers of Hanoi and discuss future work for applications to the Rubik’s cube.

Original: Introduction
Revised: Introduction

Original: As artiﬁcial intelligence (AI) continues to solve problems that humans struggle to solve, there is an emerging need for humans to understand these solutions so that we can trust AI, create new educational opportunities, and even discover new knowledge.
Revised: As artiﬁcial intelligence (AI) continues to solve problems that humans struggle to solve, there is an emerging need for humans to understand these solutions so that we can trust AI, create new educational opportunities, and even discover new knowledge.

Original: Many of these problems are pathﬁnding problems.
Revised: Many of these problems are pathﬁnding problems.

Original: That is, the problem is to ﬁnd a sequence of actions (a path) to go from any given state to a goal state.
Revised: That is, the problem is to ﬁnd a sequence of actions (a path) to go from any given state to a goal state.

Original: Examples of AI being successfully applied to pathﬁnding problems include puzzles (McAleer et al. 2019; Agostinelli et al. 2019), quantum compiling (Zhang et al. 2020), chemical synthesis (Chen et al. 2020), theorem proving (Bansal et al. 2019), and program synthesis (Ellis et al. 2021).
Revised: Examples of AI being successfully applied to pathﬁnding problems include puzzles (McAleer et al. 2019; Agostinelli et al. 2019), quantum compiling (Zhang et al. 2020), chemical synthesis (Chen et al. 2020), theorem proving (Bansal et al. 2019), and program synthesis (Ellis et al. 2021).

Original: Many explainable AI (XAI) approaches for explaining how to solve pathﬁnding problems and other sequential decision making problems focus on explaining a plan to solve a single instance of a problem (Sreedharan et al. 2022).
Revised: Many explainable AI (XAI) approaches for explaining how to solve pathﬁnding problems and other sequential decision making problems focus on explaining a plan to solve a single instance of a problem (Sreedharan et al. 2022).

Original: However, we focus on ﬁnding an explanation for solving all possible instances of a problem.
Revised: However, we focus on ﬁnding an explanation for solving all possible instances of a problem.

Original: Depending on the application, the simplicity of an explanation can matter more than the efﬁciency of a solution.
Revised: Depending on the application, the simplicity of an explanation can matter more than the efﬁciency of a solution.

Original: However, modern AI techniques, such as deep reinforcement learning, often ﬁnd very complicated strategies.
Revised: However, modern AI techniques, such as deep reinforcement learning, often ﬁnd very complicated strategies.

Original: For example, AI agents can play Go at a super human level (Silver et al. 2017) and can often solve the Rubik’s cube in the most efﬁcient way possible (Agostinelli et al. 2019).
Revised: For example, AI agents can play Go at a super human level (Silver et al. 2017) and can often solve the Rubik’s cube in the most efﬁcient way possible (Agostinelli et al. 2019).

Original: Explanations for these strategies may not be of use to most people.
Revised: Explanations for these strategies may not be of use to most people.

Original: Therefore, we seek to constrain the space of explanations to human-understandable explanations.
Revised: Therefore, we seek to constrain the space of explanations to human-understandable explanations.

Original: While the underlying planner may have a complicated and efﬁcient plan, such plans will only be incorporated if they can be explained.
Revised: While the underlying planner may have a complicated and efﬁcient plan, such plans will only be incorporated if they can be explained.

Original: Furthermore, we naturally incorporate compositionality in our approach by representing an explanation as a directed graph, allowing explanations to be re-used.
Revised: Furthermore, we naturally incorporate compositionality in our approach by representing an explanation as a directed graph, allowing explanations to be re-used.

Original: Therefore, if a plan for solving a given state is too difﬁcult to explain, we can then ﬁnd plans to reach intermediate points in the graph that may be easier to explain, thus favoring comprehensability over optimality.
Revised: Therefore, if a plan for solving a given state is too difﬁcult to explain, we can then ﬁnd plans to reach intermediate points in the graph that may be easier to explain, thus favoring comprehensability over optimality.

Original: In our approach, explanations for how to solve all instances of a problem will be derived from a directed acyclic graph whose nodes are ﬁrst-order logic programs representing sets of states and whose edges represent macro actions.
Revised: In our approach, explanations for how to solve all instances of a problem will be derived from a directed acyclic graph whose nodes are ﬁrst-order logic programs representing sets of states and whose edges represent macro actions.

Original: All nodes in this graph will have a path to the goal node.
Revised: All nodes in this graph will have a path to the goal node.

Original: A path to the goal node can be obtained from a given node by applying any macro-action associated with the outgoing macro-actions of that node.
Revised: A path to the goal node can be obtained from a given node by applying any macro-action associated with the outgoing macro-actions of that node.

Original: Intuitively, each node can be thought of as a precondition for applying one of its outgoing macro-actions.
Revised: Intuitively, each node can be thought of as a precondition for applying one of its outgoing macro-actions.

Original: The preconditions will be induced from observed solutions using inductive logic programming (ILP) techniques (Cropper et al. 2022).
Revised: The preconditions will be induced from observed solutions using inductive logic programming (ILP) techniques (Cropper et al. 2022).

Original: Given enough observations, the union of the preconditions for each node should cover the entire state space.
Revised: Given enough observations, the union of the preconditions for each node should cover the entire state space.

Original: Therefore, one can then solve any given state by ﬁnding a node that entails that state and applying any sequence of macro-actions that leads to the goal node.
Revised: Therefore, one can then solve any given state by ﬁnding a node that entails that state and applying any sequence of macro-actions that leads to the goal node.

Original: We refer to this as an explainable pathﬁnding graph, or e-graph.
Revised: We refer to this as an explainable pathﬁnding graph, or e-graph.

Original: Since logic programming is often used for XAI techniques due to its symbolic structure, given well-design predicates and inductive biases, each node can be explained to a human.
Revised: Since logic programming is often used for XAI techniques due to its symbolic structure, given well-design predicates and inductive biases, each node can be explained to a human.

Original: In this work, we focus on ﬁnding the simplest explanations possible.
Revised: In this work, we focus on ﬁnding the simplest explanations possible.

Original: We discuss a variety of ways one can quantify simplicity and how it can be achieved.
Revised: We discuss a variety of ways one can quantify simplicity and how it can be achieved.

Original: We seek to describe the entire e-graph to a human in a clear and concise manner, thereby explaining how to solve all instances of the problem.
Revised: We seek to describe the entire e-graph to a human in a clear and concise manner, thereby explaining how to solve all instances of the problem.

Original: In this work, we investigate how we can do this by learning hierarchical structures of the graph as well as using the graph to learn simple if/else rules.
Revised: In this work, we investigate how we can do this by learning hierarchical structures of the graph as well as using the graph to learn simple if/else rules.

Original: Related Work
Revised: Related Work

Original: The need for explanation in planning was articulated initially in (Fox, Long, and Magazzeni 2017; Hoffmann and
Revised: The need for explanation in planning was articulated initially in (Fox, Long, and Magazzeni 2017; Hoffmann and

Original: Magazzeni) and this lead to introduction of a number of approaches.
Revised: Magazzeni) and this lead to introduction of a number of approaches.

Original: A recent review summarizes the progress and ongoing challenges (Chakraborti, Sreedharan, and Kambhampati 2021).
Revised: A recent review summarizes the progress and ongoing challenges (Chakraborti, Sreedharan, and Kambhampati 2021).

Original: In brief, here the automated explanation method attempts to explain in a situation (state), why the planner chose an action, why the planner did not choose an action, why the planner decisions are better, why things asked (e.g., goals) can not be done, why one needs to replan, and why one does not have to replan.
Revised: In brief, here the automated explanation method attempts to explain in a situation (state), why the planner chose an action, why the planner did not choose an action, why the planner decisions are better, why things asked (e.g., goals) can not be done, why one needs to replan, and why one does not have to replan.

Original: Deﬁning solutions to sequential decision making problems as programs has been proposed to create explainable agents.
Revised: Deﬁning solutions to sequential decision making problems as programs has been proposed to create explainable agents.

Original: Using a given policy as an oracle, search can be done in a program space to match this oracle as close as possible (Verma et al. 2018).
Revised: Using a given policy as an oracle, search can be done in a program space to match this oracle as close as possible (Verma et al. 2018).

Original: A drawback to this method is that the oracle’s policy may be very complex, causing the learned program to be hard to understand.
Revised: A drawback to this method is that the oracle’s policy may be very complex, causing the learned program to be hard to understand.

Original: A program can be embedded in a latent space and a search can then be done in this latent space to ﬁnd a program that solves a given problem (Trivedi et al. 2021).
Revised: A program can be embedded in a latent space and a search can then be done in this latent space to ﬁnd a program that solves a given problem (Trivedi et al. 2021).

Original: One can model constructing a program itself as a reinforcement learning task where each function call is an action (Ellis et al. 2021).
Revised: One can model constructing a program itself as a reinforcement learning task where each function call is an action (Ellis et al. 2021).

Original: This approach allows one to grow a library of functions to augment its action space to solve increasingly complex problems.
Revised: This approach allows one to grow a library of functions to augment its action space to solve increasingly complex problems.

Original: Though programs have a discrete structure, one can optimize this a relaxed continuous structure instead (Liu, Simonyan, and Yang 2018).
Revised: Though programs have a discrete structure, one can optimize this a relaxed continuous structure instead (Liu, Simonyan, and Yang 2018).

Original: This relaxed structure has been used to allow the learning of nested if/then/else programs for solving mazes (Qiu and Zhu 2021).
Revised: This relaxed structure has been used to allow the learning of nested if/then/else programs for solving mazes (Qiu and Zhu 2021).

Original: The e-graph is similar to the planning graph created as part of the Graphplan planner (Blum and Furst 1997) but has notable differences.
Revised: The e-graph is similar to the planning graph created as part of the Graphplan planner (Blum and Furst 1997) but has notable differences.

Original: In Graphplan, the planner ﬁrst creates a graph, alternating in states and actions levels, starting with the initial state and representing actions that become applicable and states that could be reached, and so on.
Revised: In Graphplan, the planner ﬁrst creates a graph, alternating in states and actions levels, starting with the initial state and representing actions that become applicable and states that could be reached, and so on.

Original: In
Revised: In

Original: Graphplan, the graph is used to search for a plans to solve a particular instance of a problem.
Revised: Graphplan, the graph is used to search for a plans to solve a particular instance of a problem.

Original: In our work, we construct the e-graph that can solve all instances of a problem.
Revised: In our work, we construct the e-graph that can solve all instances of a problem.

Original: Background Towers of Hanoi (ToH)
Revised: Background Towers of Hanoi (ToH)

Original: The game of Towers of Hanoi is one player game with three pegs and a varying number of disks.
Revised: The game of Towers of Hanoi is one player game with three pegs and a varying number of disks.

Original: The disks are of different sizes and are arranged in ascending order of size.
Revised: The disks are of different sizes and are arranged in ascending order of size.

Original: To solve the problem all the disks must be moved to the last peg.
Revised: To solve the problem all the disks must be moved to the last peg.

Original: This is done by moving one disk at a time so that the disk can be directly kept on a peg or over a larger disk only.
Revised: This is done by moving one disk at a time so that the disk can be directly kept on a peg or over a larger disk only.

Original: The action space we use allows one to move any disk to any peg (in format ( disk, to-peg )) resulting in nine actions in total.
Revised: The action space we use allows one to move any disk to any peg (in format ( disk, to-peg )) resulting in nine actions in total.

Original: While a typical action space would also require specifying the peg that the disk is moving from, we use this action set as it allows us to learn a smaller graph.
Revised: While a typical action space would also require specifying the peg that the disk is moving from, we use this action set as it allows us to learn a smaller graph.

Original: Invalid actions result in no change.
Revised: Invalid actions result in no change.

Original: Problem Deﬁnition
Revised: Problem Deﬁnition

Original: The pathﬁnding problem is represented as a Markov decision process (MDP) (Puterman 2014).
Revised: The pathﬁnding problem is represented as a Markov decision process (MDP) (Puterman 2014).

Original: The MDP can be deﬁned as a set of states S , set of actions A , state-transition probability function T , and expected reward function r .
Revised: The MDP can be deﬁned as a set of states S , set of actions A , state-transition probability function T , and expected reward function r .

Original: The state-transition probability function returns the probability of transitioning from some state s to another states s (cid:48) using action a , T ( s, a, s (cid:48) ) .
Revised: The state-transition probability function returns the probability of transitioning from some state s to another states s (cid:48) using action a , T ( s, a, s (cid:48) ) .

Original: We limit the scope of this work to deterministic environments, therefore, we can replace T with a function A that returns the resulting state s (cid:48) when taking action a in state s , A ( s, a ) .
Revised: We limit the scope of this work to deterministic environments, therefore, we can replace T with a function A that returns the resulting state s (cid:48) when taking action a in state s , A ( s, a ) .

Original: The reward function r returns the expected reward when taking action a in state s and transitioning to state s (cid:48) , r ( s, a, s (cid:48) ) .
Revised: The reward function r returns the expected reward when taking action a in state s and transitioning to state s (cid:48) , r ( s, a, s (cid:48) ) .

Original: In the context of pathﬁnding, this is the negative transition cost.
Revised: In the context of pathﬁnding, this is the negative transition cost.

Original: The objective of pathﬁnding is to ﬁnd a path from any given state s to a goal state g , where g is in the set of goal states which is a subset of S .
Revised: The objective of pathﬁnding is to ﬁnd a path from any given state s to a goal state g , where g is in the set of goal states which is a subset of S .

Original: Given any planner that is used to solve a pathﬁnding problem (i.e. based on search, dynamic programming (Bellman 1957), or a combination of the two (Chen and Wei 2011; Agostinelli et al. 2019)), we seek to use this planner to ﬁnd an explanation of how to ﬁnd a path from any given state to a goal state.
Revised: Given any planner that is used to solve a pathﬁnding problem (i.e. based on search, dynamic programming (Bellman 1957), or a combination of the two (Chen and Wei 2011; Agostinelli et al. 2019)), we seek to use this planner to ﬁnd an explanation of how to ﬁnd a path from any given state to a goal state.

Original: While it is often desirable to ﬁnd a shortest path, we instead seek to ﬁnd an explanation that is easiest for humans to understand.
Revised: While it is often desirable to ﬁnd a shortest path, we instead seek to ﬁnd an explanation that is easiest for humans to understand.

Original: In the following section, we will describe how we formally quantify the simplicity of an explanation.
Revised: In the following section, we will describe how we formally quantify the simplicity of an explanation.

Original: Approach Graph Semantics and Structure
Revised: Approach Graph Semantics and Structure

Original: The explainable pathﬁnding graph, or e-graph, is a graph,
Revised: The explainable pathﬁnding graph, or e-graph, is a graph,

Original: G , whose nodes represent sets of states with ﬁrst-order logic programs and whose edges represent macro-actions, where macro-actions are a sequence of atomic actions.
Revised: G , whose nodes represent sets of states with ﬁrst-order logic programs and whose edges represent macro-actions, where macro-actions are a sequence of atomic actions.

Original: We refer to the ﬁrst-order logic programs as preconditions, since they must be satisﬁed in order for the macro-action to be applicable.
Revised: We refer to the ﬁrst-order logic programs as preconditions, since they must be satisﬁed in order for the macro-action to be applicable.

Original: The graph is directed and acyclic.
Revised: The graph is directed and acyclic.

Original: Every node in the graph has at least one child node except the single terminal node which represents the goal.
Revised: Every node in the graph has at least one child node except the single terminal node which represents the goal.

Original: The graph ﬁrst starts out with only the goal node.
Revised: The graph ﬁrst starts out with only the goal node.

Original: Every node added must have a child node that is already in the graph.
Revised: Every node added must have a child node that is already in the graph.

Original: Therefore, the graph should be complete, meaning there is a path to the goal from any goal node.
Revised: Therefore, the graph should be complete, meaning there is a path to the goal from any goal node.

Original: Furthermore, given enough observations, the graph should also be sound, meaning it should not be possible to come up with an invalid plan.
Revised: Furthermore, given enough observations, the graph should also be sound, meaning it should not be possible to come up with an invalid plan.

Original: The preconditions at each node are constructed from userdeﬁned predicates.
Revised: The preconditions at each node are constructed from userdeﬁned predicates.

Original: When these predicates are learned, the hypothesis space is constrained by a user-deﬁned inductive bias.
Revised: When these predicates are learned, the hypothesis space is constrained by a user-deﬁned inductive bias.

Original: Given predicates that a user can understand and an inductive bias that prunes complicated programs, every learned precondition should be explainable.
Revised: Given predicates that a user can understand and an inductive bias that prunes complicated programs, every learned precondition should be explainable.

Original: The overall explanation is then any method that summarizes the entire graph.
Revised: The overall explanation is then any method that summarizes the entire graph.

Original: This includes a hierarchical representation, a representation that groups similar macro-actions together, or a set of examples whose solutions match the preconditions of every node.
Revised: This includes a hierarchical representation, a representation that groups similar macro-actions together, or a set of examples whose solutions match the preconditions of every node.

Original: In this work, we concern ourselves only with ﬁnding explanations that are easy to understand.
Revised: In this work, we concern ourselves only with ﬁnding explanations that are easy to understand.

Original: To measure this, we deﬁne a cost for the e-graph, c ( G ) .
Revised: To measure this, we deﬁne a cost for the e-graph, c ( G ) .

Original: This cost can take into consideration the number of predicates, number of nodes, and number of macro actions.
Revised: This cost can take into consideration the number of predicates, number of nodes, and number of macro actions.

Original: Furthermore, it could even be based on the compactness of the ﬁnal explanation used to summarize the graph, such as one obtained from natural language generation techniques.
Revised: Furthermore, it could even be based on the compactness of the ﬁnal explanation used to summarize the graph, such as one obtained from natural language generation techniques.

Original: In our work, we simply seek to minimize the sum of the cost of the edges in the graph where the cost of an edge is determined by the number of macro actions.
Revised: In our work, we simply seek to minimize the sum of the cost of the edges in the graph where the cost of an edge is determined by the number of macro actions.

Original: Therefore, compositionality will be needed to achieve this.
Revised: Therefore, compositionality will be needed to achieve this.

Original: For example, one could have a node with an edge to the goal with a cost of one and another node with an edge to the goal with a cost of two, resulting in a graph with a cost of three.
Revised: For example, one could have a node with an edge to the goal with a cost of one and another node with an edge to the goal with a cost of two, resulting in a graph with a cost of three.

Original: However, if one can compose these nodes using shorter edges, then we can get a graph of a lower cost.
Revised: However, if one can compose these nodes using shorter edges, then we can get a graph of a lower cost.

Original: A ﬁgure illustrating this is shown in Figure 1.
Revised: A ﬁgure illustrating this is shown in Figure 1.

Original: We take a greedy approach to this optimization process.
Revised: We take a greedy approach to this optimization process.

Original: However, in future work, we aim to use other optimization approaches and as well as other metrics of cost to get the most humanunderstandable explanations.
Revised: However, in future work, we aim to use other optimization approaches and as well as other metrics of cost to get the most humanunderstandable explanations.

Original: [Figure]
Revised: [Figure]

Original: Learning the Graph
Revised: Learning the Graph

Original: The graph is initialized with a single goal node that represents the set of all solved states.
Revised: The graph is initialized with a single goal node that represents the set of all solved states.

Original: We then initialize a set of states that will be used to generate macro actions.
Revised: We then initialize a set of states that will be used to generate macro actions.

Original: This set of states is obtained by randomly generating states and using a pathﬁnding algorithm, ρ , to ﬁnd a path from all generated states to a goal state.
Revised: This set of states is obtained by randomly generating states and using a pathﬁnding algorithm, ρ , to ﬁnd a path from all generated states to a goal state.

Original: We then seek to add nodes to the graph until every state in this set is entailed by a node in the graph.
Revised: We then seek to add nodes to the graph until every state in this set is entailed by a node in the graph.

Original: Each node added to the graph must have as its child an existing node in the graph.
Revised: Each node added to the graph must have as its child an existing node in the graph.

Original: The edge connecting the two nodes must represent a macro action that can transform any state entailed by the parent’s precondition to a state entailed by the child’s precondition.
Revised: The edge connecting the two nodes must represent a macro action that can transform any state entailed by the parent’s precondition to a state entailed by the child’s precondition.

Original: To achieve this, we maintain a priority queue that contains tuples of macro-actions and nodes, where each macro-action has successfully transformed some state into a state entailed by its corresponding node.
Revised: To achieve this, we maintain a priority queue that contains tuples of macro-actions and nodes, where each macro-action has successfully transformed some state into a state entailed by its corresponding node.

Original: Since we are taking a greedy approach, this priority queue is sorted according to the macro action length.
Revised: Since we are taking a greedy approach, this priority queue is sorted according to the macro action length.

Original: When there is a tie, the closest ancestor of the goal node is given priority.
Revised: When there is a tie, the closest ancestor of the goal node is given priority.

Original: To obtain these tuples of macro actions and nodes, for a given node, ρ is used to ﬁnd a path from all unsolved states to a state entailed by that node.
Revised: To obtain these tuples of macro actions and nodes, for a given node, ρ is used to ﬁnd a path from all unsolved states to a state entailed by that node.

Original: The ﬁrst node for which this is done is for the goal node.
Revised: The ﬁrst node for which this is done is for the goal node.

Original: However, since the nodes represent sets of states in ﬁrst order logic, the processes of ﬁnding a path to a set of states must be addressed.
Revised: However, since the nodes represent sets of states in ﬁrst order logic, the processes of ﬁnding a path to a set of states must be addressed.

Original: In our work, we use an underlying Prolog representation to use the ﬁrst-order logic program as a query and get all states in the set.
Revised: In our work, we use an underlying Prolog representation to use the ﬁrst-order logic program as a query and get all states in the set.

Original: We then can use a heuristic function to estimate which state is closest and ﬁnd a path to that state.
Revised: We then can use a heuristic function to estimate which state is closest and ﬁnd a path to that state.

Original: For every successful path, the macro action and corresponding node are added to the priority queue.
Revised: For every successful path, the macro action and corresponding node are added to the priority queue.

Original: However, this approach will not scale as the size of the set grows.
Revised: However, this approach will not scale as the size of the set grows.

Original: In the discussion section, we will consider other approaches.
Revised: In the discussion section, we will consider other approaches.

Original: When a macro-action and corresponding node is removed from the priority queue, we apply the macro-action to all unsolved states.
Revised: When a macro-action and corresponding node is removed from the priority queue, we apply the macro-action to all unsolved states.

Original: Unsolved states that the macro-action transforms into states that are entailed the corresponding node’s precondition are used as positive examples while the other states are used as negative examples.
Revised: Unsolved states that the macro-action transforms into states that are entailed the corresponding node’s precondition are used as positive examples while the other states are used as negative examples.

Original: Furthermore, all solved states (that is, states entailed by a node that is already in the graph) are also used as negative examples.
Revised: Furthermore, all solved states (that is, states entailed by a node that is already in the graph) are also used as negative examples.

Original: We then seek to learn a ﬁrst-order logic program that perfectly separates the positive and negative examples.
Revised: We then seek to learn a ﬁrst-order logic program that perfectly separates the positive and negative examples.

Original: Learning may not always be successful.
Revised: Learning may not always be successful.

Original: For our purposes, this is desirable because we only want to add a node to the graph if it can be explained to a human.
Revised: For our purposes, this is desirable because we only want to add a node to the graph if it can be explained to a human.

Original: We use the Popper (Cropper and Morel 2021) inductive logic programming software to learn preconditions.
Revised: We use the Popper (Cropper and Morel 2021) inductive logic programming software to learn preconditions.

Original: Popper’s ﬂexible approach to specifying inductive biases allows us to constrain the hypothesis space for both speed and for comprehensibility.
Revised: Popper’s ﬂexible approach to specifying inductive biases allows us to constrain the hypothesis space for both speed and for comprehensibility.

Original: For example, in Towers of Hanoi, including simple constrains, such as specifying that disks can only have one size and that a disk cannot be on more than one peg, result in much faster learning times.
Revised: For example, in Towers of Hanoi, including simple constrains, such as specifying that disks can only have one size and that a disk cannot be on more than one peg, result in much faster learning times.

Original: Also, including constraints for comprehensibility, such as specifying that the predicate to say a disk is not on a peg can only be used once per clause, results in programs that are much easier to understand.
Revised: Also, including constraints for comprehensibility, such as specifying that the predicate to say a disk is not on a peg can only be used once per clause, results in programs that are much easier to understand.

Original: The overall algorithm is showed in Algorithm 1 and the process to obtain tuples of macro-actions and nodes is shown in Algorithm 2.
Revised: The overall algorithm is showed in Algorithm 1 and the process to obtain tuples of macro-actions and nodes is shown in Algorithm 2.

Original: Explanation Generation
Revised: Explanation Generation

Original: Given an e-graph, we explore different approaches for obtaining an explanation.
Revised: Given an e-graph, we explore different approaches for obtaining an explanation.

Original: The ﬁrst is a simple template-based natural language generation approach that explains the preconditions for each node.
Revised: The ﬁrst is a simple template-based natural language generation approach that explains the preconditions for each node.

Original: We can even print the entire graph as a nested structure where ancestors are nested within their descendants.
Revised: We can even print the entire graph as a nested structure where ancestors are nested within their descendants.

Original: In the future, we plan to utilize language models based on transformers (Vaswani et al. 2017) to convert ﬁrst-order logic sentences into everyday language.
Revised: In the future, we plan to utilize language models based on transformers (Vaswani et al. 2017) to convert ﬁrst-order logic sentences into everyday language.

Original: The next approach is obtaining a hierarchical structure.
Revised: The next approach is obtaining a hierarchical structure.

Original: One can obtain learn hierarchical nodes by ﬁrst obtaining all states associated with a given node and all states associated with its descendants.
Revised: One can obtain learn hierarchical nodes by ﬁrst obtaining all states associated with a given node and all states associated with its descendants.

Original: Then, one can learn a logic program that entails all of the aforementioned states and none of the states entailed by any other nodes in the graph.
Revised: Then, one can learn a logic program that entails all of the aforementioned states and none of the states entailed by any other nodes in the graph.

Original: We explore this approach by ﬁrst trying to learn programs for nodes for the greatest number of descendants.
Revised: We explore this approach by ﬁrst trying to learn programs for nodes for the greatest number of descendants.

Original: If the program is successful, we then remove the node from the descendants of all other nodes.
Revised: If the program is successful, we then remove the node from the descendants of all other nodes.

Original: We then repeat this processes until all nodes are in a hierarchical node, even if the hierarchical node just contains a single node.
Revised: We then repeat this processes until all nodes are in a hierarchical node, even if the hierarchical node just contains a single node.

Original: We can also do the same thing for the ancestors of nodes.
Revised: We can also do the same thing for the ancestors of nodes.

Original: The ﬁnal approach is to group nodes together if their outgoing edge has the same macro-action.
Revised: The ﬁnal approach is to group nodes together if their outgoing edge has the same macro-action.

Original: Then, we can learn a logic program that entails all states entailed by these nodes and none of the states entailed by the other nodes.
Revised: Then, we can learn a logic program that entails all states entailed by these nodes and none of the states entailed by the other nodes.

Original: The intuition is that we can ﬁnd a simple if/then structure for when to apply macro-actions.
Revised: The intuition is that we can ﬁnd a simple if/then structure for when to apply macro-actions.

Original: If there are not many macro-actions,
Revised: If there are not many macro-actions,

Original: Input:
Revised: Input:

Original: S u : Set of unsolved states
Revised: S u : Set of unsolved states

Original: P : Goal precondition
Revised: P : Goal precondition

Original: B
Revised: B

Original: Background knowledge ρ : strategy
Revised: Background knowledge ρ : strategy

Original: Output: g : Goal node of e-graph g = Node ( P )
Revised: Output: g : Goal node of e-graph g = Node ( P )

Original: S s = {} //Set of solved states
Revised: S s = {} //Set of solved states

Original: Remove states entailed by P from S u and add to S s q =
Revised: Remove states entailed by P from S u and add to S s q =

Original: [] //Priority queue of macro actions and nodes add macros( S u , g , q , ρ ) while len( S u ) > 0 do m, n = p.pop ()
Revised: [] //Priority queue of macro actions and nodes add macros( S u , g , q , ρ ) while len( S u ) > 0 do m, n = p.pop ()

Original: //macro action and target node
Revised: //macro action and target node

Original: E + =
Revised: E + =

Original: [] //positive examples
Revised: [] //positive examples

Original: E − =
Revised: E − =

Original: S s //negative examples for s ∈ S do
Revised: S s //negative examples for s ∈ S do

Original: Apply m to s to obtain s (cid:48) // check if result is entailed by node if n.P ( s (cid:48) ) is True then append s to E + else append s to E −
Revised: Apply m to s to obtain s (cid:48) // check if result is entailed by node if n.P ( s (cid:48) ) is True then append s to E + else append s to E −

Original: P = learn precondtion( E + , E − , B ) if P learned successfully then n = Node ( P )
Revised: P = learn precondtion( E + , E − , B ) if P learned successfully then n = Node ( P )

Original: Remove states entailed by P from S u and add to S s add macros( S u , n , q , ρ )
Revised: Remove states entailed by P from S u and add to S s add macros( S u , n , q , ρ )

Original: Return g then someone can read through each one to understand the explanation.
Revised: Return g then someone can read through each one to understand the explanation.

Original: Results
Revised: Results

Original: The background knowledge we are using for Towers of
Revised: The background knowledge we are using for Towers of

Original: Hanoi contains predicates that describe a disk being on a peg, a disk not being on a peg, a disk being above another disk, a disk’s top or bottom being clear, and a peg being clear.
Revised: Hanoi contains predicates that describe a disk being on a peg, a disk not being on a peg, a disk being above another disk, a disk’s top or bottom being clear, and a peg being clear.

Original: Since we are only considering the three disk scenario, we use breadth-ﬁrst search (BFS) for ﬁnding macro actions.
Revised: Since we are only considering the three disk scenario, we use breadth-ﬁrst search (BFS) for ﬁnding macro actions.

Original: BFS is one of the simplest path ﬁnding algorithm in which the shallowest nodes are given priority.
Revised: BFS is one of the simplest path ﬁnding algorithm in which the shallowest nodes are given priority.

Original: The algorithm is optimal assuming the transition costs between each node is one.
Revised: The algorithm is optimal assuming the transition costs between each node is one.

Original: Towers of Hanoi has a state space with 27 states.
Revised: Towers of Hanoi has a state space with 27 states.

Original: The learned e-graph is shown in Figure 2a.
Revised: The learned e-graph is shown in Figure 2a.

Original: We use a nested structure to represent the graph which has 15 nodes.
Revised: We use a nested structure to represent the graph which has 15 nodes.

Original: A template-based approach was used to generate a natural language description of each node.
Revised: A template-based approach was used to generate a natural language description of each node.

Original: A total of seven unique macro actions are used in the solution and are omitted in the ﬁgure for brevity.
Revised: A total of seven unique macro actions are used in the solution and are omitted in the ﬁgure for brevity.

Original: In this case, the all macro-actions are one of the nine atomic actions.
Revised: In this case, the all macro-actions are one of the nine atomic actions.

Original: To simplify this explanation, we look at how we can obtain hierarchies.
Revised: To simplify this explanation, we look at how we can obtain hierarchies.

Original: The ﬁrst approach to create a hierarchical representation is to learn programs
Revised: The ﬁrst approach to create a hierarchical representation is to learn programs

Original: Algorithm
Revised: Algorithm

Original: Input:
Revised: Input:

Original: S : Set of states n : Node q : Priority queue ρ : Pathﬁnding strategy
Revised: S : Set of states n : Node q : Priority queue ρ : Pathﬁnding strategy

Original: M = {} //set of macro actions for s ∈ S do // m is macro action that takes s to a state entailed by the precondition of n . m
Revised: M = {} //set of macro actions for s ∈ S do // m is macro action that takes s to a state entailed by the precondition of n . m

Original: = ρ ( s, g ) if m found successfully then add m to M for m ∈ M do add ( m, n ) to q with ﬁrst priority len( m ) and second priority being the depth of n to differentiate a node and all its descendants from all other nodes.
Revised: = ρ ( s, g ) if m found successfully then add m to M for m ∈ M do add ( m, n ) to q with ﬁrst priority len( m ) and second priority being the depth of n to differentiate a node and all its descendants from all other nodes.

Original: The results of this approach are shown in Figure 2b.
Revised: The results of this approach are shown in Figure 2b.

Original: This results in a graph with 12 nodes.
Revised: This results in a graph with 12 nodes.

Original: We then create a hierarchical by learning programs to differentiate a node and all its ancestors from all other nodes.
Revised: We then create a hierarchical by learning programs to differentiate a node and all its ancestors from all other nodes.

Original: The results of this approach are shown in Figure 2c.
Revised: The results of this approach are shown in Figure 2c.

Original: The results show a very simple representation that says that the largest disk goes to peg from either peg1 or peg2.
Revised: The results show a very simple representation that says that the largest disk goes to peg from either peg1 or peg2.

Original: Finally, we investigate learning an if/else structure by grouping nodes with the same macro-action together.
Revised: Finally, we investigate learning an if/else structure by grouping nodes with the same macro-action together.

Original: Since there are seven macro-actions used, we should have seven if/else statements.
Revised: Since there are seven macro-actions used, we should have seven if/else statements.

Original: While the ILP system was not able to learn a program for moving disk1 to peg1 or moving disk to peg2, it successfully learned a precondition for the other ﬁve macro actions.
Revised: While the ILP system was not able to learn a program for moving disk1 to peg1 or moving disk to peg2, it successfully learned a precondition for the other ﬁve macro actions.

Original: The result is shown in Figure 2d.
Revised: The result is shown in Figure 2d.

Original: Discussion Applications to the Rubik’s Cube
Revised: Discussion Applications to the Rubik’s Cube

Original: We are currently extending this work to the Rubik’s cube.
Revised: We are currently extending this work to the Rubik’s cube.

Original: The Rubik’s cube poses unique challenges for any XAI approach due to its symmetries.
Revised: The Rubik’s cube poses unique challenges for any XAI approach due to its symmetries.

Original: Furthermore, the fact that solutions to the Rubik’s cube can be explained to young children sets a standard that current XAI approaches based on modern AI techniques, such as deep learning, cannot match.
Revised: Furthermore, the fact that solutions to the Rubik’s cube can be explained to young children sets a standard that current XAI approaches based on modern AI techniques, such as deep learning, cannot match.

Original: In our preliminary work, we using an example based explanation approach that uses landmarks for clarity.
Revised: In our preliminary work, we using an example based explanation approach that uses landmarks for clarity.

Original: That is, after the graph is constructed, we ﬁnd a set of example states where each node entails at least one state in the union of states in the solution paths.
Revised: That is, after the graph is constructed, we ﬁnd a set of example states where each node entails at least one state in the union of states in the solution paths.

Original: We identify speciﬁc landmarks based on the number and length of incoming edges to a node.
Revised: We identify speciﬁc landmarks based on the number and length of incoming edges to a node.

Original: We then ensure we explicitly explain these landmarks.
Revised: We then ensure we explicitly explain these landmarks.

Original: On the other hand, we implicitly explain other nodes that only have one parent or whose incoming edges are short.
Revised: On the other hand, we implicitly explain other nodes that only have one parent or whose incoming edges are short.

Original: This way, we can try to avoid obvious explanations while focusing on more difﬁcult aspects of the problem.
Revised: This way, we can try to avoid obvious explanations while focusing on more difﬁcult aspects of the problem.

Original: For ﬁnding macro-actions using a search strategy ρ , we extend the DeepCubeA algorithm (Agostinelli et al. 2019).
Revised: For ﬁnding macro-actions using a search strategy ρ , we extend the DeepCubeA algorithm (Agostinelli et al. 2019).

Original: In this domain, however, the preconditions for nodes often only contain a subset of the stickers on the Rubik’s cube.
Revised: In this domain, however, the preconditions for nodes often only contain a subset of the stickers on the Rubik’s cube.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Therefore, there are many elements of the state that are not speciﬁed.
Revised: Therefore, there are many elements of the state that are not speciﬁed.

Original: To allow the neural network to ﬁnd paths to partially speciﬁed states, we use hindsight experience replay (Andrychowicz et al. 2017) and a special color for nonspeciﬁed stickers, to learn a cost-to-go function that estimates the distance between any pair of partially speciﬁed states.
Revised: To allow the neural network to ﬁnd paths to partially speciﬁed states, we use hindsight experience replay (Andrychowicz et al. 2017) and a special color for nonspeciﬁed stickers, to learn a cost-to-go function that estimates the distance between any pair of partially speciﬁed states.

Original: Searching in the Space of States vs Sets of States
Revised: Searching in the Space of States vs Sets of States

Original: Currently, we are ﬁnding macro-actions that can transform a state to another state entailed by the precondition of a node by generating states with Prolog and ﬁnding paths in the state space.
Revised: Currently, we are ﬁnding macro-actions that can transform a state to another state entailed by the precondition of a node by generating states with Prolog and ﬁnding paths in the state space.

Original: However, this process will become unsustainable as the number of states entailed by each node grows.
Revised: However, this process will become unsustainable as the number of states entailed by each node grows.

Original: One way to try to address this is by using subgoals where states are only partially speciﬁed.
Revised: One way to try to address this is by using subgoals where states are only partially speciﬁed.

Original: The state space of a partially speciﬁed subgoal can be much smaller than the overall state space.
Revised: The state space of a partially speciﬁed subgoal can be much smaller than the overall state space.

Original: Such an approach is often seen when describing solutions to the Rubik’s cube.
Revised: Such an approach is often seen when describing solutions to the Rubik’s cube.

Original: If this is not possible, then one will have to be able to search in the space of sets of states instead of the space of states.
Revised: If this is not possible, then one will have to be able to search in the space of sets of states instead of the space of states.

Original: One possible approach to accomplish this is to use hindsight experience replay (Andrychowicz et al. 2017) to learn a cost-to-go function that takes as input a given state and a logic program.
Revised: One possible approach to accomplish this is to use hindsight experience replay (Andrychowicz et al. 2017) to learn a cost-to-go function that takes as input a given state and a logic program.

Original: The cost-to-go function should estimate the distance between a given state and the closest state entailed by the logic program.
Revised: The cost-to-go function should estimate the distance between a given state and the closest state entailed by the logic program.

Original: One could generate training examples by ﬁrst generating a goal state in the state space and then creating a ﬁrst-order logic program that entails that goal state.
Revised: One could generate training examples by ﬁrst generating a goal state in the state space and then creating a ﬁrst-order logic program that entails that goal state.

Original: Abstraction While Learning the Graph
Revised: Abstraction While Learning the Graph

Original: Our current explanation methods are able to build abstractions by learning a hierarchical graph after the e-graph has been learned.
Revised: Our current explanation methods are able to build abstractions by learning a hierarchical graph after the e-graph has been learned.

Original: However, abstraction may be needed during the learning process, itself.
Revised: However, abstraction may be needed during the learning process, itself.

Original: For example, learning to transition to a speciﬁc node may be too cumbersome for the ILP system, especially if many variables are involved due to interdependence.
Revised: For example, learning to transition to a speciﬁc node may be too cumbersome for the ILP system, especially if many variables are involved due to interdependence.

Original: Instead, one could relax the exact transition that should take place by allowing it to transition to a set of nodes.
Revised: Instead, one could relax the exact transition that should take place by allowing it to transition to a set of nodes.

Original: Explanation Aware Search
Revised: Explanation Aware Search

Original: Part of the ease of an explanation involves how easy it is to describe the macro action.
Revised: Part of the ease of an explanation involves how easy it is to describe the macro action.

Original: For example, there a longer macro-action composed of only one atomic action may be easier to understand than a shorter macro-action composed of many atomic actions.
Revised: For example, there a longer macro-action composed of only one atomic action may be easier to understand than a shorter macro-action composed of many atomic actions.

Original: Furthermore, atomic-actions can be represented as a predicate with arguments.
Revised: Furthermore, atomic-actions can be represented as a predicate with arguments.

Original: For example, in
Revised: For example, in

Original: Towers of Hanoi, the disk and destination peg can be the arguments to the action predicate.
Revised: Towers of Hanoi, the disk and destination peg can be the arguments to the action predicate.

Original: In this formulation, the precondition and macro-action can share variables and create more compact descriptions of macro-actions.
Revised: In this formulation, the precondition and macro-action can share variables and create more compact descriptions of macro-actions.

Original: It is possible that this work could be combined with methods based on learning with deep neural networks, such as Dreamcoder (Ellis et al. 2021) or DeepCubeA (Agostinelli et al.
Revised: It is possible that this work could be combined with methods based on learning with deep neural networks, such as Dreamcoder (Ellis et al. 2021) or DeepCubeA (Agostinelli et al.

Original: to exploit this relationship.
Revised: to exploit this relationship.

Revised: Acknowledgments

Revised: This work was funded by the University of South Carolina

Revised: ASPIRE-II program.