Original: D IVIDE - AND -C ONQUER M ONTE C ARLO T REE S EARCH
Revised: D IVIDE - AND -C ONQUER M ONTE C ARLO T REE S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Standard planners for sequential decision making (including Monte Carlo planning, tree search, dynamic programming, etc.) are constrained by an implicit sequential planning assumption :
Revised: Standard planners for sequential decision making (including Monte Carlo planning, tree search, dynamic programming, etc.) are constrained by an implicit sequential planning assumption :

Original: The order in which a plan is constructed is the same in which it is executed.
Revised: The order in which a plan is constructed is the same in which it is executed.

Original: We consider alternatives to this assumption for the class of goal-directed Reinforcement Learning (RL) problems.
Revised: We consider alternatives to this assumption for the class of goal-directed Reinforcement Learning (RL) problems.

Original: Instead of an environment transition model, we assume an imperfect, goal-directed policy.
Revised: Instead of an environment transition model, we assume an imperfect, goal-directed policy.

Original: This low-level policy can be improved by a plan, consisting of an appropriate sequence of subgoals that guide it from the start to the goal state.
Revised: This low-level policy can be improved by a plan, consisting of an appropriate sequence of subgoals that guide it from the start to the goal state.

Original: We propose a planning algorithm,
Revised: We propose a planning algorithm,

Original: Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS), for approximating the optimal plan by means of proposing intermediate sub-goals which hierarchically partition the initial tasks into simpler ones that are then solved independently and recursively.
Revised: Divide-and-Conquer Monte Carlo Tree Search (DC-MCTS), for approximating the optimal plan by means of proposing intermediate sub-goals which hierarchically partition the initial tasks into simpler ones that are then solved independently and recursively.

Original: The algorithm critically makes use of a learned sub-goal proposal for ﬁnding appropriate partition trees of new tasks based on prior experience.
Revised: The algorithm critically makes use of a learned sub-goal proposal for ﬁnding appropriate partition trees of new tasks based on prior experience.

Original: Different strategies for learning sub-goal proposals give rise to different planning strategies that strictly generalize sequential planning.
Revised: Different strategies for learning sub-goal proposals give rise to different planning strategies that strictly generalize sequential planning.

Original: We show that this algorithmic ﬂexibility wrt.
Revised: We show that this algorithmic ﬂexibility wrt.

Original: planning order leads to improved results in navigation tasks in grid-worlds as well as in challenging continuous control environments.
Revised: planning order leads to improved results in navigation tasks in grid-worlds as well as in challenging continuous control environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This is the ﬁrst sentence of this paper, but it was not the ﬁrst one we wrote.
Revised: This is the ﬁrst sentence of this paper, but it was not the ﬁrst one we wrote.

Original: In fact, the entire introduction section was actually one of the last sections to be added to this manuscript.
Revised: In fact, the entire introduction section was actually one of the last sections to be added to this manuscript.

Original: The discrepancy between the order of inception of ideas and the order of their presentation in this paper probably does not come as a surprise to the reader.
Revised: The discrepancy between the order of inception of ideas and the order of their presentation in this paper probably does not come as a surprise to the reader.

Original: Nonetheless, it serves as a point for reﬂection that is central to the rest of this work, and that can be summarized as “the order in which we construct a plan does not have to coincide with the order in which we execute it” .
Revised: Nonetheless, it serves as a point for reﬂection that is central to the rest of this work, and that can be summarized as “the order in which we construct a plan does not have to coincide with the order in which we execute it” .

Original: Most standard planners for sequential decision making problems—including Monte Carlo planning,
Revised: Most standard planners for sequential decision making problems—including Monte Carlo planning,

Original: Monte Carlo Tree Search (MCTS) and dynamic programming—have a baked-in sequential planning assumption (Bertsekas et al., 1995;
Revised: Monte Carlo Tree Search (MCTS) and dynamic programming—have a baked-in sequential planning assumption (Bertsekas et al., 1995;

Original: Browne et al., 2012).
Revised: Browne et al., 2012).

Original: These methods begin at either the initial or ﬁnal state and then plan actions sequentially forward or backwards in time.
Revised: These methods begin at either the initial or ﬁnal state and then plan actions sequentially forward or backwards in time.

Original: However, this sequential approach faces two main challenges.
Revised: However, this sequential approach faces two main challenges.

Original: (i)
Revised: (i)

Original: The transition model used for planning needs to be reliable over long horizons, which is often difﬁcult to achieve when it has to be inferred from data.
Revised: The transition model used for planning needs to be reliable over long horizons, which is often difﬁcult to achieve when it has to be inferred from data.

Original: (ii) Credit assignment to each individual action is difﬁcult:
Revised: (ii) Credit assignment to each individual action is difﬁcult:

Original: In a planning problem spanning a horizon ofsteps, to assign credit to the ﬁrst action, we have to compute the optimal cost-to-go for the remaining problem with a horizon of 99 steps, which is only slightly easier than solving the original problem.
Revised: In a planning problem spanning a horizon ofsteps, to assign credit to the ﬁrst action, we have to compute the optimal cost-to-go for the remaining problem with a horizon of 99 steps, which is only slightly easier than solving the original problem.

Original: [Figure]
Revised: [Figure]

Original: To overcome these two fundamental challenges, here we consider alternatives to the basic assumptions of sequential planners.
Revised: To overcome these two fundamental challenges, here we consider alternatives to the basic assumptions of sequential planners.

Original: We focus on goal-directed decision making problems where an agent should reach a goal state from a start state.
Revised: We focus on goal-directed decision making problems where an agent should reach a goal state from a start state.

Original: Instead of a transition and reward model of the environment, we assume a given goal-directed policy (the “low-level” policy) and the associated value oracle that returns its success probability on any given task.
Revised: Instead of a transition and reward model of the environment, we assume a given goal-directed policy (the “low-level” policy) and the associated value oracle that returns its success probability on any given task.

Original: 1 In general, a low-level policy will not be not optimal, e.g. it might be too “myopic” to reliably reach goal states that are far away from its current state.
Revised: 1 In general, a low-level policy will not be not optimal, e.g. it might be too “myopic” to reliably reach goal states that are far away from its current state.

Original: We now seek to improve the low-level policy via a suitable sequence of sub-goals thateffectively guide it from the start to the ﬁnal goal, thus maximizing the overall task success probability.
Revised: We now seek to improve the low-level policy via a suitable sequence of sub-goals that guide it from the start to the ﬁnal goal, thus maximizing the overall task success probability.

Original: This formulation of planning as ﬁnding good sub-goal sequences, makes learning of explicit environment models unnecessary, as they are replaced by low-level policies and their value functions.
Revised: This formulation of planning as ﬁnding good sub-goal sequences, makes learning of explicit environment models unnecessary, as they are replaced by low-level policies and their value functions.

Original: The sub-goal planning problem can still be solved by a conventional sequential planner that begins by searching for the ﬁrst sub-goal to reach from the start state, then planning the next sub-goal in sequence, and so on.
Revised: The sub-goal planning problem can still be solved by a conventional sequential planner that begins by searching for the ﬁrst sub-goal to reach from the start state, then planning the next sub-goal in sequence, and so on.

Original: Indeed, this is the approach taken in most hierarchical RL settings based on options or sub-goals (e.g. Dayan & Hinton, 1993;
Revised: Indeed, this is the approach taken in most hierarchical RL settings based on options or sub-goals (e.g. Dayan & Hinton, 1993;

Original: Sutton et al., 1999; Vezhnevets et al., 2017).
Revised: Sutton et al., 1999; Vezhnevets et al., 2017).

Original: However, the credit assignment problem mentioned above persists, as assessing if the ﬁrst sub-goal is useful still requires evaluating the success probability of the remaining plan.
Revised: However, the credit assignment problem mentioned above persists, as assessing if the ﬁrst sub-goal is useful still requires evaluating the success probability of the remaining plan.

Original: Instead, it could be substantially easier to reason about the utility of a sub-goal “in the middle” of the plan, as this breaks the long-horizon problem into two sub-problems with much shorter horizons: how to get to the sub-goal and how to get from there to the ﬁnal goal.
Revised: Instead, it could be substantially easier to reason about the utility of a sub-goal “in the middle” of the plan, as this breaks the long-horizon problem into two sub-problems with much shorter horizons: how to get to the sub-goal and how to get from there to the ﬁnal goal.

Original: Based on this intuition, we propose the
Revised: Based on this intuition, we propose the

Original: Divide-and-Conquer MCTS (DC-MCTS) planner that searches for sub-goals to split the original task into two independent sub-tasks of comparable complexity and then recursively solves these, thereby drastically facilitating credit assignment.
Revised: Divide-and-Conquer MCTS (DC-MCTS) planner that searches for sub-goals to split the original task into two independent sub-tasks of comparable complexity and then recursively solves these, thereby drastically facilitating credit assignment.

Original: To search the space of intermediate sub-goals efﬁciently,
Revised: To search the space of intermediate sub-goals efﬁciently,

Original: DC-MCTS uses a heuristic for proposing promising sub-goals that is learned from previous search results and agent experience.
Revised: DC-MCTS uses a heuristic for proposing promising sub-goals that is learned from previous search results and agent experience.

Revised: Humans can plan efﬁciently over long horizons to solve complex tasks, such as theorem proving or navigation, and some plans even span over decades (e.g. economic measures):

Revised: In these situations, planning sequentially in terms of next steps – such as what arm to move, or what phone call to make – will cover a tiny proportion of the horizon, neglecting the long uncertainty beyond the last planned step.

Revised: The algorithm put forward in this paper is a step in the direction of efﬁcient planners that tackle long horizons by recursively and parallelly splitting them into many smaller and smaller sub-problems.

Original: In Section 2, we formulate planning in terms of sub-goals instead of primitive actions.
Revised: In Section 2, we formulate planning in terms of sub-goals instead of primitive actions.

Original: In Section 3, as our main contribution, we propose the novel Divide-and-Conquer Monte Carlo Tree Search algorithm for this planning problem.
Revised: In Section 3, as our main contribution, we propose the novel Divide-and-Conquer Monte Carlo Tree Search algorithm for this planning problem.

Original: In Section 4 we position DC-MCTS within the literature of related work.
Revised: In Section 4 we position DC-MCTS within the literature of related work.

Original: In Section 5, we show that it outperforms sequential planners both on grid world and continuous control navigation tasks, demonstrating the utility of constructing plans in a ﬂexible order that can be different from their execution order.
Revised: In Section 5, we show that it outperforms sequential planners both on grid world and continuous control navigation tasks, demonstrating the utility of constructing plans in a ﬂexible order that can be different from their execution order.

Original:  I MPROVING G OAL -D IRECTED P OLICIES WITH P LANNING
Revised:  I MPROVING G OAL -D IRECTED P OLICIES WITH P LANNING

Original: Let S and A be ﬁnite sets of states and actions.
Revised: Let S and A be ﬁnite sets of states and actions.

Original: We consider a multi-task setting, where for each episode the agent has to solve a new task consisting of a new Markov Decision Process (MDP) M over S and A .
Revised: We consider a multi-task setting, where for each episode the agent has to solve a new task consisting of a new Markov Decision Process (MDP) M over S and A .

Original: Each M has a single start state s 0 and a special absorbing state s ∞ , also termed the goal state.
Revised: Each M has a single start state s 0 and a special absorbing state s ∞ , also termed the goal state.

Original: If the agent transitions into s ∞ at any time it receives a reward of 1 and the episode terminates; otherwise the reward is 0.
Revised: If the agent transitions into s ∞ at any time it receives a reward of 1 and the episode terminates; otherwise the reward is 0.

Original: We assume that the agent observes the start and goal states ( s 0 , s ∞ ) at the beginning of each episode, as well as an encoding vector c M ∈ R d .
Revised: We assume that the agent observes the start and goal states ( s 0 , s ∞ ) at the beginning of each episode, as well as an encoding vector c M ∈ R d .

Original: This vector provides the agent with additional information about the MDP M of the current episode and will be key to transfer learning across tasks in the multi-task setting.
Revised: This vector provides the agent with additional information about the MDP M of the current episode and will be key to transfer learning across tasks in the multi-task setting.

Original: A stochastic, goal-directed policy π is a mapping from S × S × R d into distributions over A , where π ( a | s, s ∞ , c M ) denotes the probability of taking action a in state s in order to get to goal s ∞ .
Revised: A stochastic, goal-directed policy π is a mapping from S × S × R d into distributions over A , where π ( a | s, s ∞ , c M ) denotes the probability of taking action a in state s in order to get to goal s ∞ .

Original: For a ﬁxed goal s ∞ , we can interpret π as a regular policy, here denoted as π s ∞ , mapping states to action probabilities.
Revised: For a ﬁxed goal s ∞ , we can interpret π as a regular policy, here denoted as π s ∞ , mapping states to action probabilities.

Original: We denote the value of π in state s for goal s ∞ as v π ( s, s ∞ | c M ) ; we assume no discounting γ = 1 .
Revised: We denote the value of π in state s for goal s ∞ as v π ( s, s ∞ | c M ) ; we assume no discounting γ = 1 .

Original: Under the above deﬁnition of the reward, the value is equal to the success probability of π on the task, i.e. the absorption probability of the stochastic process starting in s 0 deﬁned by running π s ∞ :
Revised: Under the above deﬁnition of the reward, the value is equal to the success probability of π on the task, i.e. the absorption probability of the stochastic process starting in s 0 deﬁned by running π s ∞ :

Original: [Equation]
Revised: [Equation]

Original: where τ π s ∞ sis the trajectory generated by running π s ∞ from state s 0 2 .
Revised: where τ π s ∞ sis the trajectory generated by running π s ∞ from state s 0 2 .

Original: To keep the notation compact, we will omit the explicit dependence on c M and abbreviate tasks with pairs of states in S × S .
Revised: To keep the notation compact, we will omit the explicit dependence on c M and abbreviate tasks with pairs of states in S × S .

Original: 2. P LANNING OVER S UB -G OAL S EQUENCES
Revised: 2. P LANNING OVER S UB -G OAL S EQUENCES

Original: Assume a given goal-directed policy π , which we also refer to as the low-level policy.
Revised: Assume a given goal-directed policy π , which we also refer to as the low-level policy.

Original: If π is not already optimal, we can potentially improve it by planning: If π has a low probability of directly reaching s ∞ from the initial state s 0 , i.e. v π ( s 0 , s ∞ ) ≈ 0 , we will try to ﬁnd a plan consisting of a sequence of intermediate sub-goals such that they guide π from the start s 0 to the goal state s ∞ .
Revised: If π is not already optimal, we can potentially improve it by planning: If π has a low probability of directly reaching s ∞ from the initial state s 0 , i.e. v π ( s 0 , s ∞ ) ≈ 0 , we will try to ﬁnd a plan consisting of a sequence of intermediate sub-goals such that they guide π from the start s 0 to the goal state s ∞ .

Original: Concretely, let S ∗ = ∪ ∞ n =0 S n be the set of sequences over S , and let | σ | be the length of a sequence σ ∈ S ∗ .
Revised: Concretely, let S ∗ = ∪ ∞ n =0 S n be the set of sequences over S , and let | σ | be the length of a sequence σ ∈ S ∗ .

Original: We deﬁne for convenience ¯ S :
Revised: We deﬁne for convenience ¯ S :

Original: = S ∪ { ∅ } , where ∅ is the empty sequence representing no sub-goal.
Revised: = S ∪ { ∅ } , where ∅ is the empty sequence representing no sub-goal.

Original: We refer to σ as a plan for task ( s 0 , s ∞ ) if σ 1 = s 0 and σ | σ | = s ∞ , i.e. if the ﬁrst and last elements of σ are equal to s 0 and s ∞ , respectively.
Revised: We refer to σ as a plan for task ( s 0 , s ∞ ) if σ 1 = s 0 and σ | σ | = s ∞ , i.e. if the ﬁrst and last elements of σ are equal to s 0 and s ∞ , respectively.

Original: s 0 S ∗ s ∞ denotes the set of plans for this task.
Revised: s 0 S ∗ s ∞ denotes the set of plans for this task.

Original: To execute a plan σ , we construct a policy π σ by conditioning the low-level policy π on each of the sub-goals in order: Starting with n = 1 , we feed sub-goal σ n +1 to π , i.e. we run π σ n +1 ; if σ n +1 is reached, we will execute π σ n +2 and so on.
Revised: To execute a plan σ , we construct a policy π σ by conditioning the low-level policy π on each of the sub-goals in order: Starting with n = 1 , we feed sub-goal σ n +1 to π , i.e. we run π σ n +1 ; if σ n +1 is reached, we will execute π σ n +2 and so on.

Original: We now wish to do open-loop planning , i.e. ﬁnd the plan with the highest success probability P ( s ∞ ∈ τ π σ s 0 ) of reaching s ∞ .
Revised: We now wish to do open-loop planning , i.e. ﬁnd the plan with the highest success probability P ( s ∞ ∈ τ π σ s 0 ) of reaching s ∞ .

Original: However, this success probability depends on the transition kernels of the underlying MDPs, which might not be known.
Revised: However, this success probability depends on the transition kernels of the underlying MDPs, which might not be known.

Original: We can instead deﬁne planning as maximizing the following lower bound of the success probability, that can be expressed in terms of the low-level value v π .
Revised: We can instead deﬁne planning as maximizing the following lower bound of the success probability, that can be expressed in terms of the low-level value v π .

Original: Proposition 1 (Lower bound of success probability) .
Revised: Proposition 1 (Lower bound of success probability) .

Original: The success probability P ( s ∞ ∈ τ
Revised: The success probability P ( s ∞ ∈ τ

Original: π σ s 0 ) ≥ L ( σ ) of a plan σ is bounded from below by L ( σ ) :
Revised: π σ s 0 ) ≥ L ( σ ) of a plan σ is bounded from below by L ( σ ) :

Original: = (cid:81)
Revised: = (cid:81)

Original: | σ |− 1 i =1 v π ( σ i , σ i +1 ) , i.e. the product of the success probabilities of π on the sub-tasks deﬁned by ( σ i , σ i +1 ) .
Revised: | σ |− 1 i =1 v π ( σ i , σ i +1 ) , i.e. the product of the success probabilities of π on the sub-tasks deﬁned by ( σ i , σ i +1 ) .

Original: The straight-forward proof is given in Appendix A.1.
Revised: The straight-forward proof is given in Appendix A.1.

Original: Intuitively, L ( σ ) is a lower bound for the success of π σ , as it neglects the probability of “accidentally”
Revised: Intuitively, L ( σ ) is a lower bound for the success of π σ , as it neglects the probability of “accidentally”

Original: (due to stochasticity of the policy or transitions) running into the goal s ∞ before having executed the full plan.
Revised: (due to stochasticity of the policy or transitions) running into the goal s ∞ before having executed the full plan.

Original: We summarize:
Revised: We summarize:

Original: Deﬁnition 1 (Open-Loop Goal-Directed Planning) .
Revised: Deﬁnition 1 (Open-Loop Goal-Directed Planning) .

Original: Given a goal-directed policy π and its corresponding value oracle v π , we deﬁne planning as maximizing L ( σ ) over σ ∈ s 0 S ∗ s ∞ , i.e. the set of plans for task ( s 0 , s ∞ ) .
Revised: Given a goal-directed policy π and its corresponding value oracle v π , we deﬁne planning as maximizing L ( σ ) over σ ∈ s 0 S ∗ s ∞ , i.e. the set of plans for task ( s 0 , s ∞ ) .

Original: We deﬁne the high-level (HL) value v ∗ ( s 0 , s ∞ ) :
Revised: We deﬁne the high-level (HL) value v ∗ ( s 0 , s ∞ ) :

Original: = max σ L ( σ ) as the maximum value of the planning objective.
Revised: = max σ L ( σ ) as the maximum value of the planning objective.

Original: Note the difference between the low-level value v π and the high-level v ∗ .
Revised: Note the difference between the low-level value v π and the high-level v ∗ .

Original: v π ( s, s (cid:48) ) is the probability of the agent directly reaching s (cid:48) from s following π , whereas v ∗ ( s, s (cid:48) )
Revised: v π ( s, s (cid:48) ) is the probability of the agent directly reaching s (cid:48) from s following π , whereas v ∗ ( s, s (cid:48) )

Original: the probability reaching s (cid:48) from s under the optimal plan , which likely includes intermediate sub-goals.
Revised: the probability reaching s (cid:48) from s under the optimal plan , which likely includes intermediate sub-goals.

Original: In particular, v ∗ ≥ v π .
Revised: In particular, v ∗ ≥ v π .

Original: 2. AND/OR S EARCH T REE R EPRESENTATION
Revised: 2. AND/OR S EARCH T REE R EPRESENTATION

Original: In the following we cast the planning problem into a representation amenable to efﬁcient search.
Revised: In the following we cast the planning problem into a representation amenable to efﬁcient search.

Original: To this end, we use the natural compositionality of plans: We can concatenate a plan σ for the task ( s, s (cid:48) ) and a plan ˆ σ for the task ( s (cid:48) , s (cid:48)(cid:48) ) into a plan σ ◦ ˆ σ for the task ( s, s (cid:48)(cid:48) ) .
Revised: To this end, we use the natural compositionality of plans: We can concatenate a plan σ for the task ( s, s (cid:48) ) and a plan ˆ σ for the task ( s (cid:48) , s (cid:48)(cid:48) ) into a plan σ ◦ ˆ σ for the task ( s, s (cid:48)(cid:48) ) .

Original: Conversely, we can decompose any given plan σ for task ( s 0 , s ∞ ) by splitting it at any sub-goal s ∈ σ into σ =
Revised: Conversely, we can decompose any given plan σ for task ( s 0 , s ∞ ) by splitting it at any sub-goal s ∈ σ into σ =

Original: σ l ◦ σ r , where σ l is the “left” sub-plan for task ( s 0 , s ) , and σ r is the “right” sub-plan for task ( s, s ∞ ) .
Revised: σ l ◦ σ r , where σ l is the “left” sub-plan for task ( s 0 , s ) , and σ r is the “right” sub-plan for task ( s, s ∞ ) .

Original: Trivially, the planning objective and the optimal high-level value factorize wrt.
Revised: Trivially, the planning objective and the optimal high-level value factorize wrt.

Original: to this decomposition:
Revised: to this decomposition:

Original: [Equation]
Revised: [Equation]

Original: This allows us to recursively reformulate planning as:
Revised: This allows us to recursively reformulate planning as:

Original: [Equation]
Revised: [Equation]

Original: The above equations are the Bellman equations and the Bellman optimality equations for the classical single pair shortest path problem in graphs, where edge weights are given by − log v π ( s, s (cid:48) ) .
Revised: The above equations are the Bellman equations and the Bellman optimality equations for the classical single pair shortest path problem in graphs, where edge weights are given by − log v π ( s, s (cid:48) ) .

Original: We can represent this planning problem by an AND/OR search tree (Nilsson, N. J., 1980) with alternating levels of OR and AND nodes.
Revised: We can represent this planning problem by an AND/OR search tree (Nilsson, N. J., 1980) with alternating levels of OR and AND nodes.

Original: An OR node, also termed an action node , is labeled by a task ( s, s (cid:48)(cid:48) )
Revised: An OR node, also termed an action node , is labeled by a task ( s, s (cid:48)(cid:48) )

Original: ∈ S × S ; the root of the search tree is an OR node labeled by the original task ( s 0 , s ∞ ) .
Revised: ∈ S × S ; the root of the search tree is an OR node labeled by the original task ( s 0 , s ∞ ) .

Original: A terminal OR node ( s, s (cid:48)(cid:48) ) has a value v π ( s, s (cid:48)(cid:48) ) attached to it, which reﬂects the success probability of π s(cid:48)(cid:48) for completing the sub-task ( s, s (cid:48)(cid:48) ) .
Revised: A terminal OR node ( s, s (cid:48)(cid:48) ) has a value v π ( s, s (cid:48)(cid:48) ) attached to it, which reﬂects the success probability of π s for completing the sub-task ( s, s (cid:48)(cid:48) ) .

Original: Each non-terminal OR node has |S| + 1 AND nodes as children.
Revised: Each -terminal OR node has S AND nodes as children.

Original: Each of these is labeled bya triple( s, s (cid:48) , s (cid:48)(cid:48) ) for s (cid:48) ∈ ¯ S , which correspond to inserting 
Revised: Each of these is labeled by triple , which correspond to inserting a the , or AND or

Original: In this representation, plans are induced by solution trees .
Revised: In this representation, plans are induced by solution trees .

Original: A solution tree T σ is a sub-tree of the complete AND/OR search tree, with the properties that (i) the root ( s 0 , s ∞ ) ∈ T
Revised: A solution tree T σ is a sub-tree of the complete AND/OR search tree, with the properties that (i) the root ( s 0 , s ∞ ) ∈ T

Original: σ , (ii) each OR node in T σ has at most one child in T σ and (iii) each AND node in T σ as two children in T σ .
Revised: σ , (ii) each OR node in T σ has at most one child in T σ and (iii) each AND node in T σ as two children in T σ .

Original: The plan σ and its objective L ( σ ) can be computed from T σ by a depth-ﬁrst traversal of T σ .
Revised: The plan σ and its objective L ( σ ) can be computed from T σ by a depth-ﬁrst traversal of T σ .

Original: The correspondence of sub-trees to plans is many-to-one, as T σ , in addition to the plan itself, contains the order in which the plan was constructed.
Revised: The correspondence of sub-trees to plans is many-to-one, as T σ , in addition to the plan itself, contains the order in which the plan was constructed.

Original: Figure 6 in Section 5.3 shows an example for a search and solution tree.
Revised: Figure 6 in Section 5.3 shows an example for a search and solution tree.

Original: Below we will discuss how to construct a favourable search order heuristic.
Revised: Below we will discuss how to construct a favourable search order heuristic.

Original:  B EST -FIRST AND/OR P LANNING
Revised:  B EST -F

Original: [Table]
Revised: [Table]

Original: The planning problem from Deﬁnition 1 can be solved exactly by formulating it as shortest path problem from s 0 to s ∞ on a fully connected graph with vertex set S with non-negative edge weights given by − log v π and applying a classical Single Source or All Pairs Shortest Path (SSSP /
Revised: The planning problem from Deﬁnition 1 can be solved exactly by formulating it as shortest path problem from s 0 to s ∞ on a fully connected graph with vertex set S with non-negative edge weights given by − log v π and applying a classical Single Source or All Pairs Shortest Path (SSSP /

Original: APSP) planner.
Revised: APSP) planner.

Original: This approach is appropriate if one wants to solve all goal-directed tasks in a single MDP.
Revised: This approach is appropriate if one wants to solve all goal-directed tasks in a single MDP.

Original: Here, we focus however on the multitask setting described above, where the agent is given a new MDP with a single task ( s 0 , s ∞ ) every episode.
Revised: Here, we focus however on the multitask setting described above, where the agent is given a new MDP with a single task ( s 0 , s ∞ ) every episode.

Original: In this case, solving the SSSP / APSP problem is not feasible: Tabulating all graphs weights − log v π ( s, s (cid:48) ) would require |S| 2 evaluations of v π ( s, s (cid:48) ) for all pairs ( s, s (cid:48) ) .
Revised: In this case, solving the SSSP / APSP problem is not feasible: Tabulating all graphs weights − log v π ( s, s (cid:48) ) would require |S| 2 evaluations of v π ( s, s (cid:48) ) for all pairs ( s, s (cid:48) ) .

Original: In practice, approximate evaluations of v π could be implemented by e.g. actually running the policy π , or by calls to a powerful function approximator, both of which are often too costly to exhaustively evaluate for large state-spaces S .
Revised: In practice, approximate evaluations of v π could be implemented by e.g. actually running the policy π , or by calls to a powerful function approximator, both of which are often too costly to exhaustively evaluate for large state-spaces S .

Original: Instead, we tailor an algorithm for approximate planning to the multi-task setting, which we call Divide-and-Conquer MCTS (DC-MCTS).
Revised: Instead, we tailor an algorithm for approximate planning to the multi-task setting, which we call Divide-and-Conquer MCTS (DC-MCTS).

Original: To evaluate v π as sparsely as possible,
Revised: To evaluate v π as sparsely as possible,

Original: DC-MCTS critically makes use of two learned search heuristics that transfer knowledge from previously encountered MDPs / tasks to new problem instance: (i) a distribution p ( s (cid:48) | s, s (cid:48)(cid:48) ) , called the policy prior, for proposing promising intermediate sub-goals s (cid:48) for a task ( s, s (cid:48)(cid:48) ) ; and (ii) a learned approximation v to the high-level value v ∗ for bootstrap evaluation of partial plans.
Revised: DC-MCTS critically makes use of two learned search heuristics that transfer knowledge from previously encountered MDPs / tasks to new problem instance: (i) a distribution p ( s (cid:48) | s, s (cid:48)(cid:48) ) , called the policy prior, for proposing promising intermediate sub-goals s (cid:48) for a task ( s, s (cid:48)(cid:48) ) ; and (ii) a learned approximation v to the high-level value v ∗ for bootstrap evaluation of partial plans.

Original: In the following we present DC-MCTS and discuss design choices and training for the two search heuristics.
Revised: In the following we present DC-MCTS and discuss design choices and training for the two search heuristics.

Original: 3. D IVIDE - AND -C ONQUER M ONTE C ARLO T REE S EARCH
Revised: 3. D IVIDE - AND -C ONQUER M ONTE C ARLO T REE S EARCH

Original: The input to the DC-MCTS planner is an MDP encoding c M , a task ( s 0 , s ∞ ) as well as a planning budget, i.e. a maximum number B ∈ N of v π oracle evaluations.
Revised: The input to the DC-MCTS planner is an MDP encoding c M , a task ( s 0 , s ∞ ) as well as a planning budget, i.e. a maximum number B ∈ N of v π oracle evaluations.

Original: At each stage, DC-MCTS maintains a (partial) AND/OR search tree T whose root is the OR node ( s 0 , s ∞ ) corresponding to the original task.
Revised: At each stage, DC-MCTS maintains a (partial) AND/OR search tree T whose root is the OR node ( s 0 , s ∞ ) corresponding to the original task.

Original: Every OR node ( s, s (cid:48)(cid:48) )
Revised: Every OR node ( s, s (cid:48)(cid:48) )

Original: ∈ T maintains an estimate V ( s, s (cid:48)(cid:48) )
Revised: ∈ T maintains an estimate V ( s, s (cid:48)(cid:48) )

Original: ≈ v ∗ ( s, s (cid:48)(cid:48) ) of its high-level value.
Revised: ≈ v ∗ ( s, s (cid:48)(cid:48) ) of its high-level value.

Original: DC-MCTS searches for a plan by iteratively constructing the search tree T with T RAVERSE until the budget is exhausted, see Algorithm 1.
Revised: DC-MCTS searches for a plan by iteratively constructing the search tree T with T RAVERSE until the budget is exhausted, see Algorithm 1.

Original: During each traversal, if a leaf node of T is reached, it is expanded, followed by a recursive bottom-up backup to update the value estimates V of all OR nodes visited in this traversal.
Revised: During each traversal, if a leaf node of T is reached, it is expanded, followed by a recursive bottom-up backup to update the value estimates V of all OR nodes visited in this traversal.

Original: After this search phase, the currently best plan is extracted from T by
Revised: After this search phase, the currently best plan is extracted from T by

Original: E XTRACT P LAN (essentially depth-ﬁrst traversal, see Algorithm 2 in the Appendix).
Revised: E XTRACT P LAN (essentially depth-ﬁrst traversal, see Algorithm 2 in the Appendix).

Original: In the following we brieﬂy describe the main methods of the search.
Revised: In the following we brieﬂy describe the main methods of the search.

Original: We illustrate DC-MCTS in Figure 1.
Revised: We illustrate DC-MCTS in Figure 1.

Original: T RAVERSE and S ELECT
Revised: T RAVERSE and S ELECT

Original: T is traversed from the root ( s 0 , s ∞ ) to ﬁnd a promising node to expand.
Revised: T is traversed from the root ( s 0 , s ∞ ) to ﬁnd a promising node to expand.

Original: At an OR node ( s, s (cid:48)(cid:48) ) , S ELECT chooses one of its children s (cid:48) ∈ ¯ S to traverse into, including s = ∅ for not inserting any further sub-goals into this branch.
Revised: At an OR node ( s, s (cid:48)(cid:48) ) , S ELECT chooses one of its children s (cid:48) ∈ ¯ S to traverse into, including s = ∅ for not inserting any further sub-goals into this branch.

Original: We implemented S ELECT by the pUCT (Rosin, 2011) rule, which consists of picking the next node s (
Revised: We implemented S ELECT by the pUCT (Rosin, 2011) rule, which consists of picking the next node s (

Original: cid:48) ∈ ¯ S based on maximizing the following score:
Revised: cid:48) ∈ ¯ S based on maximizing the following score:

Original: [Equation]
Revised: [Equation]

Original: where N ( s, s (cid:48) ) , N ( s, s (cid:48) , s (cid:48)(cid:48) ) are the visit counts of the OR node ( s, s (cid:48) ) , AND node ( s, s (cid:48) , s (cid:48)(cid:48) ) respectively.
Revised: where N ( s, s (cid:48) ) , N ( s, s (cid:48) , s (cid:48)(cid:48) ) are the visit counts of the OR node ( s, s (cid:48) ) , AND node ( s, s (cid:48) , s (cid:48)(cid:48) ) respectively.

Original: The ﬁrst term is the exploitation component, guiding the search to sub-goals that currently look promising, i.e. have high estimated value.
Revised: The ﬁrst term is the exploitation component, guiding the search to sub-goals that currently look promising, i.e. have high estimated value.

Original: The second term is the exploration term favoring nodes with low visit counts.
Revised: The second term is the exploration term favoring nodes with low visit counts.

Original: Crucially, it is explicitly scaled by the policy prior p ( s (cid:48) | s, s (cid:48)(cid:48) ) to guide exploration.
Revised: Crucially, it is explicitly scaled by the policy prior p ( s (cid:48) | s, s (cid:48)(cid:48) ) to guide exploration.

Original: At an AND node ( s, s (cid:48) , s (cid:48)(cid:48) ) , T RAVERSE traverses into both the left ( s, s (cid:48) ) and right child ( s (cid:48) , s (cid:48)(cid:48) ) .
Revised: At an AND node ( s, s (cid:48) , s (cid:48)(cid:48) ) , T RAVERSE traverses into both the left ( s, s (cid:48) ) and right child ( s (cid:48) , s (cid:48)(cid:48) ) .

Original: As the two sub-problems are solved independently, computation from there on can be carried out in parallel.
Revised: As the two sub-problems are solved independently, computation from there on can be carried out in parallel.

Original: All nodes visited in a single traversal form a solution tree T σ with plan σ .
Revised: All nodes visited in a single traversal form a solution tree T σ with plan σ .

Original: E XPAND
Revised: E XPAND

Original: If a leaf OR node ( s, s (cid:48)(cid:48) ) is reached during the traversal and its depth is smaller than a given maximum depth, it is expanded by evaluating the high- and low-level values v ( s, s (cid:48)(cid:48) ) , v π ( s, s (cid:48)(cid:48) ) .
Revised: If a leaf OR node ( s, s (cid:48)(cid:48) ) is reached during the traversal and its depth is smaller than a given maximum depth, it is expanded by evaluating the high- and low-level values v ( s, s (cid:48)(cid:48) ) , v π ( s, s (cid:48)(cid:48) ) .

Original: The initial value of the node is deﬁned as max of both values, as by deﬁnition v ∗ ≥ v π , i.e. further planning should only increase the success probability on a sub-task.
Revised: The initial value of the node is deﬁned as max of both values, as by deﬁnition v ∗ ≥ v π , i.e. further planning should only increase the success probability on a sub-task.

Original: We also evaluate the policy prior p ( s (cid:48) | s, s (cid:48)(cid:48) ) for all s (cid:48) , yielding the proposal distribution over sub-goals used in SELECT.
Revised: We also evaluate the policy prior p ( s (cid:48) | s, s (cid:48)(cid:48) ) for all s (cid:48) , yielding the proposal distribution over sub-goals used in SELECT.

Original: Each node expansion costs one unit of budget B .
Revised: Each node expansion costs one unit of budget B .

Original: B ACKUP and U PDATE
Revised: B ACKUP and U PDATE

Original: We deﬁne the return G σ of the traversal tree T σ as follows.
Revised: We deﬁne the return G σ of the traversal tree T σ as follows.

Original: Let a reﬁnement
Revised: Let a reﬁnement

Original: T + σ of T σ be a solution tree such that T σ ⊆ T + σ , thus representing a plan σ + that has all sub-goals of σ with additional inserted sub-goals.
Revised: T + σ of T σ be a solution tree such that T σ ⊆ T + σ , thus representing a plan σ + that has all sub-goals of σ with additional inserted sub-goals.

Original: G σ is now deﬁned as the value of the objective L ( σ + ) of the optimal reﬁnement of T σ , i.e. it reﬂects how well one could do on task ( s 0 , s ∞ ) by starting from the plan σ and reﬁning it.
Revised: G σ is now deﬁned as the value of the objective L ( σ + ) of the optimal reﬁnement of T σ , i.e. it reﬂects how well one could do on task ( s 0 , s ∞ ) by starting from the plan σ and reﬁning it.

Original: It can be computed by a simple back-up on the tree T σ that uses the bootstrap value v ≈ v ∗ at the leafs.
Revised: It can be computed by a simple back-up on the tree T σ that uses the bootstrap value v ≈ v ∗ at the leafs.

Original: As v ∗ ( s 0 , s ∞ ) ≥ G σ ≥ L ( σ ) and G σ ∗ = v ∗ ( s 0 , s ∞ ) for the optimal plan σ ∗ , we can use G σ to update the value estimate V .
Revised: As v ∗ ( s 0 , s ∞ ) ≥ G σ ≥ L ( σ ) and G σ ∗ = v ∗ ( s 0 , s ∞ ) for the optimal plan σ ∗ , we can use G σ to update the value estimate V .

Original: Like in other MCTS variants, we employ a running average operation (line 15-16 in T RAVERSE ).
Revised: Like in other MCTS variants, we employ a running average operation (line 15-16 in T RAVERSE ).

Original: 3. D ESIGNING AND T RAINING S EARCH H EURISTICS
Revised: 3. D ESIGNING AND T RAINING S EARCH H EURISTICS

Original: Search results and experience from previous tasks can improve DC-MCTS on new problems via adapting the search heuristics, i.e. the policy prior p and the approximate value function v as follows.
Revised: Search results and experience from previous tasks can improve DC-MCTS on new problems via adapting the search heuristics, i.e. the policy prior p and the approximate value function v as follows.

Original: Bootstrap Value Function We parametrize v ( s, s (cid:48)
Revised: Bootstrap Value Function We parametrize v ( s, s (cid:48)

Original: | c M ) ≈ v ∗ ( s, s (cid:48) | c M ) as a neural network that takes as inputs the current task consisting of ( s, s (cid:48) ) and the MDP encoding c M .
Revised: | c M ) ≈ v ∗ ( s, s (cid:48) | c M ) as a neural network that takes as inputs the current task consisting of ( s, s (cid:48) ) and the MDP encoding c M .

Original: A straight-forward approach to train v is to regress it towards the non-parametric value estimates V computed by DCMCTS on previous problem instances.
Revised: A straight-forward approach to train v is to regress it towards the non-parametric value estimates V computed by DCMCTS on previous problem instances.

Original: However, initial results indicated that this leads to v being overly optimistic, an observation also made in Kaelbling (1993).
Revised: However, initial results indicated that this leads to v being overly optimistic, an observation also made in Kaelbling (1993).

Original: We therefore used more conservative training targets, that are computed by backing the low-level values v π up the solution tree T σ of the plan σ return by DC-MCTS.
Revised: We therefore used more conservative training targets, that are computed by backing the low-level values v π up the solution tree T σ of the plan σ return by DC-MCTS.

Original: Details can be found in Appendix B.1.
Revised: Details can be found in Appendix B.1.

Original: Policy Prior Best-ﬁrst search guided by a policy prior p can be understood as policy improvement of p as described in Silver et al.
Revised: Policy Prior Best-ﬁrst search guided by a policy prior p can be understood as policy improvement of p as described in Silver et al.

Original: Therefore, a straight-forward way of training p is to distill the search results back into into the policy prior, e.g. by behavioral cloning.
Revised: Therefore, a straight-forward way of training p is to distill the search results back into into the policy prior, e.g. by behavioral cloning.

Original: When applying this to
Revised: When applying this to

Original: DC-MCTS in our setting, we found empirically that this yielded very slow improvement when starting from an untrained, uniform prior p .
Revised: DC-MCTS in our setting, we found empirically that this yielded very slow improvement when starting from an untrained, uniform prior p .

Original: This is due to plans with non-zero success probability L >being very sparse in S ∗ , equivalent to the sparse reward setting in regular MDPs.
Revised: This is due to plans with non-zero success probability L >being very sparse in S ∗ , equivalent to the sparse reward setting in regular MDPs.

Original: To address this issue, we propose to apply Hindsight Experience Replay (HER, Andrychowicz et al.
Revised: To address this issue, we propose to apply Hindsight Experience Replay (HER, Andrychowicz et al.

Original: Instead of training p exclusively on search results, we additionally execute plans σ in the environment and collect the resulting trajectories, i.e. the sequence of visited states, τ π σ s 

Revised: Instead of training p exclusively on search results, we additionally execute plans σ in the environment and collect the resulting trajectories, i.e. the sequence of visited states, τ π σ s

Original: =
Revised: =

Original: ( s 0 , s 1 , . .
Revised: ( s 0 , s 1 , . .

Original: , s T ) .
Revised: , s T ) .

Original: HER then proceeds with hindsight relabeling , i.e. taking τ π σ s 0 as an approximately optimal plan for the “ﬁctional” task ( s 0 , s T ) that is likely different from the actual task ( s 0 , s ∞ ) .
Revised: HER then proceeds with hindsight relabeling , i.e. taking τ π σ s 0 as an approximately optimal plan for the “ﬁctional” task ( s 0 , s T ) that is likely different from the actual task ( s 0 , s ∞ ) .

Original: In standard HER, these ﬁctitious expert demonstrations are used for imitation learning of goal-directed policies, thereby circumventing the sparse reward problem.
Revised: In standard HER, these ﬁctitious expert demonstrations are used for imitation learning of goal-directed policies, thereby circumventing the sparse reward problem.

Original: We can apply HER to train p in our setting by extracting any ordered triplet ( s t 1 , s t 2 , s t 3 ) from τ π σ s 0 and use it as supervised learning targets for p .
Revised: We can apply HER to train p in our setting by extracting any ordered triplet ( s t 1 , s t 2 , s t 3 ) from τ π σ s 0 and use it as supervised learning targets for p .

Original: This is a sensible procedure, as p would then learn to predict optimal sub-goals s ∗ t 2 for sub-tasks ( s ∗ t 1 , s ∗ t 3 ) under the assumption that the data was generated by an oracle producing optimal plans τ π σ s 0 = σ ∗ .
Revised: This is a sensible procedure, as p would then learn to predict optimal sub-goals s ∗ t 2 for sub-tasks ( s ∗ t 1 , s ∗ t 3 ) under the assumption that the data was generated by an oracle producing optimal plans τ π σ s 0 = σ ∗ .

Original: We have considerable freedom in choosing which triplets to extract from data and use as supervision with HER.
Revised: We have considerable freedom in choosing which triplets to extract from data and use as supervision with HER.

Original: In our experiments we use a temporally balanced parsing, which creates triplets ( s t , s t +∆ / 2 , s t +∆ ) such that the resulting policy prior should then preferentially propose sub-goals “in the middle” of the task.
Revised: In our experiments we use a temporally balanced parsing, which creates triplets ( s t , s t +∆ / 2 , s t +∆ ) such that the resulting policy prior should then preferentially propose sub-goals “in the middle” of the task.

Original: In Appendix A.4 we discuss this aspect in more detail, and present alternative parsers.
Revised: In Appendix A.4 we discuss this aspect in more detail, and present alternative parsers.

Original: 3. A LGORITHMIC C OMPLEXITY OF DC-MCTS
Revised: 3. A LGORITHMIC C OMPLEXITY OF DC-MCTS

Original: Denoting an optimal plan as σ ∗ , the complexity of DC-MCTS with optimal search policy prior p = p ∗ is O ( | σ ∗ |· |S| ) .
Revised: Denoting an optimal plan as σ ∗ , the complexity of DC-MCTS with optimal search policy prior p = p ∗ is O ( | σ ∗ |· |S| ) .

Original: This could potentially be reduced to O (log( | σ ∗ | )) when using progressive widening Coulom (2007); Chaslot et al.
Revised: This could potentially be reduced to O (log( | σ ∗ | )) when using progressive widening Coulom (2007); Chaslot et al.

Original: for fewer evaluations of p and perfect parallelization of tree traversals across multiple workers; for details see Appendix A.5.
Revised: for fewer evaluations of p and perfect parallelization of tree traversals across multiple workers; for details see Appendix A.5.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Goal-directed multi-task learning is an important special case of general RL and has been extensively studied.
Revised: Goal-directed multi-task learning is an important special case of general RL and has been extensively studied.

Original: Universal value functions (Schaul et al., 2015) have been established as compact representation for this setting (Kulkarni et al., 2016; Andrychowicz et al., 2017; Ghosh et al., 2018;
Revised: Universal value functions (Schaul et al., 2015) have been established as compact representation for this setting (Kulkarni et al., 2016; Andrychowicz et al., 2017; Ghosh et al., 2018;

Original: Dhiman et al., 2018).
Revised: Dhiman et al., 2018).

Original: This allows to use sub-goals as means for planning, as done in several works such as Kaelbling & Lozano-Pérez (2017);
Revised: This allows to use sub-goals as means for planning, as done in several works such as Kaelbling & Lozano-Pérez (2017);

Original: Gao et al.
Revised: Gao et al.

Original: Savinov et al.
Revised: Savinov et al.

Original: ; Stein et al.
Revised: ; Stein et al.

Original: Nasiriany et al.
Revised: Nasiriany et al.

Original: (2019), all of which rely on forward sequential planning.
Revised: (2019), all of which rely on forward sequential planning.

Original: Gabor et al.
Revised: Gabor et al.

Original: (2019) use
Revised: (2019) use

Original: MCTS for traditional sequential planning based on heuristics, sub-goals and macro-actions.
Revised: MCTS for traditional sequential planning based on heuristics, sub-goals and macro-actions.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) apply traditional graph planners to ﬁnd abstract sub-goal sequences.
Revised: (2018) apply traditional graph planners to ﬁnd abstract sub-goal sequences.

Original: We extend this line of work by showing that the abstraction of sub-goals affords more general search strategies than sequential planning.
Revised: We extend this line of work by showing that the abstraction of sub-goals affords more general search strategies than sequential planning.

Original: Work concurrent to ours has independently investigated non-sequential sub-goals planning: Jurgenson et al.
Revised: Work concurrent to ours has independently investigated non-sequential sub-goals planning: Jurgenson et al.

Original: (2019) propose a top-down policy gradient approach that learns to predict sub-goals in a hierarchical way.
Revised: (2019) propose a top-down policy gradient approach that learns to predict sub-goals in a hierarchical way.

Original: Nasiriany et al.
Revised: Nasiriany et al.

Original: (2019) propose gradient-based search jointly over a ﬁxed number of sub-goals for continuous goal spaces.
Revised: (2019) propose gradient-based search jointly over a ﬁxed number of sub-goals for continuous goal spaces.

Original: In contrast, DC-MCTS is able to dynamically determine the complexity of the optimal plan.
Revised: In contrast, DC-MCTS is able to dynamically determine the complexity of the optimal plan.

Original: The proposed DC-MCTS planner is a MCTS (Browne et al., 2012) variant, inspired by recent advances in best-ﬁrst or guided search, such as AlphaZero (Silver et al., 2018).
Revised: The proposed DC-MCTS planner is a MCTS (Browne et al., 2012) variant, inspired by recent advances in best-ﬁrst or guided search, such as AlphaZero (Silver et al., 2018).

Original: It can also be understood as a heuristic, guided version of the classic Floyd-Warshall algorithm which exhaustively computes all shortest paths.
Revised: It can also be understood as a heuristic, guided version of the classic Floyd-Warshall algorithm which exhaustively computes all shortest paths.

Original: In the special case of planar graphs, small sub-goal sets, also known as vertex separators, can be constructed that favourably partition the remaining graph, leading to linear time ASAP algorithms (Henzinger et al., 1997).
Revised: In the special case of planar graphs, small sub-goal sets, also known as vertex separators, can be constructed that favourably partition the remaining graph, leading to linear time ASAP algorithms (Henzinger et al., 1997).

Original: The heuristic sub-goal proposer p that guides
Revised: The heuristic sub-goal proposer p that guides

Original: DC-MCTS can be loosely understood as a probabilistic version of a vertex separator.
Revised: DC-MCTS can be loosely understood as a probabilistic version of a vertex separator.

Original: Nowak-Vila et al.
Revised: Nowak-Vila et al.

Original: (2016) also consider neural networks that mimic divide-and-conquer algorithms similar to the sub-goal proposals used here.
Revised: (2016) also consider neural networks that mimic divide-and-conquer algorithms similar to the sub-goal proposals used here.

Original: However, while we do policy improvement for the proposals using search and HER, the networks in Nowak-Vila et al.
Revised: However, while we do policy improvement for the proposals using search and HER, the networks in Nowak-Vila et al.

Original: (2016) are purely trained by policy gradient methods.
Revised: (2016) are purely trained by policy gradient methods.

Original: Decomposing tasks into sub-problems has been formalized as pseudo trees (Freuder & Quinn, 1985) and AND/OR graphs (Nilsson, N. J., 1980).
Revised: Decomposing tasks into sub-problems has been formalized as pseudo trees (Freuder & Quinn, 1985) and AND/OR graphs (Nilsson, N. J., 1980).

Original: The latter have been used especially in the context of optimization (Larrosa et al., 2002;
Revised: The latter have been used especially in the context of optimization (Larrosa et al., 2002;

Original: Jégou & Terrioux, 2003; Dechter & Mateescu, 2004; Marinescu &
Revised: Jégou & Terrioux, 2003; Dechter & Mateescu, 2004; Marinescu &

Original: Dechter, 2004).
Revised: Dechter, 2004).

Original: Our approach is related to work on using AND/OR trees for sub-goal ordering in the context of logic inference (Ledeniov & Markovitch, 1998).
Revised: Our approach is related to work on using AND/OR trees for sub-goal ordering in the context of logic inference (Ledeniov & Markovitch, 1998).

Original: While DC-MCTS is closely related to the AO ∗ algorithm (Nilsson, N. J., 1980), which is the generalization of the heuristic A ∗ search to
Revised: While DC-MCTS is closely related to the AO ∗ algorithm (Nilsson, N. J., 1980), which is the generalization of the heuristic A ∗ search to

Original: AND/OR search graphs, interesting differences exist: AO ∗ assumes a ﬁxed search heuristic, which is required to be lower bound on the cost-to-go.
Revised: AND/OR search graphs, interesting differences exist: AO ∗ assumes a ﬁxed search heuristic, which is required to be lower bound on the cost-to-go.

Original: In contrast, we employ learned value functions and policy priors that are not required to be exact bounds.
Revised: In contrast, we employ learned value functions and policy priors that are not required to be exact bounds.

Original: Relaxing this assumption, thereby violating the principle of “optimism in the face of uncertainty”, necessitates explicit exploration incentives in the S ELECT method.
Revised: Relaxing this assumption, thereby violating the principle of “optimism in the face of uncertainty”, necessitates explicit exploration incentives in the S ELECT method.

Original: Alternatives for searching AND/OR spaces include proof-number search, recently applied to chemical synthesis planning (Kishimoto et al., 2019).
Revised: Alternatives for searching AND/OR spaces include proof-number search, recently applied to chemical synthesis planning (Kishimoto et al., 2019).

Original: Very recent work concurrent to ours has focused on relevant research directions:
Revised: Very recent work concurrent to ours has focused on relevant research directions:

Original: Wang et al.
Revised: Wang et al.

Original: (2020) introduce LA-MCTS as a ‘meta-algorithm’ for black-box optimization, and Chen et al.
Revised: (2020) introduce LA-MCTS as a ‘meta-algorithm’ for black-box optimization, and Chen et al.

Original: (2020) propose Retro*, a neural-based
Revised: (2020) propose Retro*, a neural-based

Original: A*-like algorithm for molecule synthesis that is also based on AND/OR trees.
Revised: A*-like algorithm for molecule synthesis that is also based on AND/OR trees.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the proposed DC-MCTS algorithm on navigation in grid-world mazes as well as on a challenging continuous control version of the same problem, comparing it to standard sequential
Revised: We evaluate the proposed DC-MCTS algorithm on navigation in grid-world mazes as well as on a challenging continuous control version of the same problem, comparing it to standard sequential

Original: MCTS (in sub-goal space) based on the fraction of “solved” mazes by executing their plans.
Revised: MCTS (in sub-goal space) based on the fraction of “solved” mazes by executing their plans.

Original: The
Revised: The

Original: MCTS baseline was implemented by restricting the DC-MCTS algorithm to only expand the “right”sub-problem in line 10 of Algorithm 1; the value G left for the “left” sub-problem is computed as in line 7, i.e. using the low-level value v π . This forces MCTS to plan forward and sequentially, as each next step needs to be reachable from the previous state, as evaluated by v π . All remaining parameters and design choice were the same for both planners except where explicitly mentioned otherwise.
Revised: MCTS baseline was implemented by restricting the DC-MCTS algorithm to only expand the “right”

Original: Legend: =
Revised: Legend: =

Original: s 0 /start =
Revised: s 0 /start =

Original: s ∞ /goal
Revised: s ∞ /goal

Original: =
Revised: =

Original: wall = empty = p ( s (cid:48)(cid:48) | s, s (cid:48) )
Revised: wall = empty = p ( s (cid:48)(cid:48) | s, s (cid:48) )

Original: = sub-goals 
Revised: = sub-goals sub-problem in line 10 of Algorithm 1; the value G left for the “left” sub-problem is computed as in line 7, i.e. using the low-level value v π . This forces MCTS to plan forward and sequentially, as each next step needs to be reachable from the previous state, as evaluated by v π . All remaining parameters and design choice were the same for both planners except where explicitly mentioned otherwise.

Original: 5. G RID -W ORLD M AZES
Revised: 5. G RID -W ORLD M AZES

Original: Each task consists of a new, procedurally generated maze on a 21 × 21 grid with start and goal locations ( s 0 , s ∞ ) ∈ { 1 , . . .
Revised: Each task consists of a new, procedurally generated maze on a 21 × 21 grid with start and goal locations ( s 0 , s ∞ ) ∈ { 1 , . . .

Original: , 21 } 2 , see Figure 2.
Revised: , 21 } 2 , see Figure 2.

Original: Task difﬁculty was controlled by the density of walls d (under connectedness constraint), where the easiest setting d = 0 .
Revised: Task difﬁculty was controlled by the density of walls d (under connectedness constraint), where the easiest setting d = 0 .

Original: 0 corresponds to no walls and the most difﬁcult one d = 1 . 0 implies so-called perfect or singly-connected mazes.
Revised: 0 corresponds to no walls and the most difﬁcult one d = 1 . 0 implies so-called perfect or singly-connected mazes.

Original: The task embedding c M was given as the maze layout and ( s 0 , s ∞ ) encoded together as a feature map of 21 × 21 categorical variables with 4 categories each (empty, wall, start and goal location).
Revised: The task embedding c M was given as the maze layout and ( s 0 , s ∞ ) encoded together as a feature map of 21 × 21 categorical variables with 4 categories each (empty, wall, start and goal location).

Original: The underlying
Revised: The underlying

Original: MDPs have 5 primitive actions: up, down, left, right and NOOP.
Revised: MDPs have 5 primitive actions: up, down, left, right and NOOP.

Original: For sake of simplicity, we ﬁrst tested our proposed approach by hard-coding a low-level policy π 0 as well as its value oracle v π 0 in the following way.
Revised: For sake of simplicity, we ﬁrst tested our proposed approach by hard-coding a low-level policy π 0 as well as its value oracle v π 0 in the following way.

Original: If in state s and conditioned on a goal s (cid:48) , and if s is adjacent to s (cid:48) , π 0 s (cid:48) successfully reaches s (cid:48) with probability 1 in one step, i.e. v π 0 ( s, s (cid:48) )
Revised: If in state s and conditioned on a goal s (cid:48) , and if s is adjacent to s (cid:48) , π 0 s (cid:48) successfully reaches s (cid:48) with probability 1 in one step, i.e. v π 0 ( s, s (cid:48) )

Original: = 1 ; otherwise v π 0 ( s, s (cid:48) )
Revised: = 1 ; otherwise v π 0 ( s, s (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: If π 0 s (cid:48) is nevertheless executed, the agent moves to a random empty tile adjacent to s .
Revised: If π 0 s (cid:48) is nevertheless executed, the agent moves to a random empty tile adjacent to s .

Original: Therefore, π 0 is the “most myopic” goal-directed policy that can still navigate everywhere.
Revised: Therefore, π 0 is the “most myopic” goal-directed policy that can still navigate everywhere.

Original: For each maze, MCTS and DC-MCTS were given a search budget of 200 calls to the low-level value oracle v π 0 .
Revised: For each maze, MCTS and DC-MCTS were given a search budget of 200 calls to the low-level value oracle v π 0 .

Original: We implemented the search heuristics, i.e. policy prior p and high-level value function v , as convolutional neural networks (CNNs) which operate on input c M ; details for the network architectures are given in Appendix B.3.
Revised: We implemented the search heuristics, i.e. policy prior p and high-level value function v , as convolutional neural networks (CNNs) which operate on input c M ; details for the network architectures are given in Appendix B.3.

Original: With untrained networks, both planners were unable to solve the task (<2% success probability), as shown in Figure 3.
Revised: With untrained networks, both planners were unable to solve the task (<2% success probability), as shown in Figure 3.

Original: This illustrates that a search budget of 200 evaluations of v π 0 is insufﬁcient for unguided planners to ﬁnd a feasible path in most mazes.
Revised: This illustrates that a search budget of 200 evaluations of v π 0 is insufﬁcient for unguided planners to ﬁnd a feasible path in most mazes.

Original: This is consistent with standard exhaustive SSSP / APSP graph planners requiring 21 4 > 10 5 (cid:29)evaluations for optimal planning in the worst case on these tasks.
Revised: This is consistent with standard exhaustive SSSP / APSP graph planners requiring 21 4 > 10 5 (cid:29)evaluations for optimal planning in the worst case on these tasks.

Original: Next, we trained both search heuristics v and p as detailed in Section 3.2.
Revised: Next, we trained both search heuristics v and p as detailed in Section 3.2.

Original: In particular, the sub-goal proposal p was also trained on hindsight-relabeled experience data, where for DC-MCTS we used the temporally balanced parser and for MCTS the corresponding left-ﬁrst parser (see Appendix A.4).
Revised: In particular, the sub-goal proposal p was also trained on hindsight-relabeled experience data, where for DC-MCTS we used the temporally balanced parser and for MCTS the corresponding left-ﬁrst parser (see Appendix A.4).

Original: Training of the heuristics greatly improved the performance of both planners.
Revised: Training of the heuristics greatly improved the performance of both planners.

Original: Figure 3 shows learning curves for mazes with wall density d = 0 .
Revised: Figure 3 shows learning curves for mazes with wall density d = 0 .

Original: 75 , as mean and std over 20 different hyperparameters.
Revised: 75 , as mean and std over 20 different hyperparameters.

Original: DC-MCTS exhibits substantially improved performance compared to MCTS, and when compared at equal performance levels, DC-MCTS requires 5 to 10 -times fewer training episodes than MCTS.
Revised: DC-MCTS exhibits substantially improved performance compared to MCTS, and when compared at equal performance levels, DC-MCTS requires 5 to 10 -times fewer training episodes than MCTS.

Original: An example of a learned sub-goal proposal p for DC-MCTS is visualized in Figure 2 (further examples are given in the Appendix in Figure 8).
Revised: An example of a learned sub-goal proposal p for DC-MCTS is visualized in Figure 2 (further examples are given in the Appendix in Figure 8).

Original: Probability mass concentrates on promising sub-goals that are far from both start and goal, approximately partitioning the task into equally hard sub-tasks.
Revised: Probability mass concentrates on promising sub-goals that are far from both start and goal, approximately partitioning the task into equally hard sub-tasks.

Original: Next, we investigated the performance of both MCTS and DC-MCTS in challenging continuous control environments with non-trivial low-level policies.
Revised: Next, we investigated the performance of both MCTS and DC-MCTS in challenging continuous control environments with non-trivial low-level policies.

Original: We embedded the grid-world mazes into a physical 3D environment simulated by MuJoCo Todorov et al.
Revised: We embedded the gridworld mazes into a physical 3D environment simulated by MuJoCo Todorov et al.

Original: (2012), rendering each grid-world cell as 4m × 4m cell in physical space.
Revised: (2012), rendering each grid-world cell as 4m × 4m cell in physical space.

Original: The agent is embodied by a quadruped “ant” body; for illustration see Figure 4.
Revised: The agent is embodied by a quadruped “ant” body; for illustration see Figure 4.

Original: For the low-level policy π m , we pre-trained a goal-directed neural network controller that gets as inputs proprioceptive features (e.g. some joint angles and velocities) of the ant body as well as a 3D-vector pointing from its current position to a target position.
Revised: For the low-level policy π m , we pre-trained a goal-directed neural network controller that gets as inputs proprioceptive features (e.g. some joint angles and velocities) of the ant body as well as a 3D-vector pointing from its current position to a target position.

Original: π m was trained to navigate to targets randomly placed less than 1.5 m away in an open area (no walls), using MPO (Abdolmaleki et al., 2018).
Revised: π m was trained to navigate to targets randomly placed less than 1.5 m away in an open area (no walls), using MPO (Abdolmaleki et al., 2018).

Original: See Appendix B.4 for more details.
Revised: See Appendix B.4 for more details.

Original: If unobstructed, π m can walk in a straight line towards its current goal.
Revised: If unobstructed, π m can walk in a straight line towards its current goal.

Original: However, this policy receives no visual input and thus can only avoid walls when guided with appropriate sub-goals.
Revised: However, this policy receives no visual input and thus can only avoid walls when guided with appropriate sub-goals.

Original: To establish an interface between the low-level π m and the planners, we used another CNN to approximate the low-level value oracle v π m ( s 0 , s ∞ | c M ) :
Revised: To establish an interface between the low-level π m and the planners, we used another CNN to approximate the low-level value oracle v π m ( s 0 , s ∞ | c M ) :

Original: It was trained to predict whether π m will succeed in solving the navigation tasks ( s 0 , s ∞ ) , c M .
Revised: It was trained to predict whether π m will succeed in solving the navigation tasks ( s 0 , s ∞ ) , c M .

Original: Its input is the corresponding discrete grid-world representation c M of the maze ( 21 × 21 feature map of categoricals as described above, details in Appendix).
Revised: Its input is the corresponding discrete grid-world representation c M of the maze ( 21 × 21 feature map of categoricals as described above, details in Appendix).

Original: Note that this setting is still challenging:
Revised: Note that this setting is still challenging:

Original: In initial experiments we veriﬁed that a model-free baseline (also based on MPO, without HER) with access to state abstraction and low-level controller, only solved about 10% of the mazes aftermillion episodes due to the extremely sparse rewards.
Revised: In initial experiments we veriﬁed that a model-free baseline (also based on MPO, without HER) with access to state abstraction and low-level controller, only solved about 10% of the mazes after 100 million episodes due to the extremely sparse rewards.

Original: 5. C ONTINUOUS C ONTROL M AZES
Revised: 5. C ONTINUOUS C ONTROL M AZES

Original: We applied MCTS and DC-MCTS to this problem to ﬁnd symbolic plans consisting of sub-goals in { 1 , . .
Revised: We applied MCTS and DC-MCTS to this problem to ﬁnd symbolic plans consisting of sub-goals in { 1 , . .

Original: The high-level heuristics p and v were trained for 65k episodes, exactly as in Section 5.1, except using v π m instead of v π 0 .
Revised: The high-level heuristics p and v were trained for 65k episodes, exactly as in Section 5.1, except using v π m instead of v π 0 .

Original: We again observed that DC-MCTS outperforms by a wide margin the MCTS planner: Figure 5 shows performance of both (with fully trained search heuristics) as a function of the search budget for the most difﬁcult mazes with wall density d = 1 . 0 .
Revised: We again observed that DC-MCTS outperforms by a wide margin the MCTS planner: Figure 5 shows performance of both (with fully trained search heuristics) as a function of the search budget for the most difﬁcult mazes with wall density d = 1 . 0 .

Original: Performance of DC-MCTS with the MuJoCo low-level controller was comparable to that with the hard-coded low-level policy from the grid-world experiment (with same wall density), showing that the abstraction of planning over low-level sub-goals successfully isolates high-level planning from low-level execution.
Revised: Performance of DC-MCTS with the MuJoCo low-level controller was comparable to that with the hard-coded low-level policy from the grid-world experiment (with same wall density), showing that the abstraction of planning over low-level sub-goals successfully isolates high-level planning from low-level execution.

Original: We did not manage to successfully train the MCTS planner on MuJoCo navigation.
Revised: We did not manage to successfully train the MCTS planner on MuJoCo navigation.

Original: This was likely due to HER, which we found — in ablation studies — essential for training DC-MCTS on both settings and
Revised: This was likely due to HER, which we found — in ablation studies — essential for training DC-MCTS on both settings and

Original: MCTS on the grid-world problem, but not appropriate for MCTS on MuJoCo navigation: Left-ﬁrst parsing for HER consistently biased the MCTS search prior p to propose next sub-goals too close to the previous sub-goal.
Revised: MCTS on the grid-world problem, but not appropriate for MCTS on MuJoCo navigation: Left-ﬁrst parsing for HER consistently biased the MCTS search prior p to propose next sub-goals too close to the previous sub-goal.

Original: This lead the MCTS planner to “micro-manage” the low-level policy, in particular in long corridors that π m can solve by itself.
Revised: This lead the MCTS planner to “micro-manage” the low-level policy, in particular in long corridors that π m can solve by itself.

Original: DC-MCTS, by recursively partitioning, found an appropriate length scale of sub-goals, leading to drastically improved performance.
Revised: DC-MCTS, by recursively partitioning, found an appropriate length scale of sub-goals, leading to drastically improved performance.

Original: 5. V ISUALIZING MCTS AND DC-MCTS
Revised: 5. V ISUALIZING MCTS AND DC-MCTS

Original: To further illustrate the difference between DC-MCTS and MCTS planning we can look at an example search tree from each method in Figure 6.
Revised: To further illustrate the difference between DC-MCTS and MCTS planning we can look at an example search tree from each method in Figure 6.

Original: Light blue nodes are part of the ﬁnal plan: note how in the case of DC-MCTS, the plan is distributed across a sub-tree within the search tree, while for the 
Revised: Light blue nodes are part of the ﬁnal plan: note how in the case of DC-MCTS, the plan is distributed across a sub-tree within the search tree, while for the standard MCTS the plan is a chain . The ﬁrst ‘actionable’ sub-goal, i.e. the ﬁrst sub-goal for the low-level , is the left most leaf DC- and ﬁrst dark node the root

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: To enable guided, divide-and-conquer style planning, we made a few strong assumptions.
Revised: To enable guided, divide-and-conquer style planning, we made a few strong assumptions.

Original: Sub-goal based planning requires a universal value function oracle of the low-level policy, which often will have to be approximated from data.
Revised: Sub-goal based planning requires a universal value function oracle of the low-level policy, which often will have to be approximated from data.

Original: Overly optimistic approximations can be exploited by the planner, leading to “delusional” plans (Little & Thiébaux, 2007).
Revised: Overly optimistic approximations can be exploited by the planner, leading to “delusional” plans (Little & Thiébaux, 2007).

Original: Joint learning of the high and low-level components can potentially address this issue.
Revised: Joint learning of the high and low-level components can potentially address this issue.

Original: In sub-goal planning, at least in its current naive implementation, the “action space” for the planner is the whole state space of the underlying MDPs.
Revised: In sub-goal planning, at least in its current naive implementation, the “action space” for the planner is the whole state space of the underlying MDPs.

Original: Therefore, the search space will have a large branching factor in large state spaces.
Revised: Therefore, the search space will have a large branching factor in large state spaces.

Original: A solution to this problem likely lies in using learned state abstractions for sub-goal speciﬁcations, which is a fundamental open research questions.
Revised: A solution to this problem likely lies in using learned state abstractions for sub-goal speciﬁcations, which is a fundamental open research questions.

Original: We also implicitly assumed that low-level skills afforded by the low-level policy need to be “universal”, i.e. if there are states that it cannot reach, no amount of high level search will lead to successful planning outcomes.
Revised: We also implicitly assumed that low-level skills afforded by the low-level policy need to be “universal”, i.e. if there are states that it cannot reach, no amount of high level search will lead to successful planning outcomes.

Original: In spite of these assumptions and open challenges, we showed that non-sequential sub-goal planning has fundamental advantages over the standard approach of search over primitive actions: (i) Abstraction and dynamic allocation: Sub-goals automatically support temporal abstraction as the high-level planner does not need to specify the exact time horizon required to achieve a sub-goal.
Revised: In spite of these assumptions and open challenges, we showed that non-sequential sub-goal planning has fundamental advantages over the standard approach of search over primitive actions: (i) Abstraction and dynamic allocation: Sub-goals automatically support temporal abstraction as the high-level planner does not need to specify the exact time horizon required to achieve a sub-goal.

Original: Plans are generated from coarse to ﬁne, and additional planning is dynamically allocated to those parts of the plan that require more compute.
Revised: Plans are generated from coarse to ﬁne, and additional planning is dynamically allocated to those parts of the plan that require more compute.

Original: (ii) Closed & open-loop: The approach combines advantages of both open- and closed loop planning: The closed-loop low-level policies can recover from failures or unexpected transitions in stochastic environments, while at the same time the high-level planner can avoid costly closed-loop planning.
Revised: (ii) Closed & open-loop: The approach combines advantages of both open- and closed loop planning: The closed-loop low-level policies can recover from failures or unexpected transitions in stochastic environments, while at the same time the high-level planner can avoid costly closed-loop planning.

Original: (iii) Long horizon credit assignment: Sub-goal abstractions open up new algorithmic possibilities for planning — as exempliﬁed by DC-MCTS — that can facilitate credit assignment and therefore reduce planning complexity.
Revised: (iii) Long horizon credit assignment: Sub-goal abstractions open up new algorithmic possibilities for planning — as exempliﬁed by DC-MCTS — that can facilitate credit assignment and therefore reduce planning complexity.

Original: (iv) Parallelization: Like other divide-and-conquer algorithms,
Revised: (iv) Parallelization: Like other divide-and-conquer algorithms,

Original: DC-MCTS lends itself to parallel execution by leveraging problem decomposition made explicit by the independence of the "left" and "right" sub-problems of an AND node.
Revised: DC-MCTS lends itself to parallel execution by leveraging problem decomposition made explicit by the independence of the "left" and "right" sub-problems of an AND node.

Original: (v) Reuse of cached search: DC-MCTS is highly amenable to transposition tables, by caching and reusing values for sub-problems solved in other branches of the search tree.
Revised: (v) Reuse of cached search: DC-MCTS is highly amenable to transposition tables, by caching and reusing values for sub-problems solved in other branches of the search tree.

Original: (vi) Generality:
Revised: (vi) Generality:

Original: DC-MCTS is strictly more general than both forward and backward goal-directed planning, both of which can be seen as special cases.
Revised: DC-MCTS is strictly more general than both forward and backward goal-directed planning, both of which can be seen as special cases.

Original: A A DDITIONAL D ETAILS FOR DC-MCTS A. P ROOF OF P ROPOSITION
Revised: A A DDITIONAL D ETAILS FOR DC-MCTS A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: The performance of π σ on the task ( s 0 , s ∞ ) is deﬁned as the probability that its trajectory τ π σ s 0 from initial state s 0 gets absorbed in the state s ∞ , i.e. P ( s ∞ ∈ τ
Revised: The performance of π σ on the task ( s 0 , s ∞ ) is deﬁned as the probability that its trajectory τ π σ s 0 from initial state s 0 gets absorbed in the state s ∞ , i.e. P ( s ∞ ∈ τ

Original: π σ s 0 ) .
Revised: π σ s 0 ) .

Original: We can bound the latter from below in the following way.
Revised: We can bound the latter from below in the following way.

Original: Let σ = ( σ 0 , . . .
Revised: Let σ = ( σ 0 , . . .

Original: , σ m ) , with σ 0 = s 0 and σ m = s ∞ .
Revised: , σ m ) , with σ 0 = s 0 and σ m = s ∞ .

Original: With ( σ 0 , . . .
Revised: With ( σ 0 , . . .

Original: , σ i ) ⊆
Revised: , σ i ) ⊆

Original: τ π σ s 

Revised: τ π σ s

Original: we denote the event that π σ visits all states σ 0 , . . .
Revised: we denote the event that π σ visits all states σ 0 , . . .

Original: , σ i in order:
Revised: , σ i in order:

Original: [Equation]
Revised: [Equation]

Original: where t i is the arrival time of π σ at σ i , and we deﬁne t 0 = 0 .
Revised: where t i is the arrival time of π σ at σ i , and we deﬁne t 0 = 0 .

Original: Obviously, the event ( σ 0 , . . .
Revised: Obviously, the event ( σ 0 , . . .

Original: , σ m ) ⊆
Revised: , σ m ) ⊆

Original: τ π σ s 0 is a subset of the event s ∞ ∈ τ
Revised: τ π σ s 0 is a subset of the event s ∞ ∈ τ

Original: π σ s 0 , and therefore
Revised: π σ s 0 , and therefore

Original: [Equation]
Revised: [Equation]

Original: Using the chain rule of probability we can write the lhs as:
Revised: Using the chain rule of probability we can write the lhs as:

Original: [Equation]
Revised: [Equation]

Original: We now use the deﬁnition of π σ : After reaching σ i − 1 and before reaching σ i , π σ is deﬁned by just executing π σ i starting from the state σ i − 1 :
Revised: We now use the deﬁnition of π σ : After reaching σ i − 1 and before reaching σ i , π σ is deﬁned by just executing π σ i starting from the state σ i − 1 :

Original: [Equation]
Revised: [Equation]

Original: We now make use of the fact that the σ i ∈ S are states of the underlying MDP that make the future independent from the past: Having reached σ i − 1 at t i − 1 , all events from there on (e.g. reaching σ j for j ≥ i ) are independent from all event before t
Revised: We now make use of the fact that the σ i ∈ S are states of the underlying MDP that make the future independent from the past: Having reached σ i − 1 at t i − 1 , all events from there on (e.g. reaching σ j for j ≥ i ) are independent from all event before t

Original: i − 1 .
Revised: i − 1 .

Original: We can therefore write:
Revised: We can therefore write:

Original: [Equation]
Revised: [Equation]

Original: Putting together equation 3 and equation 4 yields the proposition.
Revised: Putting together equation 3 and equation 4 yields the proposition.

Original: A. A DDITIONAL ALGORITHMIC DETAILS
Revised: A. A DDITIONAL ALGORITHMIC DETAILS

Original: After the search phase, in which DC-MCTS builds the search tree T , it returns its estimate of the best plan ˆ σ ∗ and the corresponding lower bound L (ˆ σ ∗ ) by calling the E XTRACT P LAN procedure on the root node ( s 0 , s ∞ ) .
Revised: After the search phase, in which DC-MCTS builds the search tree T , it returns its estimate of the best plan ˆ σ ∗ and the corresponding lower bound L (ˆ σ ∗ ) by calling the E XTRACT P LAN procedure on the root node ( s 0 , s ∞ ) .

Original: Algorithm 2 gives details on this procedure.
Revised: Algorithm 2 gives details on this procedure.

Original: A. D ESCENDING INTO ONE NODE AT THE TIME DURING SEARCH
Revised: A. D ESCENDING INTO ONE NODE AT THE TIME DURING SEARCH

Original: Instead of descending into both nodes during the T RAVERSE step of Algorithm 1, it is possible to choose only one of the two sub-problems to expand further.
Revised: Instead of descending into both nodes during the T RAVERSE step of Algorithm 1, it is possible to choose only one of the two sub-problems to expand further.

Original: This can be especially useful if parallel
Revised: This can be especially useful if parallel

Original: [Figure]
Revised: [Figure]

Original: Global
Revised: Global

Original: [Table]
Revised: [Table]

Original: v π
Revised: v π

Original: Global high-level
Revised: Global high-level

Original: [Table]
Revised: [Table]

Original: function v
Revised: function v

Original: Global policy prior p
Revised: Global policy prior p

Original: Global search tree T 1: procedure E XTRACT P LAN (OR node ( s, s (cid:48)(cid:48) ) )
Revised: Global search tree T 1: procedure E XTRACT P LAN (OR node ( s, s (cid:48)(cid:48) ) )

Original: 2: s (cid:48)
Revised: 2: s (cid:48)

Original: ← arg max ˆ s V ( s, ˆ s ) ·
Revised: ← arg max ˆ s V ( s, ˆ s ) ·

Original: V (ˆ s, s (cid:48)(cid:48) )
Revised: V (ˆ s, s (cid:48)(cid:48) )

Original: (cid:46) choose best sub-goal 3: if s = ∅ then (cid:46) no more splitting 4: return ∅ , v π ( s, s (cid:48)(cid:48) ) 5: else 6: σ l , G l ← E XTRACT P LAN ( s, s (cid:48) )
Revised: (cid:46) choose best sub-goal 3: if s = ∅ then (cid:46) no more splitting 4: return ∅ , v π ( s, s (cid:48)(cid:48) ) 5: else 6: σ l , G l ← E XTRACT P LAN ( s, s (cid:48) )

Original: (cid:46) extract "left" sub-plan 7: σ r ,
Revised: (cid:46) extract "left" sub-plan 7: σ r ,

Original: G r ← E XTRACT P LAN ( s (cid:48) , s (cid:48)(cid:48) )
Revised: G r ← E XTRACT P LAN ( s (cid:48) , s (cid:48)(cid:48) )

Original: (cid:46) extract "right
Revised: (cid:46) extract "right

Original: -plan 8: return σ l ◦ σ r , G l · G r computation is not an option, or if there are speciﬁc needs e.g. as illustrated by the following three heuristics.
Revised: -plan 8: return σ l ◦ σ r , G l · G r computation is not an option, or if there are speciﬁc needs e.g. as illustrated by the following three heuristics.

Original: These can be used to decide when to traverse into the left sub-problem ( s, s (cid:48) ) or the right sub-problem ( s (cid:48) , s (cid:48)(cid:48) ) .
Revised: These can be used to decide when to traverse into the left sub-problem ( s, s (cid:48) ) or the right sub-problem ( s (cid:48) , s (cid:48)(cid:48) ) .

Original: Note that both nodes have a corresponding current estimate for their value V , coming either from the bootstrap evaluation of v or further reﬁned from previous traversals.
Revised: Note that both nodes have a corresponding current estimate for their value V , coming either from the bootstrap evaluation of v or further reﬁned from previous traversals.

Original: • Preferentially descend into the left node encourages a more accurate evaluation of the near future, which is more relevant to the current choices of the agent.
Revised: • Preferentially descend into the left node encourages a more accurate evaluation of the near future, which is more relevant to the current choices of the agent.

Original: This makes sense when the right node can be further examined later, or there is uncertainty about the future that makes it sub-optimal to design a detailed plan at the moment.
Revised: This makes sense when the right node can be further examined later, or there is uncertainty about the future that makes it sub-optimal to design a detailed plan at the moment.

Original: • Preferentially descend into the node with a lower value , following the principle that a chain (plan) is only as good as its weakest link (sub-problem).
Revised: • Preferentially descend into the node with a lower value , following the principle that a chain (plan) is only as good as its weakest link (sub-problem).

Original: This heuristic effectively greedily optimizes for the overall value of the plan.
Revised: This heuristic effectively greedily optimizes for the overall value of the plan.

Original: • Use 2-way UCT on the values of the nodes , which acts similarly to the previous greedy heuristic, but also takes into account the conﬁdence over the value estimates given by the visit counts.
Revised: • Use 2-way UCT on the values of the nodes , which acts similarly to the previous greedy heuristic, but also takes into account the conﬁdence over the value estimates given by the visit counts.

Original: The rest of the algorithm can remain unchanged, and during the B ACKUP phase the current value estimate V of the sibling sub-problem can be used.
Revised: The rest of the algorithm can remain unchanged, and during the B ACKUP phase the current value estimate V of the sibling sub-problem can be used.

Original: A. P ARSERS FOR H INDSIGHT E XPERIENCE R EPLAY
Revised: A. P ARSERS FOR H INDSIGHT E XPERIENCE R EPLAY

Original: Given a task ( s 0 , s ∞ ) , the policy prior p deﬁnes a distribution over binary partition trees of the task via recursive application (until the terminal symbol ∅ closes a branch).
Revised: Given a task ( s 0 , s ∞ ) , the policy prior p deﬁnes a distribution over binary partition trees of the task via recursive application (until the terminal symbol ∅ closes a branch).

Original: A sample T σ from this distribution implies a plan σ as described above; but furthermore it also contains the order in which the task was partitioned.
Revised: A sample T σ from this distribution implies a plan σ as described above; but furthermore it also contains the order in which the task was partitioned.

Original: Therefore, p not only implies a distribution over plans, but also a search order : Trees with high probability under p will be discovered earlier in the search with
Revised: Therefore, p not only implies a distribution over plans, but also a search order : Trees with high probability under p will be discovered earlier in the search with

Original: DC-MCTS.
Revised: DC-MCTS.

Original: For generating training targets for supervised training of p , we need to parse a given sequence τ π σ s 0 =
Revised: For generating training targets for supervised training of p , we need to parse a given sequence τ π σ s 0 =

Original: ( s 0 , s 1 , . .
Revised: ( s 0 , s 1 , . .

Original: , s T ) into a binary tree.
Revised: , s T ) into a binary tree.

Original: Therefore, when applying HER we are free to choose any deterministic or probabilistic parser that generates a solution tree T τ πσs 0 from re-labeled
Revised: Therefore, when applying HER we are free to choose any deterministic or probabilistic parser that generates a solution tree T τ πσs 0 from re-labeled

Original: HER data τ π σ s 0 .
Revised: HER data τ π σ s 0 .

Original: As mentioned in the main text, the particular choice of HER-parser will shape the search strategy deﬁned by p .
Revised: As mentioned in the main text, the particular choice of HER-parser will shape the search strategy deﬁned by p .

Original: Possible choices for the parsers include:
Revised: Possible choices for the parsers include:

Original: Left-ﬁrst parsing creates triplets ( s t , s t +1 , s T ) .
Revised: Left-ﬁrst parsing creates triplets ( s t , s t +1 , s T ) .

Original: The resulting policy prior will then preferentially propose sub-goals close to the start state, mimicking standard forward planning.
Revised: The resulting policy prior will then preferentially propose sub-goals close to the start state, mimicking standard forward planning.

Original: Analogously right-ﬁrst parsing results in approximate backward planning; 2.
Revised: Analogously right-ﬁrst parsing results in approximate backward planning; 2.

Original: Temporally balanced parsing creates triplets ( s t , s t +∆ / 2 , s t +∆ ) .
Revised: Temporally balanced parsing creates triplets ( s t , s t +∆ / 2 , s t +∆ ) .

Original: The resulting policy prior will then preferentially propose sub-goals “in the middle” of the task.
Revised: The resulting policy prior will then preferentially propose sub-goals “in the middle” of the task.

Original: This is the one we used in our experiments; 3.
Revised: This is the one we used in our experiments; 3.

Original: Weight-balanced parsing creates triplets ( s, s (cid:48)
Revised: Weight-balanced parsing creates triplets ( s, s (cid:48)

Original: , s (cid:48)(cid:48) )
Revised: , s (cid:48)(cid:48) )

Original: such that v ( s, s (cid:48) )
Revised: such that v ( s, s (cid:48) )

Original: ≈ v ( s (cid:48) s, (cid:48)(cid:48) ) or v π ( s, s (cid:48) )
Revised: ≈ v ( s (cid:48) s, (cid:48)(cid:48) ) or v π ( s, s (cid:48) )

Original: ≈ v π ( s (cid:48) s, (cid:48)(cid:48) ) .
Revised: ≈ v π ( s (cid:48) s, (cid:48)(cid:48) ) .

Original: The resulting policy prior will attempt to propose sub-goals such that the resulting sub-tasks are equally difﬁcult.
Revised: The resulting policy prior will attempt to propose sub-goals such that the resulting sub-tasks are equally difﬁcult.

Original: A. D ETAILS ON A LGORITHMIC C OMPLEXITY
Revised: A. D ETAILS ON A LGORITHMIC C OMPLEXITY

Original: Let c v π denote the cost of evaluating the low-level value v π on any sub-problem ( s, s (cid:48) ) .
Revised: Let c v π denote the cost of evaluating the low-level value v π on any sub-problem ( s, s (cid:48) ) .

Original: We assume c v π to be independent of ( s, s (cid:48) ) which holds if e.g. v π is a ﬁxed size neural network.
Revised: We assume c v π to be independent of ( s, s (cid:48) ) which holds if e.g. v π is a ﬁxed size neural network.

Original: Denote the cost of evaluating the policy prior p on a sub-goal ( s, s (cid:48) , s (cid:48)(cid:48) ) with c p .
Revised: Denote the cost of evaluating the policy prior p on a sub-goal ( s, s (cid:48) , s (cid:48)(cid:48) ) with c p .

Original: Expanding a new OR node in the search tree incurs a cost of | S | c p for evaluating p for all children.
Revised: Expanding a new OR node in the search tree incurs a cost of | S | c p for evaluating p for all children.

Original: Assuming the computational cost of tree traversals is negligible, the total cost of running DC-MCTS for N node expansions is thus N · ( |S| c p + 2 c v π ) .
Revised: Assuming the computational cost of tree traversals is negligible, the total cost of running DC-MCTS for N node expansions is thus N · ( |S| c p + 2 c v π ) .

Original: The number of expansions to ﬁnd the optimal (or a sufﬁciently good) plan strongly depends on the quality of the policy prior (similar to A ∗ search), making an analysis of the complexity of DC-MCTS challenging for arbitrary p .
Revised: The number of expansions to ﬁnd the optimal (or a sufﬁciently good) plan strongly depends on the quality of the policy prior (similar to A ∗ search), making an analysis of the complexity of DC-MCTS challenging for arbitrary p .

Original: However, if p = p ∗ is the optimal policy prior – i.e. p ∗ ( s (cid:48) | s, s (cid:48)(cid:48) )
Revised: However, if p = p ∗ is the optimal policy prior – i.e. p ∗ ( s (cid:48) | s, s (cid:48)(cid:48) )

Original: = 

Revised: =

Original: if s (cid:48) ∈ σ ∗ is in the optimal plan σ ∗ for ( s, s (cid:48)(cid:48) ) and 0 otherwise –
Revised: if s (cid:48) ∈ σ ∗ is in the optimal plan σ ∗ for ( s, s (cid:48)(cid:48) ) and 0 otherwise –

Original: DC-MCTS will construct σ ∗ in the minimal number of step N = | σ ∗ | , therefore incurring a cost of | σ ∗ |· ( |S| c p + 2 c v π ) .
Revised: DC-MCTS will construct σ ∗ in the minimal number of step N = | σ ∗ | , therefore incurring a cost of | σ ∗ |· ( |S| c p + 2 c v π ) .

Original: The dependency on |S| for the policy prior can be further reduced — in principle down to a constant — using techniques from the literature on MCTS for continuous or large discrete action spaces (e.g. progressive widening Coulom (2007); Chaslot et al.).
Revised: The dependency on |S| for the policy prior can be further reduced — in principle down to a constant — using techniques from the literature on MCTS for continuous or large discrete action spaces (e.g. progressive widening Coulom (2007); Chaslot et al.).

Original: We can compare this to the cost of unguided, standard SSSP planners, which is |S| 2 c v π as they need to query the low-level value function for all pairs of states ( s, s (cid:48) ) .
Revised: We can compare this to the cost of unguided, standard SSSP planners, which is |S| 2 c v π as they need to query the low-level value function for all pairs of states ( s, s (cid:48) ) .

Original: Therefore, DC-MCTS can be signiﬁcantly more cost efﬁcient than conventional SSSP planners if a good policy prior can be learned and the cost of evaluating c p (cid:47) c v π is at least comparable to that of evaluating v π .
Revised: Therefore, DC-MCTS can be signiﬁcantly more cost efﬁcient than conventional SSSP planners if a good policy prior can be learned and the cost of evaluating c p (cid:47) c v π is at least comparable to that of evaluating v π .

Original: Under the assumption p = p ∗ ,
Revised: Under the assumption p = p ∗ ,

Original: MCTS and DC-MCTS have the same sample complexity.
Revised: MCTS and DC-MCTS have the same sample complexity.

Original: However MCTS represents the solution as one path of length | σ ∗ | in the search tree, whereas DC-MCTS presents it as a sub-tree with | σ ∗ | nodes.
Revised: However MCTS represents the solution as one path of length | σ ∗ | in the search tree, whereas DC-MCTS presents it as a sub-tree with | σ ∗ | nodes.

Original: Therefore, if computation can be carried out in parallel (e.g. by batching independent sub-problems at the same level of the sub-tree), the time complexity of DC-MCTS could be drastically reduced compared to MCTS, in the best case (perfect parallelism and balanced binary solution tree) from
Revised: Therefore, if computation can be carried out in parallel (e.g. by batching independent sub-problems at the same level of the sub-tree), the time complexity of DC-MCTS could be drastically reduced compared to MCTS, in the best case (perfect parallelism and balanced binary solution tree) from

Original: O ( | σ ∗ | ) to O (log( | σ ∗ | )) .
Revised: O ( | σ ∗ | ) to O (log( | σ ∗ | )) .

Original: B T RAINING DETAILS B. D ETAILS FOR TRAINING THE VALUE FUNCTION
Revised: B T RAINING DETAILS B. D ETAILS FOR TRAINING THE VALUE FUNCTION

Original: In order to train the value network v , that is used for bootstrapping in DC-MCTS, we can regress it towards targets computed from previous search results or environment experiences.
Revised: In order to train the value network v , that is used for bootstrapping in DC-MCTS, we can regress it towards targets computed from previous search results or environment experiences.

Original: A ﬁrst obvious option is to use as regression target the Monte Carlo return (i.e. 0 if the goal was reached, and 1 if it was not) from executing the DC-MCTS plans in the environment.
Revised: A ﬁrst obvious option is to use as regression target the Monte Carlo return (i.e. 0 if the goal was reached, and 1 if it was not) from executing the DC-MCTS plans in the environment.

Original: This appears to be a sensible target, as the return is an unbiased estimator of the success probability P ( s ∞ ∈ τ π σ s 0 ) of the plan.
Revised: This appears to be a sensible target, as the return is an unbiased estimator of the success probability P ( s ∞ ∈ τ π σ s 0 ) of the plan.

Original: Although this approach was used in Silver et al.
Revised: Although this approach was used in Silver et al.

Original: , its downside is that gathering environment experience is often very costly and only yields little information, i.e. one binary variable per episode.
Revised: , its downside is that gathering environment experience is often very costly and only yields little information, i.e. one binary variable per episode.

Original: Furthermore no other information from the generated search tree T except for the best plan is used.
Revised: Furthermore no other information from the generated search tree T except for the best plan is used.

Original: Therefore, a lot of valuable information might be discarded, in particular in situations where a good sub-plan for a particular sub-problem was found, but the overall plan nevertheless failed.
Revised: Therefore, a lot of valuable information might be discarded, in particular in situations where a good sub-plan for a particular sub-problem was found, but the overall plan nevertheless failed.

Original: This shortcoming could be remedied by using as regression targets the non-parametric value estimates
Revised: This shortcoming could be remedied by using as regression targets the non-parametric value estimates

Original: V ( s, s (cid:48)(cid:48) ) for all OR nodes ( s, s (cid:48)(cid:48) ) in the DC-MCTS tree at the end of the search.
Revised: V ( s, s (cid:48)(cid:48) ) for all OR nodes ( s, s (cid:48)(cid:48) ) in the DC-MCTS tree at the end of the search.

Original: With this approach, a learning signal could still be obtained from successful sub-plans of an overall failed plan.
Revised: With this approach, a learning signal could still be obtained from successful sub-plans of an overall failed plan.

Original: However, we empirically found in our experiments that this lead to drastically over-optimistic value estimates, for the following reason.
Revised: However, we empirically found in our experiments that this lead to drastically over-optimistic value estimates, for the following reason.

Original: By standard policy improvement arguments, regressing toward V leads to a bootstrap value function that converges to v ∗ .
Revised: By standard policy improvement arguments, regressing toward V leads to a bootstrap value function that converges to v ∗ .

Original: In the deﬁnition of the optimal value v ∗ ( s, s (cid:48)(cid:48) )
Revised: In the deﬁnition of the optimal value v ∗ ( s, s (cid:48)(cid:48) )

Original: =
Revised: =

Original: max s (cid:48) v ∗ ( s, s (cid:48) ) · v ∗ ( s (cid:48) , s (cid:48)(cid:48) ) , we implicitly allow for inﬁnite recursion depth for solving sub-problems.
Revised: max s (cid:48) v ∗ ( s, s (cid:48) ) · v ∗ ( s (cid:48) , s (cid:48)(cid:48) ) , we implicitly allow for inﬁnite recursion depth for solving sub-problems.

Original: However, in practice, we often used quite shallow trees (depth < 10), so that bootstrapping with approximations of v ∗ is too optimistic, as this assumes unbounded planning budget.
Revised: However, in practice, we often used quite shallow trees (depth < 10), so that bootstrapping with approximations of v ∗ is too optimistic, as this assumes unbounded planning budget.

Original: A principled solution for this could be to condition the value function for bootstrapping on the amount of remaining search budget, either in terms of remaining tree depth or node expansions.
Revised: A principled solution for this could be to condition the value function for bootstrapping on the amount of remaining search budget, either in terms of remaining tree depth or node expansions.

Original: Instead of the cumbersome, explicitly resource-aware value function, we found the following to work well.
Revised: Instead of the cumbersome, explicitly resource-aware value function, we found the following to work well.

Original: After planning with DC-MCTS, we extract the plan ˆ σ ∗ with E XTRACT P LAN from the search tree T .
Revised: After planning with DC-MCTS, we extract the plan ˆ σ ∗ with E XTRACT P LAN from the search tree T .

Original: As can be seen from Algorithm 2, the procedure computes the return G ˆ σ ∗ for all OR nodes in the solution tree T ˆ σ ∗ .
Revised: As can be seen from Algorithm 2, the procedure computes the return G ˆ σ ∗ for all OR nodes in the solution tree T ˆ σ ∗ .

Original: For training v we chose these returns G ˆ σ ∗ for all OR nodes in the solution tree as regression targets.
Revised: For training v we chose these returns G ˆ σ ∗ for all OR nodes in the solution tree as regression targets.

Original: This combines the favourable aspects of both methods described above.
Revised: This combines the favourable aspects of both methods described above.

Original: In particular, this value estimate contains no bootstrapping and therefore did not lead to
Revised: In particular, this value estimate contains no bootstrapping and therefore did not lead to

Original: [Table]
Revised: [Table]

Original: overly-optimistic bootstraps.
Revised: overly-optimistic bootstraps.

Original: Furthermore, all successfully solved sub-problems given a learning signal.
Revised: Furthermore, all successfully solved sub-problems given a learning signal.

Original: As regression loss we chose cross-entropy
Revised: As regression loss we chose cross-entropy

Original: [Table]
Revised: [Table]

Original: B.
Revised: B.

Original: [Table]
Revised: [Table]

Original: D ETAILS FOR TRAINING THE POLICY PRIOR
Revised: D ETAILS FOR TRAINING THE POLICY PRIOR

Original: The prior network is trained to match the distribution of the values of the AND nodes, also with a cross-entropy loss.
Revised: The prior network is trained to match the distribution of the values of the AND nodes, also with a cross-entropy loss.

Original: Note that we did not use visit counts as targets for the prior network — as done in
Revised: Note that we did not use visit counts as targets for the prior network — as done in

Original: AlphaGo and AlphaZero for example (Silver et al., 2016; 2018)— since for small search budgets visit counts tend to be noisy and require signiﬁcant ﬁne-tuning to avoid collapse (Hamrick et al., 2020).
Revised: AlphaGo and AlphaZero for example (Silver et al., 2016; 2018)— since for small search budgets visit counts tend to be noisy and require signiﬁcant ﬁne-tuning to avoid collapse (Hamrick et al., 2020).

Original: B. N EURAL NETWORKS ARCHITECTURES FOR GRID - WORLD EXPERIMENTS
Revised: B. N EURAL NETWORKS ARCHITECTURES FOR GRID - WORLD EXPERIMENTS

Original: The shared torso of the prior and value network used in the experiments is a 6-layer CNN with kernels of size 3, 64 ﬁlters per layer, Layer Normalization after every convolutional layer, swish (cit) as activation function, zero-padding of 1, and strides [1, 1, 2, 1, 1, 2] to increase the size of the receptive ﬁeld.
Revised: The shared torso of the prior and value network used in the experiments is a 6-layer CNN with kernels of size 3, 64 ﬁlters per layer, Layer Normalization after every convolutional layer, swish (cit) as activation function, zero-padding of 1, and strides [1, 1, 2, 1, 1, 2] to increase the size of the receptive ﬁeld.

Original: The two heads for the prior and value networks follow the pattern described above, but with three layers only instead of six, and ﬁxed strides of 1.
Revised: The two heads for the prior and value networks follow the pattern described above, but with three layers only instead of six, and ﬁxed strides of 1.

Original: The prior head ends with a linear layer and a softmax, in order to obtain a distribution over sub-goals.
Revised: The prior head ends with a linear layer and a softmax, in order to obtain a distribution over sub-goals.

Original: The value head ends with a linear layer and a sigmoid that predicts a single value, i.e. the probability of reaching the goal from the start state if we further split the problem into sub-problems.
Revised: The value head ends with a linear layer and a sigmoid that predicts a single value, i.e. the probability of reaching the goal from the start state if we further split the problem into sub-problems.

Original: We did not heavily optimize networks hyper-parameters.
Revised: We did not heavily optimize networks hyper-parameters.

Original: After running a random search over hyper-parameters for the ﬁxed architecture described above, the following were chosen to run the experiments in Figure 3.
Revised: After running a random search over hyper-parameters for the ﬁxed architecture described above, the following were chosen to run the experiments in Figure 3.

Original: The replay buffer has a maximum size of 2048.
Revised: The replay buffer has a maximum size of 2048.

Original: The prior and value networks are trained on batches of size 128 as new experiences are collected.
Revised: The prior and value networks are trained on batches of size 128 as new experiences are collected.

Original: Networks are trained using Adam with a learning rate of 1e-3, the boltzmann temperature of the softmax for the prior network set to 0.003.
Revised: Networks are trained using Adam with a learning rate of 1e-3, the boltzmann temperature of the softmax for the prior network set to 0.003.

Original: For simplicity, we used HER with the time-based rebalancing (i.e. turning experiences into temporal binary search trees).
Revised: For simplicity, we used HER with the time-based rebalancing (i.e. turning experiences into temporal binary search trees).

Original: UCB constants are sampled uniformly between 3 and 7, as these values were observed to give more robust results.
Revised: UCB constants are sampled uniformly between 3 and 7, as these values were observed to give more robust results.

Original: B. L OW - LEVEL CONTROLLER TRAINING DETAILS
Revised: B. L OW - LEVEL CONTROLLER TRAINING DETAILS

Original: For physics-based experiments using MuJoCo (Todorov et al., 2012), we trained a low-level policy ﬁrst and then trained the planning agent to reuse the low-level motor skills afforded by this body and pretrained policy.
Revised: For physics-based experiments using MuJoCo (Todorov et al., 2012), we trained a low-level policy ﬁrst and then trained the planning agent to reuse the low-level motor skills afforded by this body and pretrained policy.

Original: The low-level policy, was trained to control the quadruped (“ant”) body to go to a randomly placed target in an open area (a “go-to-target” task, essentially the same as the task used to train the humanoid in Merel et al., 2019, which is available at dm_control/locomotion).
Revised: The low-level policy, was trained to control the quadruped (“ant”) body to go to a randomly placed target in an open area (a “go-to-target” task, essentially the same as the task used to train the humanoid in Merel et al., 2019, which is available at dm_control/locomotion).

Original: The task amounts to the environment providing an instruction corresponding to a target position that the agent is is rewarded for moving to (i.e, a sparse reward when within a region of the target).
Revised: The task amounts to the environment providing an instruction corresponding to a target position that the agent is is rewarded for moving to (i.e, a sparse reward when within a region of the target).

Original: When the target is obtained, a new target is generated that is a short distance away (<1.5m).
Revised: When the target is obtained, a new target is generated that is a short distance away (<1.5m).

Original: What this means is that a policy trained on this task should be capable of producing short, direct, goal-directed locomotion behaviors in an open ﬁeld.
Revised: What this means is that a policy trained on this task should be capable of producing short, direct, goal-directed locomotion behaviors in an open ﬁeld.

Original: And at test time, the presence of obstacles will catastrophically confuse the trained low-level policy.
Revised: And at test time, the presence of obstacles will catastrophically confuse the trained low-level policy.

Original: The policy architecture, consisting of a shallow MLP for the actor and critic, was trained to solve this task using MPO Abdolmaleki et al.
Revised: The policy architecture, consisting of a shallow MLP for the actor and critic, was trained to solve this task using MPO Abdolmaleki et al.

Original: More speciﬁcally, the actor and critic had respectively 2 and 3 hidden layers, 256 units each and elu activation function.
Revised: More speciﬁcally, the actor and critic had respectively 2 and 3 hidden layers, 256 units each and elu activation function.

Original: The policy was trained to a high level of performance using a distributed, replay-based, off-policy training setup involving 64 actors.
Revised: The policy was trained to a high level of performance using a distributed, replay-based, off-policy training setup involving 64 actors.

Original: In order to reuse the low-level policy in the context of mazes, we can replace the environment-provided instruction with a message sent by a high-level policy (i.e., the planning agent).
Revised: In order to reuse the low-level policy in the context of mazes, we can replace the environment-provided instruction with a message sent by a high-level policy (i.e., the planning agent).

Original: For the planning agent that interfaces with the low-level policy, the action space of the high-level policy will, by construction, correspond to the instruction to the low-level policy.
Revised: For the planning agent that interfaces with the low-level policy, the action space of the high-level policy will, by construction, correspond to the instruction to the low-level policy.

Original: B. P SEUDOCODE
Revised: B. P SEUDOCODE

Original: We summarize the training procedure for DC-MCTS in the following pseudo-code.
Revised: We summarize the training procedure for DC-MCTS in the following pseudo-code.

Original: def train_DCMCTS():
Revised: def train_DCMCTS():

Original: [Equation]
Revised: [Equation]

Original: for episode in n episodes:
Revised: for episode in n episodes:

Original: C M ORE SOLVED MAZES
Revised: C M ORE SOLVED MAZES

Original: In Figure 8 we show more mazes as solved by the trained Divide and Conquer MCTS.
Revised: In Figure 8 we show more mazes as solved by the trained Divide and Conquer MCTS.

Original: C More solved mazes
Revised: C More solved mazes

Original: In Figure 8 we show more mazes as solved by the trained Divide and Conquer MCTS.
Revised: In Figure 8 we show more mazes as solved by the trained Divide and Conquer MCTS.

Original: C. Supplementary material and videos
Revised: C. Supplementary material and videos

Original: Additional material, including videos of several grid-world mazes as solved by the algorithm andof MuJoCo low-level policy solving mazes by following DC-MCTS plans, can be found at https://sites.google.com/view/dc-mcts/home .
Revised: Additional material, including videos of several grid-world mazes as solved by the algorithm andof MuJoCo low-level policy solving mazes by following DC-MCTS plans, can be found at https://sites.google.com/view/dc-mcts/home .

Original: = start, = goal, = wall, = walkable.
Revised: = start, = goal, = wall, = walkable.

Original: Overlapping numbers are due to the agent back-tracking while reﬁning ﬁner sub-goals.
Revised: Overlapping numbers are due to the agent back-tracking while reﬁning ﬁner sub-goals.

Original: = start, = goal, = wall, = walkable.
Revised: = start, = goal, = wall, = walkable.

Original: Overlapping numbers are due to the agent back-tracking while reﬁning ﬁner sub-goals.
Revised: Overlapping numbers are due to the agent back-tracking while reﬁning ﬁner sub-goals.

Original: C. S UPPLEMENTARY MATERIAL AND VIDEOS
Revised: C. S UPPLEMENTARY MATERIAL AND VIDEOS

Original: Additional material, including videos of several grid-world mazes as solved by the algorithm and of MuJoCo low-level policy solving mazes by following DC-MCTS plans, can be found in the supplementary material.
Revised: Additional material, including videos of several grid-world mazes as solved by the algorithm and of MuJoCo low-level policy solving mazes by following DC-MCTS plans, can be found in the supplementary material.