Original: C-P LANNING : A N A UTOMATIC C URRICULUM FOR L EARNING G OAL -R EACHING T ASKS
Revised: C-P LANNING : A N A UTOMATIC C URRICULUM FOR L EARNING G OAL -R EACHING T ASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the ﬁeld.
Revised: Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the ﬁeld.

Original: Learning to reach such goals is particularly hard without any ofﬂine data, expert demonstrations, and reward shaping.
Revised: Learning to reach such goals is particularly hard without any ofﬂine data, expert demonstrations, and reward shaping.

Original: In this paper, we propose an algorithm to solve the distant goal-reaching task by using planning at training time to automatically generate a curriculum of intermediate states.
Revised: In this paper, we propose an algorithm to solve the distant goal-reaching task by using planning at training time to automatically generate a curriculum of intermediate states.

Original: Our algorithm, Classiﬁer-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning a sequence of waypoints using graph planning, while the
Revised: Our algorithm, Classiﬁer-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning a sequence of waypoints using graph planning, while the

Original: M-step aims to learn a goal-conditioned policy to reach those waypoints.
Revised: M-step aims to learn a goal-conditioned policy to reach those waypoints.

Original: Unlike prior methods that combine goal-conditioned RL with graph search, ours performs planning only during training and not testing, signiﬁcantly decreasing the compute costs of deploying the learned policy.
Revised: Unlike prior methods that combine goal-conditioned RL with graph search, ours performs planning only during training and not testing, signiﬁcantly decreasing the compute costs of deploying the learned policy.

Original: Empirically, we demonstrate that our method is more sample efﬁcient that prior methods.
Revised: Empirically, we demonstrate that our method is more sample efﬁcient that prior methods.

Original: Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goalconditioned methods and methods based on graph search fail to solve.
Revised: Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goalconditioned methods and methods based on graph search fail to solve.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Whereas typical RL methods maximize the accumulated reward, goal-conditioned RL methods learn to reach any goal.
Revised: Whereas typical RL methods maximize the accumulated reward, goal-conditioned RL methods learn to reach any goal.

Original: Arguably, many tasks are more easily deﬁned as goal-reaching problems than as reward maximizing problems.
Revised: Arguably, many tasks are more easily deﬁned as goal-reaching problems than as reward maximizing problems.

Original: While many goal-conditioned RL algorithms learn policies that can reach nearby goals, learning to reach distant goals remains a challenging problem.
Revised: While many goal-conditioned RL algorithms learn policies that can reach nearby goals, learning to reach distant goals remains a challenging problem.

Original: Some prior methods approach this problem by performing search or optimization over subgoals at test time.
Revised: Some prior methods approach this problem by performing search or optimization over subgoals at test time.

Original: However, these test-time planning methods either rely on graph search [10, 37], which scales poorly with dimensionality [18], or continuous optimization over subgoals [32], which is expensive and can result in model exploitation.
Revised: However, these test-time planning methods either rely on graph search [10, 37], which scales poorly with dimensionality [18], or continuous optimization over subgoals [32], which is expensive and can result in model exploitation.

Original: In this paper, we take a different tack and instead use planning at training time to automatically generate a curriculum.
Revised: In this paper, we take a different tack and instead use planning at training time to automatically generate a curriculum.

Original: When training the agent to reach one goal, our method ﬁrst determines some intermediate waypoints enroute to that goal.
Revised: When training the agent to reach one goal, our method ﬁrst determines some intermediate waypoints enroute to that goal.

Original: Then, it commands the agent to reach those waypoints before navigating to the ﬁnal destination.
Revised: Then, it commands the agent to reach those waypoints before navigating to the ﬁnal destination.

Original: Collecting data in this manner improves the quality of data, allowing the agent to learn to reach distant goals.
Revised: Collecting data in this manner improves the quality of data, allowing the agent to learn to reach distant goals.

Original: Importantly, our method does not perform planning at test time, decreasing the computational demands for deploying the learned agent.
Revised: Importantly, our method does not perform planning at test time, decreasing the computational demands for deploying the learned agent.

Original: Our curriculum does not require manual engineering or prior knowledge of the tasks.
Revised: Our curriculum does not require manual engineering or prior knowledge of the tasks.

Original: Rather, the curriculum emerges automatically when we use expectation maximization (EM) to maximize a lower bound on the probability of reaching the goal.
Revised: Rather, the curriculum emerges automatically when we use expectation maximization (EM) to maximize a lower bound on the probability of reaching the goal.

Original: The M-step corresponds to a prior method for goalconditioned RL, while the E-step corresponds to graph planning.
Revised: The M-step corresponds to a prior method for goalconditioned RL, while the E-step corresponds to graph planning.

Original: Thus, EM presents a marriage between previous goal-conditioned RL algorithms and graph-planning methods.
Revised: Thus, EM presents a marriage between previous goal-conditioned RL algorithms and graph-planning methods.

Original: The main contribution of this work is a goal-conditioned RL algorithm, C-Planning, that excels at reaching distant goals.
Revised: The main contribution of this work is a goal-conditioned RL algorithm, C-Planning, that excels at reaching distant goals.

Original: Our method uses an automatic curriculum of subgoals to accelerate training.
Revised: Our method uses an automatic curriculum of subgoals to accelerate training.

Original: The automatic waypoint sampling ends up generating a curriculum of waypoints, points that are reachable by the current policy but also lead to the goal.
Revised: The automatic waypoint sampling ends up generating a curriculum of waypoints, points that are reachable by the current policy but also lead to the goal.

Original: We show that our curriculum emerges from applying variational inference to the goal-reaching problem.
Revised: We show that our curriculum emerges from applying variational inference to the goal-reaching problem.

Original: Unlike prior work, our method does not require graph search or optimization at test-time.
Revised: Unlike prior work, our method does not require graph search or optimization at test-time.

Original: Empirically, C-Planning not only matches but surpasses the performance of these prior search-based methods, suggesting that it is not just amortizing the cost of graph search.
Revised: Empirically, C-Planning not only matches but surpasses the performance of these prior search-based methods, suggesting that it is not just amortizing the cost of graph search.

Original: We empirically evaluate C-Planning on temporally-extended 2D navigation tasks and complex 18D robotic manipulation tasks.
Revised: We empirically evaluate C-Planning on temporally-extended 2D navigation tasks and complex 18D robotic manipulation tasks.

Original: C-Planning improves the sample efﬁciency of prior goal-conditioned RL algorithms and manages to solve more difﬁcult manipulations tasks such as rearranging multiple objects in sequence (see Fig. 1.).
Revised: C-Planning improves the sample efﬁciency of prior goal-conditioned RL algorithms and manages to solve more difﬁcult manipulations tasks such as rearranging multiple objects in sequence (see Fig. 1.).

Original: To the best of our knowledge, no prior method has learned tasks of such difﬁculty without requiring additional assumptions.
Revised: To the best of our knowledge, no prior method has learned tasks of such difﬁculty without requiring additional assumptions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The problem of learning to achieve goals has a long history, both in the control community Lyapunov [27] and the RL community [20].
Revised: The problem of learning to achieve goals has a long history, both in the control community Lyapunov [27] and the RL community [20].

Original: Many prior papers approach goal-conditioned RL as a rewarddriven, multi-task learning problem, assuming access to a goal-conditioned reward function.
Revised: Many prior papers approach goal-conditioned RL as a rewarddriven, multi-task learning problem, assuming access to a goal-conditioned reward function.

Original: One unique feature of goal-conditioned RL, compared to other multi-task RL settings, is that it can also be approached using reward-free methods, such as goal-conditioned behavior cloning [8, 16, 28, 37] and RL methods that employ hindsight relabeling [11, 20, 26, 38].
Revised: One unique feature of goal-conditioned RL, compared to other multi-task RL settings, is that it can also be approached using reward-free methods, such as goal-conditioned behavior cloning [8, 16, 28, 37] and RL methods that employ hindsight relabeling [11, 20, 26, 38].

Original: While goal-conditioned behavior cloning methods are simple to implement and have shown excellent results on a number of realworld settings [29, 39], they are not guaranteed to recover the optimal policy without additional assumptions (e.g., determinism, online data collection).
Revised: While goal-conditioned behavior cloning methods are simple to implement and have shown excellent results on a number of realworld settings [29, 39], they are not guaranteed to recover the optimal policy without additional assumptions (e.g., determinism, online data collection).

Original: While both methods excel at certain control tasks, both often struggle to solve tasks with longer horizons.
Revised: While both methods excel at certain control tasks, both often struggle to solve tasks with longer horizons.

Original: To solve longer-horizon tasks, prior work has combined goal-conditioned RL with graph search, noting that goal-conditioned value functions can be interpreted as dynamical distances [9, 19, 32, 37].
Revised: To solve longer-horizon tasks, prior work has combined goal-conditioned RL with graph search, noting that goal-conditioned value functions can be interpreted as dynamical distances [9, 19, 32, 37].

Original: These prior methods typically proceed in two phases: the ﬁrst phase learns a goal-conditioned policy, and the second phase combines that policy with graph search.
Revised: These prior methods typically proceed in two phases: the ﬁrst phase learns a goal-conditioned policy, and the second phase combines that policy with graph search.

Original: While these methods have demonstrated excellent results on a number of challenging control tasks, including real-world robotic navigation [29, 39], the stage-wise approach has a few limitations.
Revised: While these methods have demonstrated excellent results on a number of challenging control tasks, including real-world robotic navigation [29, 39], the stage-wise approach has a few limitations.

Original: First, the post-hoc use of graph search means that graph search cannot improve the underlying goal-conditioned policy.
Revised: First, the post-hoc use of graph search means that graph search cannot improve the underlying goal-conditioned policy.

Original: It is well known that the performance of RL algorithms is highly dependent on the quality of the collected data [21, 22, 24].
Revised: It is well known that the performance of RL algorithms is highly dependent on the quality of the collected data [21, 22, 24].

Original: By integrating planning into the learning of the underlying goal-conditioned policy, our method will improve the quality of the data used to train that policy.
Revised: By integrating planning into the learning of the underlying goal-conditioned policy, our method will improve the quality of the data used to train that policy.

Original: A second limitation of these prior methods is the cost of deployment: choosing actions using graph search requires at least O ( |V| ) queries to a neural network.
Revised: A second limitation of these prior methods is the cost of deployment: choosing actions using graph search requires at least O ( |V| ) queries to a neural network.

Original: In contrast, our method performs planning at training time rather than test time, so the cost of deployment is O (1) .
Revised: In contrast, our method performs planning at training time rather than test time, so the cost of deployment is O (1) .

Original: While this design decision does increase the computational complexity of training, it signiﬁcantly decreases the latency at deployment.
Revised: While this design decision does increase the computational complexity of training, it signiﬁcantly decreases the latency at deployment.

Original: Our method is similar to RIS [5], which also performs planning during training instead of deployment.
Revised: Our method is similar to RIS [5], which also performs planning during training instead of deployment.

Original: Our method differs from RIS in how planning is used: whereas RIS modify the objective , our method uses planning to modify the data .
Revised: Our method differs from RIS in how planning is used: whereas RIS modify the objective , our method uses planning to modify the data .

Original: This difference not only uniﬁes the objectives for RL and graph planning, but also signiﬁcantly improves the performance of the algorithm.
Revised: This difference not only uniﬁes the objectives for RL and graph planning, but also signiﬁcantly improves the performance of the algorithm.

Original: , a subtle distinction that avoids favoring the learned policy and uniﬁes the objectives for RL and graph planning.
Revised: , a subtle distinction that avoids favoring the learned policy and uniﬁes the objectives for RL and graph planning.

Original: We demonstrate the importance of this difference in our experiments.
Revised: We demonstrate the importance of this difference in our experiments.

Original: Effectively solving goal-conditioned RL problems requires performing good exploration.
Revised: Effectively solving goal-conditioned RL problems requires performing good exploration.

Original: In the goal-conditioned setting, the quality of exploration depends on how goals are sampled, a problem studied in many prior methods [9, 13, 14, 33, 34, 36, 45, 46].
Revised: In the goal-conditioned setting, the quality of exploration depends on how goals are sampled, a problem studied in many prior methods [9, 13, 14, 33, 34, 36, 45, 46].

Original: These methods craft objectives that try to optimize for learning progress, and the resulting algorithms achieve good results across a range of environments.
Revised: These methods craft objectives that try to optimize for learning progress, and the resulting algorithms achieve good results across a range of environments.

Original: Our method differs from these methods in that the method for optimizing the goal-conditioned policy and the method for sampling waypoints are jointly optimized using the same objective.
Revised: Our method differs from these methods in that the method for optimizing the goal-conditioned policy and the method for sampling waypoints are jointly optimized using the same objective.

Original: We demonstrate the importance of this difference in our experiments.
Revised: We demonstrate the importance of this difference in our experiments.

Original: Our approaches can also be viewed as one Hierarchical RL [7, 25, 43] method with the distance function speciﬁed by the classiﬁer.
Revised: Our approaches can also be viewed as one Hierarchical RL [7, 25, 43] method with the distance function speciﬁed by the classiﬁer.

Original: However, one important difference between C-Planning and many other HRL methods is that our approach doesn’t require training a high-level policy for generating subgoals for the lower-level policy to reach [16, 25, 30].
Revised: However, one important difference between C-Planning and many other HRL methods is that our approach doesn’t require training a high-level policy for generating subgoals for the lower-level policy to reach [16, 25, 30].

Original: Comparing with approaches try to build a graph/tree for planning [10, 19, 32], our approach doesn’t require any planning at deployment, saves a lot of computation time.
Revised: Comparing with approaches try to build a graph/tree for planning [10, 19, 32], our approach doesn’t require any planning at deployment, saves a lot of computation time.

Original: Our method builds on the idea that reinforcement learning can be cast as inference problems [4, 23, 35, 41, 42, 47].
Revised: Our method builds on the idea that reinforcement learning can be cast as inference problems [4, 23, 35, 41, 42, 47].

Original: The observation that variational inference for certain problems corresponds to graph planning is closely related to Attias [4].
Revised: The observation that variational inference for certain problems corresponds to graph planning is closely related to Attias [4].

Original: Our work extends this inferential perspective to hierarchical models.
Revised: Our work extends this inferential perspective to hierarchical models.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We introduce the goal-conditioned RL problem and a recent goal-conditioned RL algorithm, Clearning, upon which our method builds.
Revised: We introduce the goal-conditioned RL problem and a recent goal-conditioned RL algorithm, Clearning, upon which our method builds.

Original: Goal-Conditioned RL.
Revised: Goal-Conditioned RL.

Original: We focus on controlled Markov processes deﬁned by states s t 2 S and actions a t .
Revised: We focus on controlled Markov processes deﬁned by states s t ∈ S and actions a t .

Original: The initial state is sampled s [-
 ⇠-] p 0 ( s 0 ) and subsequent states are sampled s t +1 ⇠ p ( s t +1 | s t , a t ) .
Revised: The initial state is sampled s [-
 ⇠-]0 ∼ p 0 ( s 0 ) and subsequent states are sampled s t +1 ∼ p ( s t +1 | s t , a t ) .

Original: Our aim is to navigate to the goal states, s g 2 S , which are sampled from s g ⇠ p g( s g ) .
Revised: Our aim is to navigate to the goal states, s g ∈ S , which are sampled from s g ∼ p g

Original:  We deﬁne a policy ⇡ ( a t | s t , s g ) conditioned on both the current state and the goal.
Revised: ( s g ) . We deﬁne a policy π ( a t | s t , s g ) conditioned on both the current state and the goal.

Original: The objective is to maximize the probability (density) of reaching the goal in the future.
Revised: The objective is to maximize the probability (density) of reaching the goal in the future.

Original: To derive our method, or any other goal-conditioned RL algorithm, we must make a modeling assumption about when we would like the agent to reach the goal.
Revised: To derive our method, or any other goal-conditioned RL algorithm, we must make a modeling assumption about when we would like the agent to reach the goal.

Original: This modeling assumption is only used to derive the algorithm, not for evaluation.
Revised: This modeling assumption is only used to derive the algorithm, not for evaluation.

Original: Formally, let + be a random integer indicating when we reach the goal.
Revised: Formally, let ∆ ∈ N + be a random integer indicating when we reach the goal.

Original: The user speciﬁes a prior p (  ) .
Revised: The user speciﬁes a prior p (∆) .

Original: Most prior work (implicitly) uses the geometric distribution, p (  )
Revised: Most prior work (implicitly) uses the geometric distribution, p (∆)

Original: =
Revised: =

Original: G EOM (1   ) .
Revised: G EOM (1 − γ ) .

Original: The geometric distribution is ubiquitous because it is easy to incorporate into temporal difference learning.
Revised: The geometric distribution is ubiquitous because it is easy to incorporate into temporal difference learning.

Original: Despite its wide usage, it may not reﬂect the prior belief on when the agent will solve the task or when the episode will terminate.
Revised: Despite its wide usage, it may not reﬂect the prior belief on when the agent will solve the task or when the episode will terminate.

Original: Thus, while it is natural to use geometric distribution, other choices of distributions can also be consdiered.
Revised: Thus, while it is natural to use geometric distribution, other choices of distributions can also be consdiered.

Original: Prior work has also considered non-geometric distributions [12].
Revised: Prior work has also considered non-geometric distributions [12].

Original: In this paper, we use the negative binomial distribution, p (  ) =
Revised: In this paper, we use the negative binomial distribution, p (∆) =

Original: N EG B INOM ( p = 1   , n = 2) .
Revised: N EG B INOM ( p = 1 − γ, n = 2) .

Original: Given a prior p (  ) , we deﬁne the distribution over future states as
Revised: Given a prior p (∆) , we deﬁne the distribution over future states as

Original: [Equation]
Revised: [Equation]

Original: where p ⇡  ( s t +  | s t , a t ) is the probability density of reaching state s t +  exactly  steps in the future when sampling actions from ⇡ ( a t | s t , s g ) .
Revised: where p π ∆ ( s t +∆ | s t , a t ) is the probability density of reaching state s t +∆ exactly ∆ steps in the future when sampling actions from π ( a t | s t , s g ) .

Original: For example, the  -discounted state occupancy measure [17, 31] can be written as p ⇡ ( ·|· ,s g ) G EOM (1   ) .
Revised: For example, the γ -discounted state occupancy measure [17, 31] can be written as p π ( ·|· ,s g ) G EOM 1 − γ ) .

Original: Our objective for goal-reaching is to maximize the probability of reaching the desired goal:
Revised: Our objective for goal-reaching is to maximize the probability of reaching the desired goal:

Original: [Equation]
Revised: [Equation]

Original: C-Learning.
Revised: C-Learning.

Original: Our method builds upon a prior method for goal-conditioned RL, C-Learning [11].
Revised: Our method builds upon a prior method for goal-conditioned RL, C-Learning [11].

Original: C-Learning learns a classiﬁer for predicting whether a state s t comes from a future state density p ⇡ ( ·|· , s g ) p (  ) ( s t + = s t + | s t , a t ) or a marginal state density:
Revised: C-Learning learns a classiﬁer for predicting whether a state s t comes from a future state density p π ( ·|· , s g ) p (∆) ( s t + = s t + | s t , a t ) or a marginal state density:

Original: [Equation]
Revised: [Equation]

Original: The Bayes-optimal classiﬁer can be written in terms of these two distributions:
Revised: The Bayes-optimal classiﬁer can be written in terms of these two distributions:

Original: [Equation]
Revised: [Equation]

Original: In C-Learning, this classiﬁer acts as a value function for training the policy.
Revised: In C-Learning, this classiﬁer acts as a value function for training the policy.

Original: Our method will use this classiﬁer not only to update the policy, but also to sample waypoints.
Revised: Our method will use this classiﬁer not only to update the policy, but also to sample waypoints.

Original:  C-P LANNING : G OAL -C ONDITIONED RL WITH P LANNING
Revised:  C-P LANNING : G OAL -C ONDITIONED RL WITH P LANNING

Original: We now present our method, C-Planning.
Revised: We now present our method, C-Planning.

Original: The main idea behind our method is to decompose the objective of Eq.
Revised: The main idea behind our method is to decompose the objective of Eq.

Original: 2 into a sequence of easier goal-reaching problems.
Revised: 2 into a sequence of easier goal-reaching problems.

Original: Intuitively, if we want a suboptimal agent to navigate from some initial state s 0 to a distant goal state s g , having the agent try to reach that state again and again will likely prove futile.
Revised: Intuitively, if we want a suboptimal agent to navigate from some initial state s 0 to a distant goal state s g , having the agent try to reach that state again and again will likely prove futile.

Original: [Table]
Revised: [Figure]

Original: (cid:19)

Original: Instead, we can command a sequence of goals, like a trail of breadcrumbs leading to the goal s g .
Revised: Instead, we can command a sequence of goals, like a trail of breadcrumbs leading to the goal s g .

Original: To select these waypoints, we will command the policy to reach those states it would visit if it were successful at reaching s g .
Revised: To select these waypoints, we will command the policy to reach those states it would visit if it were successful at reaching s g .

Original: Sampling subgoals in this way can ease the difﬁculty of training the agent (see Fig. 2.).
Revised: Sampling subgoals in this way can ease the difﬁculty of training the agent (see Fig. 2.).

Original: At a high-level algorithm summary, our method consists of two steps: using waypoing sampling to collect experience and performing standard goal-conditioned RL using the collected experience.
Revised: At a high-level algorithm summary, our method consists of two steps: using waypoing sampling to collect experience and performing standard goal-conditioned RL using the collected experience.

Original: The main challenge is the ﬁrst step, as modeling the distribution over waypoints is difﬁcult.
Revised: The main challenge is the ﬁrst step, as modeling the distribution over waypoints is difﬁcult.

Original: In Sec. 4.1, we solve analytically for waypoint distribution, and then propose a simple practical method to sample from this distribution in Sec. 4.2.
Revised: In Sec. 4.1, we solve analytically for waypoint distribution, and then propose a simple practical method to sample from this distribution in Sec. 4.2.

Original: 4. P LANNING AND V ARIATIONAL I NFERENCE
Revised: 4. P LANNING AND V ARIATIONAL I NFERENCE

Original: To formally derive our method for waypoint sampling, we cast the problem of goal-reaching as a latent variable problem.
Revised: To formally derive our method for waypoint sampling, we cast the problem of goal-reaching as a latent variable problem.

Original: We assume that the agent starts at state s 0 and ends at state s g , but the intermediate states that the agent will visit are unknown latent variables.
Revised: We assume that the agent starts at state s 0 and ends at state s g , but the intermediate states that the agent will visit are unknown latent variables.

Original: This problem resembles standard latent variable modeling problems (e.g., VAE), where the intermediate state serves the role of the inferred representation.
Revised: This problem resembles standard latent variable modeling problems (e.g., VAE), where the intermediate state serves the role of the inferred representation.

Original: We can thus derive an evidence lower bound on the likelihood of reaching the desired goal:
Revised: We can thus derive an evidence lower bound on the likelihood of reaching the desired goal:

Original: Lemma 1.
Revised: Lemma 1.

Original: The objective L , deﬁned below, is a lower bound on the goal-reaching objective (Eq. 2):
Revised: The objective L , deﬁned below, is a lower bound on the goal-reaching objective (Eq. 2):

Original: [Equation]
Revised: [Equation]

Original: See Appendix A.1 for the proof.
Revised: See Appendix A.1 for the proof.

Original: The lower bound, L , depends on two quantities: the goalconditioned policy and an inferred distribution over waypoints, q ( s w | s 0 , s g ) .
Revised: The lower bound, L , depends on two quantities: the goalconditioned policy and an inferred distribution over waypoints, q ( s w | s 0 , s g ) .

Original: This bound holds for any choice of q ( s w | s 0 , s g ) , and we can optimize this lower bound with respect to this waypoint distribution.
Revised: This bound holds for any choice of q ( s w | s 0 , s g ) , and we can optimize this lower bound with respect to this waypoint distribution.

Original: We can see more intuition behind the negative binomial distribution here.
Revised: We can see more intuition behind the negative binomial distribution here.

Original: With the negative binomial distribution, we can decompose a hard problem into a sequence of easier RL problems.
Revised: With the negative binomial distribution, we can decompose a hard problem into a sequence of easier RL problems.

Original: Each of these easier (goal-conditioned) RL problems uses geometric distributions, and the sum of geometric random variables is a negative binomial distribution.
Revised: Each of these easier (goal-conditioned) RL problems uses geometric distributions, and the sum of geometric random variables is a negative binomial distribution.

Original: We will perform expectation maximization [6] to optimize the lower bound, alternating between estimating waypoint distribution and optimizing the goal-conditioned policy using this waypoint distribution.
Revised: We will perform expectation maximization [6] to optimize the lower bound, alternating between estimating waypoint distribution and optimizing the goal-conditioned policy using this waypoint distribution.

Original: We now describe these two steps in detail.
Revised: We now describe these two steps in detail.

Original: E-Step.
Revised: E-Step.

Original: The E-step estimates the waypoint distribution, q ( s w | s 0 , s g ) .
Revised: The E-step estimates the waypoint distribution, q ( s w | s 0 , s g ) .

Original: The three terms in the lower bound indicate that the sampled waypoint should be reachable from the initial state, the goal state should be reachable from the sampled waypoint, and that the waypoint distribution should have high entropy.
Revised: The three terms in the lower bound indicate that the sampled waypoint should be reachable from the initial state, the goal state should be reachable from the sampled waypoint, and that the waypoint distribution should have high entropy.

Original: These ﬁrst two terms resemble shortest-path planning, where distances are measured using log probabilities.
Revised: These ﬁrst two terms resemble shortest-path planning, where distances are measured using log probabilities.

Original: Importantly, reachability is deﬁned in terms of the capabilities of the current policy.
Revised: Importantly, reachability is deﬁned in terms of the capabilities of the current policy.

Original: We can analytically solve for the waypoint distribution:
Revised: We can analytically solve for the waypoint distribution:

Original: Lemma 2.
Revised: Lemma 2.

Original: The waypoint distribution that optimizes our lower bound (Lemma 1) satisﬁes:
Revised: The waypoint distribution that optimizes our lower bound (Lemma 1) satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: See Appendix A.1 for a full derivation.
Revised: See Appendix A.1 for a full derivation.

Original: We illustrate this waypoint distribution in Fig.
Revised: We illustrate this waypoint distribution in Fig.

Original: In general, accurately estimating the distribution, q ⇤ , is challenging.
Revised: In general, accurately estimating the distribution, q ∗ , is challenging.

Original: However, in the next section, we develop a simple algorithm that only requires learning a classiﬁer instead of a generative model.
Revised: However, in the next section, we develop a simple algorithm that only requires learning a classiﬁer instead of a generative model.

Original: M-step.
Revised: M-step.

Original: The M-step optimizes the lower bound with respect to the goal-conditioned policy.
Revised: The M-step optimizes the lower bound with respect to the goal-conditioned policy.

Original: We can ignore the  log q ( s w | s 0 , s g ) term, which does not depend on the policy.
Revised: We can ignore the − log q ( s w | s 0 , s g ) term, which does not depend on the policy.

Original: The remaining two terms look like goal-conditioned RL objectives, with a subtle but important difference in how the trajectories are sampled.
Revised: The remaining two terms look like goal-conditioned RL objectives, with a subtle but important difference in how the trajectories are sampled.

Original: When collecting data, a standard goal-conditioned RL algorithm initially samples a goal and then collects a trajectory where the policy attempts to reach that goal.
Revised: When collecting data, a standard goal-conditioned RL algorithm initially samples a goal and then collects a trajectory where the policy attempts to reach that goal.

Original: Our method collects data in a different manner.
Revised: Our method collects data in a different manner.

Original: The agent samples a goal, then samples an intermediate waypoint that should lead to that goal.
Revised: The agent samples a goal, then samples an intermediate waypoint that should lead to that goal.

Original: The agent attempts to reach the waypoint before attempting to reach the goal.
Revised: The agent attempts to reach the waypoint before attempting to reach the goal.

Original: After the trajectory has been collected, the policy updates are the same as a standard goal-conditioned RL algorithm.
Revised: After the trajectory has been collected, the policy updates are the same as a standard goal-conditioned RL algorithm.

Original: Two mental models.
Revised: Two mental models.

Original: The algorithm sketch we have presented, which will be ﬂeshed out in the subsequent section, can be interpreted in two ways.
Revised: The algorithm sketch we have presented, which will be ﬂeshed out in the subsequent section, can be interpreted in two ways.

Original: One interpretation is that the method performs a soft version of graph planning during exploration, using a distance function of d ( s 1 , s 2 )
Revised: One interpretation is that the method performs a soft version of graph planning during exploration, using a distance function of d ( s 1 , s 2 )

Original: = log p ⇡ ( ·|· , s 2 ) G EOM ( s t + = s 2 | s 1 ) .
Revised: = log p π ( ·|· , s 2 ) G EOM ( s t + = s 2 | s 1 ) .

Original: From this perspective, the method is similar to prior work that performs graph search during test-time [10, 37].
Revised: From this perspective, the method is similar to prior work that performs graph search during test-time [10, 37].

Original: However, our method will perform planning at training time.
Revised: However, our method will perform planning at training time.

Original: Extensive ablations in Sec.
Revised: Extensive ablations in Sec.

Original: 5 show that our method outperforms alternative approaches that only perform search during test-time.
Revised: 5 show that our method outperforms alternative approaches that only perform search during test-time.

Original: The second interpretation focuses on the task of reaching the goal from different initial states.
Revised: The second interpretation focuses on the task of reaching the goal from different initial states.

Original: The choice of waypoint distribution, and the policy’s success at reaching those waypoints, determines the initial state distribution for this task.
Revised: The choice of waypoint distribution, and the policy’s success at reaching those waypoints, determines the initial state distribution for this task.

Original: Intuitively, the best initial state distribution is one that sometimes starts the agent at the true initial state s 0 , sometimes starts the agent at the ﬁnal state s g , and sometimes starts the agent in between.
Revised: Intuitively, the best initial state distribution is one that sometimes starts the agent at the true initial state s 0 , sometimes starts the agent at the ﬁnal state s g , and sometimes starts the agent in between.

Original: In fact, prior work has formally shown that the optimal initial state distribution is the marginal state distribution of the optimal policy [21].
Revised: In fact, prior work has formally shown that the optimal initial state distribution is the marginal state distribution of the optimal policy [21].

Original: Under somewhat strong assumptions, 2 this is precisely what our waypoint sampling achieves.
Revised: Under somewhat strong assumptions, 2 this is precisely what our waypoint sampling achieves.

Original: We refer the reader for the discussion and experiments in Appendix.
Revised: We refer the reader for the discussion and experiments in Appendix.

Original: A.
Revised: A.

Original: A P RACTICAL I MPLEMENTATION
Revised: A P RACTICAL I MPLEMENTATION

Original: We now describe how we implement the policy updates (M-step) and waypoint sampling (E-step).
Revised: We now describe how we implement the policy updates (M-step) and waypoint sampling (E-step).

Original: For the policy updates, we simply apply C-learning [11].
Revised: For the policy updates, we simply apply C-learning [11].

Original: The main challenge is waypoint sampling, which requires sampling a potentially high-dimensional waypoint distribution.
Revised: The main challenge is waypoint sampling, which requires sampling a potentially high-dimensional waypoint distribution.

Original: While prior work [14] has approached such problems by ﬁtting high-capacity generative models, our approach will avoid such generative models and instead use importance sampling.
Revised: While prior work [14] has approached such problems by ﬁtting high-capacity generative models, our approach will avoid such generative models and instead use importance sampling.

Original: Let b ( · ) be the background distribution, which we take to be the replay buffer.
Revised: Let b ( · ) be the background distribution, which we take to be the replay buffer.

Original: Our algorithm corresponds to the following two-step procedure.
Revised: Our algorithm corresponds to the following two-step procedure.

Original: First, we sample a batch of waypoints from the buffer.
Revised: First, we sample a batch of waypoints from the buffer.

Original: Then, we sample one waypoint from within that batch using the normalized importance weights, q ( s w | s g , s 0 ) b ( s w ) .
Revised: Then, we sample one waypoint from within that batch using the normalized importance weights, q ( s w | s g , s 0 ) b ( s w ) .

Original: We can estimate these importance weights using the classiﬁer learned by C-learning (Eq. 3):
Revised: We can estimate these importance weights using the classiﬁer learned by C-learning (Eq. 3):

Original: Lemma 3.
Revised: Lemma 3.

Original: We can write the importance weights in terms of this value function
Revised: We can write the importance weights in terms of this value function

Original: [Equation]
Revised: [Equation]

Original: where Z ( s 0 , s g ) is a normalizing constant.
Revised: where Z ( s 0 , s g ) is a normalizing constant.

Original: See Appendix A.3 for a full derivation.
Revised: See Appendix A.3 for a full derivation.

Original: Since we will eventually normalize these importance weights (over s w ), we do not need to estimate Z ( s 0 , s g ) .
Revised: Since we will eventually normalize these importance weights (over s w ), we do not need to estimate Z ( s 0 , s g ) .

Original: We call our complete algorithm C-Planning , and summarize it in Alg. 1.
Revised: We call our complete algorithm C-Planning , and summarize it in Alg. 1.

Original: Note that our algorithm requires changing only a single line of pseudocode: during data collection, we command the agent to the intermediate waypoint s w instead of the ﬁnal goal s g .
Revised: Note that our algorithm requires changing only a single line of pseudocode: during data collection, we command the agent to the intermediate waypoint s w instead of the ﬁnal goal s g .

Original: The sampling procedure in Alg.
Revised: The sampling procedure in Alg.

Original: 2 is scalable and easy to implement.
Revised: 2 is scalable and easy to implement.

Original: The core idea of our method is to sample waypoints enroute to the goal.
Revised: The core idea of our method is to sample waypoints enroute to the goal.

Original: Of course, at some point we must command the agent to directly reach the goal.
Revised: Of course, at some point we must command the agent to directly reach the goal.

Original: We introduce a hyperparameter n g to indicate how many times we resample the intermediate waypoint before commanding the agent to reach the goal.
Revised: We introduce a hyperparameter n g to indicate how many times we resample the intermediate waypoint before commanding the agent to reach the goal.

Original: This parameter is not the planning horizon, which is always set to 2.[-1: D ;  i 4: s 
-]
Revised: This parameter is not the planning horizon, which is always set to 2.[-1: D ;  i 4: s 
-]

Original: ⇠ p 0 ( s 0 ) , s g

Original: ⇠ p g ( s g ) 5: (((((((( ⌧ ⇠ p ⇡ ( ⌧ | s 0 , s g ) 6: s g

Original: CP LAN

Original: NING ( g

Original: C, s g ) 7: ⌧ ( ⌧ 1 ⇠ p ⇡ ( ⌧ | s s w ) , ⌧ 2 ⇠ p ⇡ ( ⌧ | s s w )) 8:

Original: D D [ { ⌧ } 9: ⇡

Original: C

Original: CL d log C ( F =1 | s w s g ) C

Original: ( F =0 | s w s g )

Original: + log C

Original: ( F

Original: 1 | s s w

Original: C ( F

Original: 0 | s

Original: 7: s

Original: Set

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments study whether C-Planning can compete with prior goal-conditioned RL methods both on benchmark tasks and on tasks designed to pose a signiﬁcant planning and exploration challenge.
Revised: Our experiments study whether C-Planning can compete with prior goal-conditioned RL methods both on benchmark tasks and on tasks designed to pose a signiﬁcant planning and exploration challenge.

Original: Our ﬁrst experiments use these tasks to compare C-Planning to prior methods for goalconditioned RL.
Revised: Our ﬁrst experiments use these tasks to compare C-Planning to prior methods for goalconditioned RL.

Original: We then study the importance of a few design decisions through ablation experiments.
Revised: We then study the importance of a few design decisions through ablation experiments.

Original: The beneﬁts of C-Planning, compared with prior methods, are especially pronounced on these challenging tasks.
Revised: The beneﬁts of C-Planning, compared with prior methods, are especially pronounced on these challenging tasks.

Original: To provide more intuition for why C-Planning outperforms prior goal-conditioned
Revised: To provide more intuition for why C-Planning outperforms prior goal-conditioned

Original: RL methods, we plot the gradient norm for the actor and critic functions, ﬁnding that C-Planning enjoys larger norm gradients for the critic and smaller variance of the gradient for the actor.
Revised: RL methods, we plot the gradient norm for the actor and critic functions, ﬁnding that C-Planning enjoys larger norm gradients for the critic and smaller variance of the gradient for the actor.

Original: Ablation experiments study the number of waypoints used in training.
Revised: Ablation experiments study the number of waypoints used in training.

Original: We ﬁnally measure the inference time of choosing actions, ﬁnding that C-Planning is up to 4 .
Revised: We ﬁnally measure the inference time of choosing actions, ﬁnding that C-Planning is up to 4 .

Original: 7 ⇥ faster than prior methods that perform search at test time.
Revised: 7 × faster than prior methods that perform search at test time.

Original: [Figure]
Revised: [Figure]

Original: Environments.
Revised: Environments.

Original: We investigate environments of various difﬁculty, some of which we visualize in in Fig.
Revised: We investigate environments of various difﬁculty, some of which we visualize in in Fig.

Original: The ﬁrst set of environments is taken directly from the Metaworld benchmark suite [44], a common benchmark for goalconditioned RL.
Revised: The ﬁrst set of environments is taken directly from the Metaworld benchmark suite [44], a common benchmark for goalconditioned RL.

Original: These tasks are challenging because of the manipulation skills required to push and reorient objects.
Revised: These tasks are challenging because of the manipulation skills required to push and reorient objects.

Original: While prior goalconditioned RL methods do solve some of these tasks (e.g., Push and Reach ), they stand at the edge of the capabilities of current methods.
Revised: While prior goalconditioned RL methods do solve some of these tasks (e.g., Push and Reach ), they stand at the edge of the capabilities of current methods.

Original: The second set of environments, 2D navigation mazes, are designed to stress-test the planning capabilities of different methods.
Revised: The second set of environments, 2D navigation mazes, are designed to stress-test the planning capabilities of different methods.

Original: Successfully solving these tasks requires reasoning over long horizons of up to 50 steps.
Revised: Successfully solving these tasks requires reasoning over long horizons of up to 50 steps.

Original: These are challenging because, unlike many benchmarks, they require non-greedy exploration: greedy strategies get stuck in local optima.
Revised: These are challenging because, unlike many benchmarks, they require non-greedy exploration: greedy strategies get stuck in local optima.

Original: Our ﬁnal set of environments combine the challenges of these environments.
Revised: Our ﬁnal set of environments combine the challenges of these environments.

Original: We extend the tasks in Metaworld to involve sequential manipulation of multiple objects, similar to the tasks in prior work [40].
Revised: We extend the tasks in Metaworld to involve sequential manipulation of multiple objects, similar to the tasks in prior work [40].

Original: For example, the Obstacle-Drawer-Close task require the robotic arm to manipulate multiple objects in sequence, ﬁrst pushing an object and then opening a drawer.
Revised: For example, the Obstacle-Drawer-Close task require the robotic arm to manipulate multiple objects in sequence, ﬁrst pushing an object and then opening a drawer.

Original: Solving these tasks are difﬁcult because the agent is given no demonstrations, no reward shaping, and no manually speciﬁed distance metrics.
Revised: Solving these tasks are difﬁcult because the agent is given no demonstrations, no reward shaping, and no manually speciﬁed distance metrics.

Original: Baselines.
Revised: Baselines.

Original: We will compare C-Planning to a number of baselines.
Revised: We will compare C-Planning to a number of baselines.

Original: C-Learning is a recent goalconditioned RL method that does not perform planning. 
Revised: C-Learning is a recent goalconditioned RL method that does not perform planning. C-Learning performs the same gradient [Figure] C-learning

Revised: RIS

Original: C-Learning performs the same gradient updates as C-Planning, and only differs in how experience is collected.
Revised: SoRB(C-learning) C-Planning C-learning RIS SoRB(C-learning)

Revised: C-Planning

Original:  Comparing to C-Learning will allow us to identify the marginal contribution of our curriculum of waypoints.
Revised: C-learning RIS SoRB(C-learning) updates as C-Planning, and only differs in how experience is collected. Comparing to C-Learning will allow us to identify the marginal contribution of our curriculum of waypoints.

Original: The second baseline is SoRB [10], a goal-conditioned RL method that performs search at test-time, rather than training time.
Revised: The second baseline is SoRB [10], a goal-conditioned RL method that performs search at test-time, rather than training time.

Original: We didn’t manage to run the comparison with [19] since it requires a reward function and a termination function from the environment.
Revised: We didn’t manage to run the comparison with [19] since it requires a reward function and a termination function from the environment.

Original: While SoRB was originally implemented using
Revised: While SoRB was originally implemented using

Original: Q-learning, we ﬁnd that a version based on C-learning worked substantially better, so we use this stronger baseline in our experiments.
Revised: Q-learning, we ﬁnd that a version based on C-learning worked substantially better, so we use this stronger baseline in our experiments.

Original: Comparing against SoRB will allow us to study the tradeoffs between performing planning during training versus testing.
Revised: Comparing against SoRB will allow us to study the tradeoffs between performing planning during training versus testing.

Original: Because SoRB performs search at testing, it is considerably more expensive to deploy in terms of computing.
Revised: Because SoRB performs search at testing, it is considerably more expensive to deploy in terms of computing.

Original: Thus, even matching the performance of SoRB, without incurring the computational costs of deployment would be a useful result.
Revised: Thus, even matching the performance of SoRB, without incurring the computational costs of deployment would be a useful result.

Original: The third baseline is an ablation of our method designed to resemble RIS [5].
Revised: The third baseline is an ablation of our method designed to resemble RIS [5].

Original: Like CPlanning, RIS performs planning during training and not testing, but the planning is used differently from C-Planning.
Revised: Like CPlanning, RIS performs planning during training and not testing, but the planning is used differently from C-Planning.

Original: Whereas C-Planning uses planning to collect data, leaving the gradient updates unchanged, RIS modiﬁes the RL objective to include an additional term that resembles behavior cloning.
Revised: Whereas C-Planning uses planning to collect data, leaving the gradient updates unchanged, RIS modiﬁes the RL objective to include an additional term that resembles behavior cloning.

Original: Our comparison with RIS thus allows us to study how our method for using the sampled waypoints compares to alternative methods to learn from those same sampled waypoints.
Revised: Our comparison with RIS thus allows us to study how our method for using the sampled waypoints compares to alternative methods to learn from those same sampled waypoints.

Original: 5. C OMPARISON WITH P RIOR G OAL -C ONDITIONED RL M ETHODS
Revised: 5. C OMPARISON WITH P RIOR G OAL -C ONDITIONED RL M ETHODS

Original: To compare C-Planning to prior goal-conditioned RL algorithms, we start with three tasks from the
Revised: To compare C-Planning to prior goal-conditioned RL algorithms, we start with three tasks from the

Original: MetaWorld suite: Reach , Push , Push-Wall .
Revised: MetaWorld suite: Reach , Push , Push-Wall .

Original: While C-Planning learns slightly slower than prior methods on the easiest task ( Reach ), we see a noticeable improvement over prior methods on the more challenging Push and Push-Wall tasks, tasks that require reasoning over longer horizons to solve.
Revised: While C-Planning learns slightly slower than prior methods on the easiest task ( Reach ), we see a noticeable improvement over prior methods on the more challenging Push and Push-Wall tasks, tasks that require reasoning over longer horizons to solve.

Original: This observation suggests that that the waypoint sampling performed by C-Planning might be especially beneﬁcial for solving tasks that require planning over long horizons.
Revised: This observation suggests that that the waypoint sampling performed by C-Planning might be especially beneﬁcial for solving tasks that require planning over long horizons.

Original: To test the planning capabilities of C-Planning, we design a series of 2D navigation mazes.
Revised: To test the planning capabilities of C-Planning, we design a series of 2D navigation mazes.

Original: While the underlying locomotion motions are simple, these tasks post challenges for planning and are designed so that a greedy planning algorithm would fail.
Revised: While the underlying locomotion motions are simple, these tasks post challenges for planning and are designed so that a greedy planning algorithm would fail.

Original: On all three mazes, C-Planning learns faster than all baselines and achieves a lower asymptotic distance to the goal, as compared to the baselines. 
Revised: On all three mazes, C-Planning learns faster than all baselines and achieves a lower asymptotic distance to the goal, as compared to the baselines. On

Original: On the most challenging maze, Maze-11x11 , only our method is able to make any learning progress.
Revised: [Figure] the most challenging maze, Maze-11x11 , only our method is able to make any learning progress.

Original: Of particular note is the comparison with SoRB, which uses the same underlying goal-conditioned
Revised: Of particular note is the comparison with SoRB, which uses the same underlying goal-conditioned

Original: RL algorithm as C-Planning, but differs in that it performs search at testing, rather than training.
Revised: RL algorithm as C-Planning, but differs in that it performs search at testing, rather than training.

Original: While SoRB performs better than C-learning, which does not perform planning, SoRB consistently performs worse than C-Planning.
Revised: While SoRB performs better than C-learning, which does not perform planning, SoRB consistently performs worse than C-Planning.

Original: This observation suggests that C-Planning is not just amortizing the cost of performing search.
Revised: This observation suggests that C-Planning is not just amortizing the cost of performing search.

Original: Rather, these results suggest that incorporating planning into training can produce signiﬁcantly larger gains than incorporating planning into the deployed policy.
Revised: Rather, these results suggest that incorporating planning into training can produce signiﬁcantly larger gains than incorporating planning into the deployed policy.

Original: As the last set of experiments, study higher dimensional tasks that require long-range planning.
Revised: As the last set of experiments, study higher dimensional tasks that require long-range planning.

Original: We design three environments: Obstacle-Drawer-Close , Obstacle-Drawer-Open and
Revised: We design three environments: Obstacle-Drawer-Close , Obstacle-Drawer-Open and

Original: Push-Two .
Revised: Push-Two .

Original: These tasks have a fairly large dimension (15 to 18) and require sequential manipulation of multiple objects.
Revised: These tasks have a fairly large dimension (15 to 18) and require sequential manipulation of multiple objects.

Original: C-Planning outperforms all baselines on all three tasks.
Revised: C-Planning outperforms all baselines on all three tasks.

Original: While C-learning makes some learning progress, SoRB (which modiﬁes C-learning with search at test time) does not improve the results.
Revised: While C-learning makes some learning progress, SoRB (which modiﬁes C-learning with search at test time) does not improve the results.

Original: RIS, which also samples waypoints during training but uses those waypoints differently, does not make any learning progress on these tasks.
Revised: RIS, which also samples waypoints during training but uses those waypoints differently, does not make any learning progress on these tasks.

Original: Taken together, these results suggest that waypoint sampling (as performed by C-Planning) can signiﬁcantly aid in the solving of complex manipulation tasks, but only if that waypoint sampling is performed during training.
Revised: Taken together, these results suggest that waypoint sampling (as performed by C-Planning) can signiﬁcantly aid in the solving of complex manipulation tasks, but only if that waypoint sampling is performed during training.

Original: Moreover, the comparison with RIS suggests that those waypoints should be used to collect new data, rather than to augment the policy learning objective.
Revised: Moreover, the comparison with RIS suggests that those waypoints should be used to collect new data, rather than to augment the policy learning objective.

Original: To the best of our knowledge, C-Planning is the ﬁrst method to learn manipulation behavior of this complexity without additional assumptions (such as dense rewards or demonstrations).
Revised: To the best of our knowledge, C-Planning is the ﬁrst method to learn manipulation behavior of this complexity without additional assumptions (such as dense rewards or demonstrations).

Original: 5. W HY D OES C-P 
Revised: 5. W HY D OES C-P LANNING W ORK ?

Original: LANNING W ORK ?

Original: We ran many additional experiments to understand why C-Planning works so well.
Revised: We ran many additional experiments to understand why C-Planning works so well.

Original: To start, we run ablation experiments to answer two questions: (1) Although C-Planning only applies planning at training time, does additionally performing planning at test time further improve performance?
Revised: To start, we run ablation experiments to answer two questions: (1) Although C-Planning only applies planning at training time, does additionally performing planning at test time further improve performance?

Original: How many times should we resample the waypoint before directing the agent to the goal?
Revised: How many times should we resample the waypoint before directing the agent to the goal?

Original: Fig 5 shows the results of ablation experiments.
Revised: Fig 5 shows the results of ablation experiments.

Original: C-Planning performs the same at test-time with and without SoRB-based planning, suggesting that our training-time planning procedure already makes the policy sufﬁciently capable of reaching distant goals, such that it does not beneﬁt from additional test-time planning.
Revised: C-Planning performs the same at test-time with and without SoRB-based planning, suggesting that our training-time planning procedure already makes the policy sufﬁciently capable of reaching distant goals, such that it does not beneﬁt from additional test-time planning.

Original: Fig.
Revised: Fig.

Original: 5 also shows that C-Planning is relatively robust to n g , the number of waypoints used before directing the agent to the goal.
Revised: 5 also shows that C-Planning is relatively robust to n g , the number of waypoints used before directing the agent to the goal.

Original: For all choices of this hyperparameter, CPlanning still manages to solve all the tasks.
Revised: For all choices of this hyperparameter, CPlanning still manages to solve all the tasks.

Original: Gradient analysis.
Revised: Gradient analysis.

Original: We provide an empirical observation why C-Planning is better than vanilla goal-conditioned RL methods.
Revised: We provide an empirical observation why C-Planning is better than vanilla goal-conditioned RL methods.

Original: We hypothesize that, by introducing waypoints, C-Planning provides a better learning signal, increasing the gradient norm for training the critic and decreasing the variance of the gradient for the policy.
Revised: We hypothesize that, by introducing waypoints, C-Planning provides a better learning signal, increasing the gradient norm for training the critic and decreasing the variance of the gradient for the policy.

Original: Prior work has theoretically analyzed the importance of gradient norms for RL [1] and we will provide some empirical evidence for the importance of controlling gradient norms.
Revised: Prior work has theoretically analyzed the importance of gradient norms for RL [1] and we will provide some empirical evidence for the importance of controlling gradient norms.

Original: In addition, variance reduction has also long been a core problem in
Revised: In addition, variance reduction has also long been a core problem in

Original: RL [3, 15] and smaller variance will help stabilize the RL algorithm.
Revised: RL [3, 15] and smaller variance will help stabilize the RL algorithm.

Original: In Fig.
Revised: In Fig.

Original: 7, we plot the norm of the gradient (mean and variance) of both C-Planning and C-Learning using the Maze-11x11 environment, ﬁnding that our method increases the norm of the critic norm and decreases the variance of the actor gradient norm.
Revised: 7, we plot the norm of the gradient (mean and variance) of both C-Planning and C-Learning using the Maze-11x11 environment, ﬁnding that our method increases the norm of the critic norm and decreases the variance of the actor gradient norm.

Original: Test-time latency.
Revised: Test-time latency.

Original: One important advantage of C-Planning is that it enables direct execution at test time, does not require any online planning.
Revised: One important advantage of C-Planning is that it enables direct execution at test time, does not require any online planning.

Original: At test time, we directly set the agent’s goal to the ﬁnal goal state without commanding any intermediate states.
Revised: At test time, we directly set the agent’s goal to the ﬁnal goal state without commanding any intermediate states.

Original: This makes C-Planning considerably different from previous planning methods, such as SoRB [10].
Revised: This makes C-Planning considerably different from previous planning methods, such as SoRB [10].

Original: Not only does our method achieve a higher return, but it does so with signiﬁcantly less computing at deployment.
Revised: Not only does our method achieve a higher return, but it does so with signiﬁcantly less computing at deployment.

Original: We measure the execution time at evaluation time for C-Planning and SoRB with a various number of waypoints in Fig.
Revised: We measure the execution time at evaluation time for C-Planning and SoRB with a various number of waypoints in Fig.

Original: C-Planning is 1.74 ⇥ faster than planning over four waypoints and 4.71 ⇥ faster than planning overwaypoints in Push-Wall environment.
Revised: C-Planning is 1.74 × faster than planning over four waypoints and 4.71 × faster than planning overwaypoints in Push-Wall environment.

Original: Visualizing the training dynamics.
Revised: Visualizing the training dynamics.

Original: To further gain intuition into the mechanics of our method, we visualize how the distribution over waypoints changes during training of the 2D navigation of the four rooms environment.
Revised: To further gain intuition into the mechanics of our method, we visualize how the distribution over waypoints changes during training of the 2D navigation of the four rooms environment.

Original: Fig.
Revised: Fig.

Original: 6 shows the sampled waypoints.
Revised: 6 shows the sampled waypoints.

Original: The value functions (i.e., future state classiﬁers) are randomly initialized at the start of training, so the waypoints sampled are roughly uniform over the state space.
Revised: The value functions (i.e., future state classiﬁers) are randomly initialized at the start of training, so the waypoints sampled are roughly uniform over the state space.

Original: As training progresses, the distribution over waypoints converges to the states that an optimal policy would visit enroute to the goal.
Revised: As training progresses, the distribution over waypoints converges to the states that an optimal policy would visit enroute to the goal.

Original: While we have only shown one goal here, our method trains the policy for reaching all goals.
Revised: While we have only shown one goal here, our method trains the policy for reaching all goals.

Original: This set of experiments provides intuition of how C-Planning works as the distribution of waypoints shrinks from a uniform distribution to the single path connecting start state and goal state.
Revised: This set of experiments provides intuition of how C-Planning works as the distribution of waypoints shrinks from a uniform distribution to the single path connecting start state and goal state.

Original: This visualization is aligned with our theory (Eq. 2), which says that the distribution of waypoints should resemble the states visited by the optimal policy.
Revised: This visualization is aligned with our theory (Eq. 2), which says that the distribution of waypoints should resemble the states visited by the optimal policy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced C-Planning, a method for incorporating planning into the training of goal-conditioned agents.
Revised: In this paper, we introduced C-Planning, a method for incorporating planning into the training of goal-conditioned agents.

Original: The method enables the automatic generation of a curriculum over intermediate waypoints.
Revised: The method enables the automatic generation of a curriculum over intermediate waypoints.

Original: Unlike prior methods, our approach avoids the computational complexity of performing search at test time.
Revised: Unlike prior methods, our approach avoids the computational complexity of performing search at test time.

Original: C-Planning is simple to implement on top of existing goal-conditioned
Revised: C-Planning is simple to implement on top of existing goal-conditioned

Original: RL algorithms and achieves state-of-the-art results on a range of navigation and manipulation tasks.
Revised: RL algorithms and achieves state-of-the-art results on a range of navigation and manipulation tasks.

Original: Our work suggests a few directions for future work.
Revised: Our work suggests a few directions for future work.

Original: First, C-Planning samples waypoint that correspond to the optimal state distribution, which is proovably optimal in some settings [21].
Revised: First, C-Planning samples waypoint that correspond to the optimal state distribution, which is proovably optimal in some settings [21].

Original: How might similar ideas be applied to reward-driven tasks?
Revised: How might similar ideas be applied to reward-driven tasks?

Original: Second, while this paper studied sought to learn agents for reaching distant goals, we assumed that those distant goals were provided as part of the task deﬁnition.
Revised: Second, while this paper studied sought to learn agents for reaching distant goals, we assumed that those distant goals were provided as part of the task deﬁnition.

Original: Goal-conditioned algorithms that can propose their own distant goals, while already studied in some prior work [14, 34], remains an important direction for future work.
Revised: Goal-conditioned algorithms that can propose their own distant goals, while already studied in some prior work [14, 34], remains an important direction for future work.

Original: Ethics Statement.
Revised: Ethics Statement.

Original: Our paper focus on one of the core questions in RL, i e. can we design an search/planning algorithm to help goal-conditioned RL agent learn complex long-range behavior.
Revised: Our paper focus on one of the core questions in RL, i e. can we design an search/planning algorithm to help goal-conditioned RL agent learn complex long-range behavior.

Original: As a general algorithm for goal-conditioned RL, currently we cannot foresee any positive/negative social impact of our work.
Revised: As a general algorithm for goal-conditioned RL, currently we cannot foresee any positive/negative social impact of our work.

Original: As we conduct our experiments on standard environments like 2D navigation maze and robotic manipulation benchmarks (MetaWorld), our experiments should not suffer from any discrimination/bias/fairness concerns.
Revised: As we conduct our experiments on standard environments like 2D navigation maze and robotic manipulation benchmarks (MetaWorld), our experiments should not suffer from any discrimination/bias/fairness concerns.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: Our code is directly adapted from the open-sourced code base from [11] and we have introduced our implementation details in Appendix C.2 and Appendix C.
Revised: Our code is directly adapted from the open-sourced code base from [11] and we have introduced our implementation details in Appendix C.2 and Appendix C.

Original: We believe our results can be easily reproduced and our code is available at https://sites.
Revised: We believe our results can be easily reproduced and our code is available at https://sites.

Original: google.com/view/cplanning
Revised: google.com/view/cplanning

Revised: A P ROOFS AND A DDITIONAL A NALYSIS A. D ERIVING THE L OWER B OUND (P ROOF OF L EMMA 1)

Revised: We ﬁrst formulate the sampling procedure on starting states s 0 , waypoints s w , goals s g and the corresponding time horizon variable t 1 and t 2 .

Revised: Then we derive a lower bound on the target log density of Eq.

Revised: We then show that optimizing the lower bound through an EM procedure is equivalent to breaking the goal-reaching task into a sequence of easier sub-problems.

Revised: Finally, we wrapped up this section with a practical algorithm.

Revised: Data Generation Process.

Revised: The generative model for which inference corresponds to our planning procedure can be formulated as follows.

Revised: The episode starts by sampling an initial state s 0 ∼ p 0 ( s 0 ) .

Revised: Then it samples a geometric random variable t 1 ∼ Geom (1 − γ ) and roll out the policy π ( a | s, s g ) for exactly t 1 steps, starting from state s 0 .

Revised: We deﬁne s w to be the state where we end up (i.e., s w (cid:44) s t 1 ).

Revised: Thus, s w is sampled s w ∼ p π ( ·|· ,s g ) G EOM ( s t + | s 0 ) .

Revised: We then sample another geometric random variable t 2 ∼ Geom (1 − γ ) and roll out the policy π ( a | s, s g ) for exactly t 1 steps, starting from state s w .

Revised: We deﬁne s g to be the state where we end up (i.e., s g (cid:44) s t 1 + t 2 ).

Revised: Thus, s g is sampled s g ∼ p π ( ·|· ,s g )

Revised: Geom ( s t + | s w ) .

Revised: Note that the time index of the ﬁnal state s g is a sample from a negative binomial distribution: t 1 + t 2 d = NB ( p = 1 − γ, n = 2) .

Revised: We can equivalently express the sampling of s g as s g ∼ p π ( ·|· ,s g )

Revised: NegBinom ( s t + | s 0 ) .

Revised: Inference process.

Revised: Under the formulation of the data generation process above, we then aim to answer the following question in the inference procedure: what intermediate states would a policy visit if it eventually reached the goal state s g ?

Revised: Formally, we will estimate a distribution q ( s w | s 0 , s g )

Revised: ≈ p

Revised: ( s w | s 0 , s g ) .

Revised: We learn q ( s w | s 0 , s g ) by optimizing a evidence lower bound on our main objective (Eq. 2).

Revised: [Equation]

Revised: Note that s g is conditionally independent of s 0 given s w , so the p π ( s t + = s g | s w ) terms on the

Revised: RHS need not be conditioned on s 0 .

Revised: The evidence lower bound, L , depends on two quantities: the goal-conditioned policy and the distribution over waypoints.

Revised: The objective for the goal-conditioned policy is to maximize the probabilities of reaching the waypoint and reaching the ﬁnal state.

Revised: The objective for the waypoint distribution is to select waypoints s w that satisfy two important properties: the current policy should have a high probability of successfully navigating from the initial state to the waypoint and from the waypoint to the ﬁnal goal.

Revised: Note that the optimal choice for the waypoint distribution automatically depends on the current capabilities of the goal-conditioned policy.

Revised: Before optimizing the lower bound, we introduce a subtle modiﬁcation to the lower bound:

Revised: [Equation]

Revised: The difference, highlighted in orange, is that the probability of reaching the waypoint is computed for a goal-conditioned policy that is commanded to reach that waypoint, rather than the ﬁnal goal.

Revised: We show that this new objective is also an evidence lower bound on the same goal-reaching objective

Revised: (Eq. 2), but modiﬁed such that the sequence of commanded goals is treated as an additional latent variable.

Revised: Assume that the initial state s 0 and the goal state s g are given.

Revised: As before, we want to ﬁnd a policy that maximizes the probability of reaching s g .

Revised: However, we now consider jointly optimizing over both the policy and the sequence of goals we command for that policy.

Revised: We use s ( t ) c to denote the goal commanded at time t .

Revised: We can write this optimization problem as follows:

Revised: [Equation]

Revised: Applying Jensen’s inequality, we obtain a lower bound that looks similar to before:

Revised: [Equation]

Revised: In the second line, we introduce t w as the time when the waypoint is reached.

Revised: This allows us to clarify that the probability of reaching the waypoint only depends on the commanded goals through time t w , s (1: t w ) c .

Revised: This lower bound holds for any choice of the commanded waypoints, s (1: ∞ ) c .

Revised: Directly optimizing for the sequence of waypoints is challenging for two reasons.

Revised: First, it requires estimating the probability of commanding one goal but reaching any other state.

Revised: Second, if we command one goal when trying to reach a different goal, then the goal conditioned policy may not learn to associate the commanded goal with the desired outcome.

Revised: For both these reasons, we choose to not optimize the lower bound with respect to the commanded goals, but rather manually specify the commanded goals as follows:

Revised: [Equation]

Revised: Thus, we have recovered the objective in Eq.

Revised: As our lower bound holds for any choice of commanded waypoint, it also holds for this choice.

Revised: A. T HE O PTIMAL W AYPOINT D ISTRIBUTION (P ROOF OF L EMMA

Revised: This section proves Lemma 2.

Revised: Proof.

Revised: Recall that our goal is to solve the following maximization problem:

Revised: [Equation]

Revised: Note that the waypoint distribution must integrate to one.

Revised: The Lagrangian can be written as

Revised: [Equation]

Revised: (19) where λ is a Lagrange multiplier.

Revised: We then take the derivative with respect to q ( s w | s g , s 0 ) :

Revised: [Equation]

Revised: We then set this derivative equal to zero and solve for q ( s w | s g , s 0 ) :

Revised: [Equation]

Revised: S

Revised: S w1 S w

Revised: S w

Revised: S w

Revised: S w

Revised: S w

Revised: S w

Revised: S w

Revised: S w

Revised: S

Revised: S

Revised: S g

Revised: S g

Revised: S g (a) S w sampled at early stage of training (b) S w sampled at middle stage of training (c) S

Revised: w sampled at end stage of training

Revised: S

Revised: S g Vanilla training for a goalconditioned policy

Revised: Without Planning

Revised: With Planning !

Revised: (#&,#() Start : S

Revised: Goal : S g

Revised: Waypoints : S w : P(S w )

Revised: Finally, we determine the value of λ such that q ( s w | s 0 , s g ) integrates to one.

Revised: We can then express the optimal waypoint distribution as follows:

Revised: [Equation]

Revised: A. E STIMATING I MPORTANCE W EIGHTS (P ROOF OF L EMMA 3)

Revised: This section proves Lemma 3.

Revised: Proof.

Revised: Deﬁne the normalizing constant as follows

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: Substituting Z ( s 0 , s g ) into the RHS of Eq. 4 and simplifying the result, we show that it equals the

Revised: LHS of Eq. 4:

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: ) b ( s

Revised: [Equation]

Revised: B A DDITIONAL E XPERIMENTS B

Revised: A N O RACLE E XPERIMENT

Revised: [Figure]

Revised: [Figure]

Revised: We run an experiment to conﬁrm the wellknown result [21] that the optimal initial state distribution for learning a task is the state distribution of that task’s optimal policy.

Revised: Intuitively, if we could sample exactly from the state distribution of the optimal policy, then an RL agent could learn to reach goals more quickly.

Revised: This is not surprising as it resembles behavioral cloning of the optimal policy.

Revised: In this oracle experiment, it is assumed we are given the access to an optimal policy.

Revised: We achieve this by following the sampling procedure from Algorithm.

Revised: 1, but replace the classiﬁer C with that learned from the optimal policy C ∗ .

Revised: We then perform RL to reach waypoints and then the ﬁnal goal.

Revised: Results in Fig.

Revised: 9 show that using the optimal policy as the initial state distribution results in faster learning than using the original state distribution.

Revised: We also visualize the state density distribution of the optimal policy to provide more qualitative intuition on how the algorithm works.

Revised: Please refer to Appendix B.2 for more details.

Revised: In addition, we are also interested in measuring how well does planning through a learned model (CPlanning) performs if we don’t have access to the optimal generative model.

Revised: Results in Fig.

Revised: 9 also show that C-Planning achieves performance comparable to planning via optimal generative model in the Push and the Push-Wall environment.

Revised: B. V ISUALIZATION OF S TATE D ENSITY M AP OF O PTIMAL POLICY

Revised: We conduct this experiment on a 2D navigation task shown in Fig.

Revised: 9 (left), where we have also visualized the original initial state distribution, the state distribution of an optimal policy, and the goal state.

Revised: To conduct this experiment, we apply a state-of-the-art goal-conditioned RL algorithm (C-learning) in the two settings with different initial state distributions.

Revised: For fair evaluation, we evaluate the policies learned in both settings using the original initial state distribution.

Revised: The results shown in Fig.

Revised: 9 (right) show that starting from the optimal initial state distribution results in 2.4x faster learning.

Revised: B. A BLATIONS TO HER, S KEW F IT AND S O RB (C-P LANNING )

Revised: We perform additional experiments, comparing C-Planning with HER [2], SkewFit [34] and

Revised: SoRB [10] on C-Planning.

Revised: HER barely works on only simple experiments (e.g., Reach) but fails on harder ones.

Revised: Similar results are also been observed in the C-Learning [11] paper.

Revised: We implement

Revised: [Figure]

Revised: a version of SkewFit algorithm on top of our framework.

Revised: The only change it did is SkewFit samples swaypoints according to the inverse of a state density model.

Revised: We see that SkewFit has better performance in Reach task but fails in PushWall task.

Revised: We also compare C-Planning with SoRB(CPlanning) which does SoRB at test time.

Revised: Results show that SoRB(C-Planning) performs as well as

Revised: C-Planning at test time.

Revised: B. A BLATIONS ON P LANNING H ORIZON

Revised: We perform an ablation study on the planning horizon (number of waypoints used when doing sampling).

Revised: We implement this in a way of iterative planning: take horizon of 4 for example, we ﬁrst planning the middle point s w 2 via starting point and goal of ( s 0 , s g ) ; then planning s w 1 via starting point and goal ( s 0 , s w 2 ) .

Revised: Results show that planning for a more ﬁne-grained fashion (larger number of waypoints) is not always helping.

Revised: In the Fig.

Revised: 12 left, in Reach task, planning over more waypoints helps but it decreases the performance in Push task.

Revised: Note that here the number of waypoints is used for planning, while in Fig.

Revised: 5 the number of waypoints is how many waypoints to reach before changing to the ﬁnal goal.

Revised: These two are different quantities.

Revised: B. A DDITIONAL E XPERIMENTS ON A NT -M AZE

Revised: We performance an additional experiments in the Ant-Maze environment.

Revised: We design a very simple environment: command the ant to a speciﬁc goal position with a short (Ant-Small) or a long (AntMid) distance.

Revised: The distance between starting point and goal is 2.5 and 7.5 repectively for Ant-Small and Ant-Mid.

Revised: The only two modiﬁcations we did comparing to the original RIS [5] is change the maximum time horizon to 300 (RIS uses max time steps of 600) and reset the agent to a ﬁxed area (RIS resets the position of the agent uniformly).

Revised: In Fig.

Revised: 12 right, RIS achieves better performance in

Revised: Ant-Small environment while fails in Ant-Mid environment.

Revised: Note that C-Planning doesn’t use any termination function and reward function while RIS heavily relies on them.

Revised: 2.5 Environment Steps 1e

Revised: S u b s e t

Revised: M i n D

Revised: i s t a n c e

Revised: Ant-Mid

Revised: RIS Rand

Revised: RIS Rand 0.

Revised: RIS Fix

Revised: RIS Fix 0.

Revised: design factors that might affect the performance of the performance of RIS

Revised: We also study the performance of

Revised: RIS under these design choices.

Revised: We mainly concerned with two settings: random initialize the agent’s position versus ﬁxed initialize the agent’s position and choosing a termination threshold of 1.0 versus 0.

Revised: In

Revised: Fig.

Revised: 13, we can clearly see the results that using a slightly different threshold factor for termination will greatly affect the performance of RIS: using 1.0, the agent almost fails to learn any policy; while using 0.5, the agent is learning very fast.

Revised: This shows that RIS is very sensitive to the choice of this termination function.

Revised: It again proves the beneﬁt of

Revised: C-Planning since we don’t rely on any reward function and termination function.

Revised: C E XPERIMENTAL D ETAILS

Revised: In this section, We provide the essential hyperparameters for reproducing our experiments in this section.

Revised: We also introduce the hyperparameters used in baselines and provide a detailed description of environmental design.

Revised: C. I MPLEMENTATION D ETAILS

Revised: We introduce the hyperparameters used in C-Planning.

Revised: Note that in C-Learning, only a classiﬁer on ( s t , s t + , a t ) is needed.

Revised: In C-Planning, in order to sample waypoints from the distribution, an additional classiﬁer on ( s t , s t + ) is needed.

Revised: We also introduce two hyperparameters: Maximum Steps

Revised: Reaching Goal ( N g in Alg.1) forces the agent to change the intermediate goal if the original goal hasn’t been reached for some steps; Distance Threshold Reaching Goal ( (cid:15) d in Alg.1) for determing whether a goal is reached or not.

Revised: The rest are the standard hyperparameters for SAC algorithm

Revised: and we list here for reference.

Revised: [Table]

Revised: Note that we use a slightly different hyperparameters for 2D maze environment and we list below, all the other hyperparameters remains the same.

Revised: Note that we follow the goal relabeling changes by

Revised: C-Learning [11].

Revised: [Table]

Revised: C. E NVIRONMENTS

Revised: We follow the envioronment design of [11] with only one noticeable difference: in the original environments of C-Learning, for the ease of training, the author set the initial position of objects to be relatively near the arm so the arm can easily push the object, getting a better learning signal.

Revised: We intentionally set the initial state of object to be far away from the arm.

Revised: This signiﬁcantly increase the diffuculty of learning.

Revised: We’ll release these environment with the code.

Revised: [Table]

Revised: C. B ASELINES

Revised: We also provide the hyperparameters associated with the baselines.

Revised: The two baselines we care about:

Revised: C-Learning and RIS.

Revised: We summarize their hyperparameters in the table:

Revised: [Table]

Revised: D V ISUALIZATION OF THE L EARNED P OLICY

Revised: In order to more intuitively visualize the behavior of the learned policy and emphasize the importance of the difﬁculty of the learning task

Revised: [Table]

Revised: we plot the visualization of our learned policy for several snap shots in an episode for environment of Push-Two and Obstacle-Drawer-Open .

Revised: We see that our method successfully guide the agent to learn the behavior of push the green object ﬁrst, the push the object; And ﬁrst open the drawer then push the object to the desired location.

Revised: We

Revised: would

Revised: [Figure]

Revised: Our method successfully trains the agent to ﬁrst close the drawer and then push the object to the desired location.

Revised: We emphasize that such behavior is hard to obtain and most of the prior methods only manage to close the drawer.