Original: L EARNING TO R EACH G OALS VIA I TERATED S UPERVISED L EARNING
Revised: L EARNING TO R EACH G OALS VIA I TERATED S UPERVISED L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current reinforcement learning (RL) algorithms can be brittle and difﬁcult to use, especially when learning goal-reaching behaviors from sparse rewards.
Revised: Current reinforcement learning (RL) algorithms can be brittle and difﬁcult to use, especially when learning goal-reaching behaviors from sparse rewards.

Original: Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor.
Revised: Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor.

Original: In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function.
Revised: In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function.

Original: In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the ﬁnal state in that same trajectory.
Revised: In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the ﬁnal state in that same trajectory.

Original: We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch.
Revised: We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch.

Original: Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy.
Revised: Each iteration, the agent collects new trajectories using the latest policy and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy.

Original: We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks.
Revised: We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) provides an elegant framework for agents to learn general-purpose behaviors supervised by only a reward signal.
Revised: Reinforcement learning (RL) provides an elegant framework for agents to learn general-purpose behaviors supervised by only a reward signal.

Original: When combined with neural networks, RL has enabled many notable successes, but our most successful deep RL algorithms are far from a turnkey solution.
Revised: When combined with neural networks, RL has enabled many notable successes, but our most successful deep RL algorithms are far from a turnkey solution.

Original: Despite striving for data efﬁciency, RL algorithms, especially those using temporal difference learning, are highly sensitive to hyperparameters (Henderson et al., 2018) and face challenges of stability and optimization (Tsitsiklis & Van Roy, 1997; van Hasselt et al., 2018; Kumar et al., 2019b), making such algorithms difﬁcult to use in practice.
Revised: Despite striving for data efﬁciency, RL algorithms, especially those using temporal difference learning, are highly sensitive to hyperparameters (Henderson et al., 2018) and face challenges of stability and optimization (Tsitsiklis & Van Roy, 1997; van Hasselt et al., 2018; Kumar et al., 2019b), making such algorithms difﬁcult to use in practice.

Original: If agents are supervised not with a reward signal, but rather demonstrations from an expert, the resulting class of algorithms is signiﬁcantly more stable and easy to use.
Revised: If agents are supervised not with a reward signal, but rather demonstrations from an expert, the resulting class of algorithms is signiﬁcantly more stable and easy to use.

Original: Imitation learning via behavioral cloning provides a simple paradigm for training control policies: maximizing the likelihood of optimal actions via supervised learning.
Revised: Imitation learning via behavioral cloning provides a simple paradigm for training control policies: maximizing the likelihood of optimal actions via supervised learning.

Original: Imitation learning algorithms using deep learning are mature and robust; these algorithms have demonstrated success in acquiring behaviors reliably from high-dimensional sensory data such as images (Bojarski et al., 2016; Lynch et al., 2019).
Revised: Imitation learning algorithms using deep learning are mature and robust; these algorithms have demonstrated success in acquiring behaviors reliably from high-dimensional sensory data such as images (Bojarski et al., 2016; Lynch et al., 2019).

Original: Although imitation learning via supervised learning is not a replacement for RL – the paradigm is limited by the difﬁculty of obtaining kinesthetic demonstrations from a supervisor – the idea of learning policies via supervised learning can serve as inspiration for RL agents that learn behaviors from scratch.
Revised: Although imitation learning via supervised learning is not a replacement for RL – the paradigm is limited by the difﬁculty of obtaining kinesthetic demonstrations from a supervisor – the idea of learning policies via supervised learning can serve as inspiration for RL agents that learn behaviors from scratch.

Original: In this paper, we present a simple RL algorithm for learning goal-directed policies that leverages the stability of supervised imitation learning without requiring an expert supervisor.
Revised: In this paper, we present a simple RL algorithm for learning goal-directed policies that leverages the stability of supervised imitation learning without requiring an expert supervisor.

Original: We show that when learning goal-directed behaviors using RL, demonstrations of optimal behavior can be generated from sub-optimal data in a fully self-supervised manner using the principle of data relabeling: that every trajectory is a successful demonstration for the state that it actually reaches, even if it is sub-optimal for the goal that was originally commanded to generate the trajectory.
Revised: We show that when learning goal-directed behaviors using RL, demonstrations of optimal behavior can be generated from sub-optimal data in a fully self-supervised manner using the principle of data relabeling: that every trajectory is a successful demonstration for the state that it actually reaches, even if it is sub-optimal for the goal that was originally commanded to generate the trajectory.

Original: A similar observation of hindsight relabelling was originally made by Kaelbling (1993), more recently popularized in the deep RL literature (Andrychowicz et al., 2017), for learning with off-policy value-based methods and policy-gradient methods (Rauber et al., 2017).
Revised: A similar observation of hindsight relabelling was originally made by Kaelbling (1993), more recently popularized in the deep RL literature (Andrychowicz et al., 2017), for learning with off-policy value-based methods and policy-gradient methods (Rauber et al., 2017).

Original: When goal-relabelling, these algorithms recompute the received rewards as though a different goal had been commanded.
Revised: When goal-relabelling, these algorithms recompute the received rewards as though a different goal had been commanded.

Original: In this work, we instead notice that goal-relabelling to the ﬁnal state in the trajectory allows an algorithm to re-interpret an action collected by a sub-optimal agent as though it were collected by an expert agent, just for a different goal.
Revised: In this work, we instead notice that goal-relabelling to the ﬁnal state in the trajectory allows an algorithm to re-interpret an action collected by a sub-optimal agent as though it were collected by an expert agent, just for a different goal.

Original: This leads to a substantially simpler algorithm that relies only on a supervised imitation learning primitive, avoiding the challenges of value function estimation.
Revised: This leads to a substantially simpler algorithm that relies only on a supervised imitation learning primitive, avoiding the challenges of value function estimation.

Original: By generating demonstrations using hindsight relabelling, we are able to apply goal-conditioned imitation learning primitives (Gupta et al., 2019;
Revised: By generating demonstrations using hindsight relabelling, we are able to apply goal-conditioned imitation learning primitives (Gupta et al., 2019;

Original: Ding et al., 2019) on data collected by sub-optimal agents, not just from an expert supervisor.
Revised: Ding et al., 2019) on data collected by sub-optimal agents, not just from an expert supervisor.

Original: We instantiate these ideas as an algorithm that we call goal-conditioned supervised learning (GCSL).
Revised: We instantiate these ideas as an algorithm that we call goal-conditioned supervised learning (GCSL).

Original: At each iteration, trajectories are collected commanding the current goal-conditioned policy for some set of desired goals, and then relabeled using hindsight to be optimal for the set of goals that were actually reached.
Revised: At each iteration, trajectories are collected commanding the current goal-conditioned policy for some set of desired goals, and then relabeled using hindsight to be optimal for the set of goals that were actually reached.

Original: Supervised imitation learning with this generated “expert” data is used to train an improved goal-conditioned policy for the next iteration.
Revised: Supervised imitation learning with this generated “expert” data is used to train an improved goal-conditioned policy for the next iteration.

Original: Interestingly, this simple procedure provably optimizes a lower bound on a well-deﬁned RL objective; by performing self-imitation on all of its own trajectories, an agent can iteratively improve its own policy to learn optimal goal-reaching behaviors without requiring any external demonstrations and without learning a value function.
Revised: Interestingly, this simple procedure provably optimizes a lower bound on a well-deﬁned RL objective; by performing self-imitation on all of its own trajectories, an agent can iteratively improve its own policy to learn optimal goal-reaching behaviors without requiring any external demonstrations and without learning a value function.

Original: While self-imitation RL algorithms typically choose a small subset of trajectories to imitate (Oh et al., 2018;
Revised: While self-imitation RL algorithms typically choose a small subset of trajectories to imitate (Oh et al., 2018;

Original: Hao et al., 2019) or learn a separate value function to reweight past experience (Neumann & Peters, 2009; Abdolmaleki et al., 2018; Peng et al., 2019), we show that GCSL learns efﬁciently while training on every previous trajectory without reweighting, thereby maximizing data reuse.
Revised: Hao et al., 2019) or learn a separate value function to reweight past experience (Neumann & Peters, 2009; Abdolmaleki et al., 2018; Peng et al., 2019), we show that GCSL learns efﬁciently while training on every previous trajectory without reweighting, thereby maximizing data reuse.

Original: The main contribution of our work is GCSL, a simple goal-reaching RL algorithm that uses supervised learning to acquire policies from scratch.
Revised: The main contribution of our work is GCSL, a simple goal-reaching RL algorithm that uses supervised learning to acquire policies from scratch.

Original: We show, both formally and empirically, that any trajectory taken by the agent can be turned into an optimal one using hindsight relabelling, and that imitation of these trajectories (provably) enables an agent to (iteratively) learn goal-reaching behaviors.
Revised: We show, both formally and empirically, that any trajectory taken by the agent can be turned into an optimal one using hindsight relabelling, and that imitation of these trajectories (provably) enables an agent to (iteratively) learn goal-reaching behaviors.

Original: That iteratively imitating all the data from a sub-optimal agent leads to optimal behavior is a non-trivial conclusion; we formally verify that the procedure optimizes a lower-bound on a goal-reaching RL objective and derive performance bounds when the supervised learning objective is sufﬁciently minimized.
Revised: That iteratively imitating all the data from a sub-optimal agent leads to optimal behavior is a non-trivial conclusion; we formally verify that the procedure optimizes a lower-bound on a goal-reaching RL objective and derive performance bounds when the supervised learning objective is sufﬁciently minimized.

Original: In practice, GCSL is simpler, more stable, and less sensitive to hyperparameters than value-based methods, while still retaining the beneﬁts of off-policy learning.
Revised: In practice, GCSL is simpler, more stable, and less sensitive to hyperparameters than value-based methods, while still retaining the beneﬁts of off-policy learning.

Original: Moreover, GCSL can leverage demonstrations (if available) to accelerate learning.
Revised: Moreover, GCSL can leverage demonstrations (if available) to accelerate learning.

Original: We demonstrate that GCSL outperforms value-based and policy gradient methods on several challenging robotic domains.
Revised: We demonstrate that GCSL outperforms value-based and policy gradient methods on several challenging robotic domains.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Goal reaching.
Revised: Goal reaching.

Original: The goal reaching problem is characterized by the tuple (cid:104)S , A , T , ρ ( s 0 ) , T, p ( g ) (cid:105) , where S and A are the state and action spaces, T ( s (cid:48) | s, a ) is the transition function, ρ ( s 0 ) is the initial state distribution, T the horizon length, and p ( g ) is the distribution over goal states g ∈ S .
Revised: The goal reaching problem is characterized by the tuple (cid:104)S , A , T , ρ ( s 0 ) , T, p ( g ) (cid:105) , where S and A are the state and action spaces, T ( s (cid:48) | s, a ) is the transition function, ρ ( s 0 ) is the initial state distribution, T the horizon length, and p ( g ) is the distribution over goal states g ∈ S .

Original: We aim to ﬁnd a time-varying goal-conditioned policy π ( ·| s, g, h ) :
Revised: We aim to ﬁnd a time-varying goal-conditioned policy π ( ·| s, g, h ) :

Original: S × S ×
Revised: S × S ×

Original: [ T ] → ∆( A ) , where ∆( A ) is the probability simplex over the action space A and h is the remaining horizon.
Revised: [ T ] → ∆( A ) , where ∆( A ) is the probability simplex over the action space A and h is the remaining horizon.

Original: We will say that a goal is achieved if the agent has reached the goal at the end of the episode.
Revised: We will say that a goal is achieved if the agent has reached the goal at the end of the episode.

Original: Correspondingly, the learning problem is to acquire a policy that maximizes the probability of achieving the desired goal:
Revised: Correspondingly, the learning problem is to acquire a policy that maximizes the probability of achieving the desired goal:

Original: [Equation]
Revised: [Equation]

Original: Notice that unlike a shortest-path objective, this ﬁnal-timestep objective provides no incentive to ﬁnd the shortest path to the goal.
Revised: Notice that unlike a shortest-path objective, this ﬁnal-timestep objective provides no incentive to ﬁnd the shortest path to the goal.

Original: We shall see in Section 3 that this notion of optimality is more than a simple design choice: hindsight relabeling for optimality emerges naturally when maximizing the probability of achieving the goal, but does not when minimizing the time to reach the goal.
Revised: We shall see in Section 3 that this notion of optimality is more than a simple design choice: hindsight relabeling for optimality emerges naturally when maximizing the probability of achieving the goal, but does not when minimizing the time to reach the goal.

Original: The ﬁnal timestep objective is especially useful in practical applications where reaching a particular goal is challenging, but once a goal is reached, it is possible to remain at the goal.
Revised: The ﬁnal timestep objective is especially useful in practical applications where reaching a particular goal is challenging, but once a goal is reached, it is possible to remain at the goal.

Original: When reaching the goal is itself challenging, forcing the agent to reach the goal as fast as possible can make the learning problem unduly difﬁcult.
Revised: When reaching the goal is itself challenging, forcing the agent to reach the goal as fast as possible can make the learning problem unduly difﬁcult.

Original: In contrast, this objective just requires the agent to eventually reach, and then stay at the goal, a more straightforward learning problem.
Revised: In contrast, this objective just requires the agent to eventually reach, and then stay at the goal, a more straightforward learning problem.

Original: In addition, the ﬁnal timestep objective is useful when trying to learn robust solutions that potentially take longer over shorter solutions that have a lower probability of success.
Revised: In addition, the ﬁnal timestep objective is useful when trying to learn robust solutions that potentially take longer over shorter solutions that have a lower probability of success.

Original: There are certain drawbacks to this formulation: it is not as appropriate when the environment stochasticity prevents the agent from remaining at a goal, and it requires knowing an upper bound on the time it takes to reach a goal.
Revised: There are certain drawbacks to this formulation: it is not as appropriate when the environment stochasticity prevents the agent from remaining at a goal, and it requires knowing an upper bound on the time it takes to reach a goal.

Original: Goal-conditioned RL.
Revised: Goal-conditioned RL.

Original: The goal reaching problem can be equivalently deﬁned using the nomenclature of RL as a collection of Markov decision processes (MDPs) {M g } g ∈S .
Revised: The goal reaching problem can be equivalently deﬁned using the nomenclature of RL as a collection of Markov decision processes (MDPs) {M g } g ∈S .

Original: Each MDP M g is deﬁned as the tuple (cid:104)S , A , T g , r g , ρ, T (cid:105) , where the state space, action space, initial state distribution, and horizon as above.
Revised: Each MDP M g is deﬁned as the tuple (cid:104)S , A , T g , r g , ρ, T (cid:105) , where the state space, action space, initial state distribution, and horizon as above.

Original: For each goal, a reward function is deﬁned as r g ( s ) =
Revised: For each goal, a reward function is deﬁned as r g ( s ) =

Original: 1 ( s = g ) .
Revised: 1 ( s = g ) .

Original: Using this notation, an optimal goal-conditioned policy maximizes the return in an MDP M g sampled according to the goal distribution,
Revised: Using this notation, an optimal goal-conditioned policy maximizes the return in an MDP M g sampled according to the goal distribution,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Since the transition dynamics are equivalent for different goals, off-policy value-based methods can use transitions collected for one goal to compute the value function for arbitrary other goals.
Revised: Since the transition dynamics are equivalent for different goals, off-policy value-based methods can use transitions collected for one goal to compute the value function for arbitrary other goals.

Original: Namely, Kaelbling (1993)
Revised: Namely, Kaelbling (1993)

Original: ﬁrst showed that if the transition ( s, a, s (cid:48) , r ) was witnessed when reaching a speciﬁc goal g , it can be relabeled to ( s, a, s (cid:48) , r g (cid:48) ( s )) for an arbitrary goal g (cid:48) ∈ S if the underlying goal reward function is known.
Revised: ﬁrst showed that if the transition ( s, a, s (cid:48) , r ) was witnessed when reaching a speciﬁc goal g , it can be relabeled to ( s, a, s (cid:48) , r g (cid:48) ( s )) for an arbitrary goal g (cid:48) ∈ S if the underlying goal reward function is known.

Original: Hindsight experience replay (Andrychowicz et al., 2017) considers a speciﬁc case of relabeling to when the relabeled goal is another state further down the trajectory.
Revised: Hindsight experience replay (Andrychowicz et al., 2017) considers a speciﬁc case of relabeling to when the relabeled goal is another state further down the trajectory.

Original: Goal-conditioned imitation learning.
Revised: Goal- [Table] imitation learning.

Original: If an agent is additionally provided expert demonstrations for reaching particular goals, behavioral cloning is a simple algorithm to learn the optimal policy by maximizing the likelihood of the demonstration data under the policy.
Revised: If an agent is additionally provided expert demonstrations for reaching particular goals, behavioral cloning is a simple algorithm to learn the optimal policy by maximizing the likelihood of the demonstration data under the policy.

Original: Formally, demonstrations are provided as a dataset of expert behavior D ∗ = { τ 1 , τ 2 , . . .
Revised: Formally, demonstrations are provided as a dataset of expert behavior D ∗ = { τ 1 , τ 2 , . . .

Original: } from an expert policy π ∗ , where each trajectory τ i = { s i 0 , a i 0 , s i 1 , a i 1 , ....
Revised: } from an expert policy π ∗ , where each trajectory τ i = { s i 0 , a i 0 , s i 1 , a i 1 , ....

Original: s 1 T } is optimal for reaching the ﬁnal state in the trajectory.
Revised: s 1 T } is optimal for reaching the ﬁnal state in the trajectory.

Original: Given a parametric class of stochastic, time-varying policies Π , the behavioral cloning objective is to maximize the likelihood of actions seen in the data when attempting to reach this desired goal,
Revised: Given a parametric class of stochastic, time-varying policies Π , the behavioral cloning objective is to maximize the likelihood of actions seen in the data when attempting to reach this desired goal,

Original: [Equation]
Revised: [Equation]

Original:  L EARNING G OAL -C ONDITIONED P OLICIES WITH S ELF -I MITATION
Revised:  L EARNING G OAL -C ONDITIONED P OLICIES WITH S ELF -I MITATION

Original: In this section, we show how imitation learning via behavior cloning with data relabeling can be utilized in an iterative procedure that optimizes a lower bound on the RL objective.
Revised: In this section, we show how imitation learning via behavior cloning with data relabeling can be utilized in an iterative procedure that optimizes a lower bound on the RL objective.

Original: The resulting procedure, in which an agent continually relabels and imitates its own experience, is not an imitation learning algorithm, but rather an RL algorithm for learning goal-reaching from scratch without any expert demonstrations.
Revised: The resulting procedure, in which an agent continually relabels and imitates its own experience, is not an imitation learning algorithm, but rather an RL algorithm for learning goal-reaching from scratch without any expert demonstrations.

Original: This algorithm, illustrated in Fig.
Revised: This algorithm, illustrated in Fig.

Original: 1, is simple and allows us to perform off-policy reinforcement learning for goal reaching without learning value functions.
Revised: 1, is simple and allows us to perform off-policy reinforcement learning for goal reaching without learning value functions.

Original: 3. G OAL -C ONDITIONED S UPERVISED L EARNING
Revised: 3. G OAL -C ONDITIONED S UPERVISED L EARNING

Original: We can attain the beneﬁts of behavioral cloning without the dependence on human supervision by leveraging the following insight: under last-timestep optimality (Equation 1), a trajectory that fails to reach the intended goal is nonetheless optimal for reaching the goal it actually reached.
Revised: We can attain the beneﬁts of behavioral cloning without the dependence on human supervision by leveraging the following insight: under last-timestep optimality (Equation 1), a trajectory that fails to reach the intended goal is nonetheless optimal for reaching the goal it actually reached.

Original: As a result, a trajectory from a sub-optimal agent can be re-interpreted by goal-conditioned behavior cloning as an optimal trajectory for reaching a potentially different goal.
Revised: As a result, a trajectory from a sub-optimal agent can be re-interpreted by goal-conditioned behavior cloning as an optimal trajectory for reaching a potentially different goal.

Original: This insight will allow us to convert sub-optimal trajectories into optimal goal reaching trajectories for different goals, without the need for any human supervision.
Revised: This insight will allow us to convert sub-optimal trajectories into optimal goal reaching trajectories for different goals, without the need for any human supervision.

Original: More precisely, consider a trajectory τ = { s 1 , a 1 , s 2 , a 2 , . .
Revised: More precisely, consider a trajectory τ = { s 1 , a 1 , s 2 , a 2 , . .

Original: , s T , a T } obtained by commanding the policy π θ ( a | s, g, h ) to reach some goal g .
Revised: , s T , a T } obtained by commanding the policy π θ ( a | s, g, h ) to reach some goal g .

Original: For any time step t and horizon h , the action a t in state s t is likely to be a good action for reaching s t
Revised: For any time step t and horizon h , the action a t in state s t is likely to be a good action for reaching s t

Original: + h in h time steps (even if it is not a good action for reaching the originally commanded goal g ), and thus can be treated as expert supervision for π θ ( · | s t , s t + h , h ) .
Revised: + h in h time steps (even if it is not a good action for reaching the originally commanded goal g ), and thus can be treated as expert supervision for π θ ( · | s t , s t + h , h ) .

Original: To obtain a concrete algorithm, we can relabel all time steps and horizons in a trajectory to create an expert dataset according to
Revised: To obtain a concrete algorithm, we can relabel all time steps and horizons in a trajectory to create an expert dataset according to

Original: [Equation]
Revised: [Equation]

Original:  with states s t , corresponding actions a t , the corresponding goal set to future state s t + h and matching horizon h .
Revised: ) with states s t , corresponding actions a t , the corresponding goal set to future state s t + h and matching horizon h .

Original: Because the relabeling procedure is valid for any horizon, we can use any valid combination of ( s t , a t , s t + h , h ) tuples as supervision, for a total of (cid:0) T 2 (cid:1) optimal datapoints of ( s, a, g, h ) from a single trajectory.
Revised: Because the relabeling procedure is valid for any horizon, we can use any valid combination of ( s t , a t , s t + h , h ) tuples as supervision, for a total of (cid:0) T 2 (cid:1) optimal datapoints of ( s, a, g, h ) from a single trajectory.

Original: This idea is related to data-relabeling for estimating the value function (Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2017), but our work shows that data-relabelling can also be used to re-interpret data from a sub-optimal agent as though the data came from an optimal agent (with a different goal).
Revised: This idea is related to data-relabeling for estimating the value function (Kaelbling, 1993; Andrychowicz et al., 2017; Rauber et al., 2017), but our work shows that data-relabelling can also be used to re-interpret data from a sub-optimal agent as though the data came from an optimal agent (with a different goal).

Original: We then use this relabeled dataset for goal-conditioned behavior cloning.
Revised: We then use this relabeled dataset for goal-conditioned behavior cloning.

Original: Algorithm 1 summarizes the approach: (1) Sample a goal from a target goal distribution p ( g ) .
Revised: Algorithm 1 summarizes the approach: (1) Sample a goal from a target goal distribution p ( g ) .

Original: (2) Execute the current policy π ( a | s, g, h ) for T steps in the environment to collect a potentially suboptimal trajectory τ .
Revised: (2) Execute the current policy π ( a | s, g, h ) for T steps in the environment to collect a potentially suboptimal trajectory τ .

Original: (3) Relabel the trajectory (Equation. 3) to add (cid:0) T 2 (cid:1) new expert tuples ( s t , a t , s t
Revised: (3) Relabel the trajectory (Equation. 3) to add (cid:0) T 2 (cid:1) new expert tuples ( s t , a t , s t

Original: + h , h ) to the training dataset.
Revised: + h , h ) to the training dataset.

Original: (4) Perform supervised learning on the entire dataset to update the policy π ( a | s, g, h ) via maximum likelihood.
Revised: (4) Perform supervised learning on the entire dataset to update the policy π ( a | s, g, h ) via maximum likelihood.

Original: We term this iterative procedure of sampling trajectories, relabeling them, and training a policy until convergence goal-conditioned supervised learning (GCSL).
Revised: We term this iterative procedure of sampling trajectories, relabeling them, and training a policy until convergence goal-conditioned supervised learning (GCSL).

Original: This algorithm can use all of the prior off-policy data in the training dataset because this data continues to remain optimal under the notion of goal-reaching optimality that was deﬁned in Section 2, but does not require any explicit value function learning.
Revised: This algorithm can use all of the prior off-policy data in the training dataset because this data continues to remain optimal under the notion of goal-reaching optimality that was deﬁned in Section 2, but does not require any explicit value function learning.

Original: Perhaps surprisingly, this procedure optimizes a lower bound on an
Revised: Perhaps surprisingly, this procedure optimizes a lower bound on an

Original: RL objective, as we will show in Section 3.2.
Revised: RL objective, as we will show in Section 3.2.

Original: The GCSL algorithm (as described above) can learn to reach goals from the target distribution p ( g ) simply using iterated behavioral cloning.
Revised: The GCSL algorithm (as described above) can learn to reach goals from the target distribution p ( g ) simply using iterated behavioral cloning.

Original: This goal reaching algorithm is off-policy, optimizes a simple supervised learning objective, and is easy to implement and tune without the need for any explicit reward function engineering or demonstrations.
Revised: This goal reaching algorithm is off-policy, optimizes a simple supervised learning objective, and is easy to implement and tune without the need for any explicit reward function engineering or demonstrations.

Original: Additionally, since GCSL uses a goalconditioned imitation learning algorithm as a sub procedure, if demonstrations or off-policy data are available, it is easier to incorporate this data into training than with off-policy value function methods.
Revised: Additionally, since GCSL uses a goalconditioned imitation learning algorithm as a sub procedure, if demonstrations or off-policy data are available, it is easier to incorporate this data into training than with off-policy value function methods.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: We now formally analyze GCSL to verify that it solves the goal-reaching problem, quantify how errors in approximation of the objective manifest in goal-reaching performance, and understand how it relates to existing RL algorithms.
Revised: We now formally analyze GCSL to verify that it solves the goal-reaching problem, quantify how errors in approximation of the objective manifest in goal-reaching performance, and understand how it relates to existing RL algorithms.

Original: Speciﬁcally, we derive the algorithm as the optimization of a lower bound of the true goal-reaching objective, and we show that under certain conditions on the environment, minimizing the GCSL objective enables performance guarantees on the learned policy.
Revised: Speciﬁcally, we derive the algorithm as the optimization of a lower bound of the true goal-reaching objective, and we show that under certain conditions on the environment, minimizing the GCSL objective enables performance guarantees on the learned policy.

Original: We start by describing the objective function being optimized by GCSL.
Revised: We start by describing the objective function being optimized by GCSL.

Original: For ease of presentation, we make the simplifying assumption that the trajectories are collected from a single policy π old , and that relabelling is only done with goals at the last timestep ( g = s T ).
Revised: For ease of presentation, we make the simplifying assumption that the trajectories are collected from a single policy π old , and that relabelling is only done with goals at the last timestep ( g = s T ).

Original: GCSL performs goal-conditioned behavioral cloning on a distribution of trajectories
Revised: GCSL performs goal-conditioned behavioral cloning on a distribution of trajectories

Original: π old ( τ ) =
Revised: π old ( τ ) =

Original: E g ∼ p
Revised: E g ∼ p

Original: ( g ) [ π old ( τ | g )] , resulting in the following objective:
Revised: ( g ) [ π old ( τ | g )] , resulting in the following objective:

Original: [Equation]
Revised: [Equation]

Original: Our main result shows that, under certain assumptions about the off-policy data distribution, optimizing the GCSL objective J GCSL ( π ) optimizes a lower bound on the desired objective, J ( π ) .
Revised: Our main result shows that, under certain assumptions about the off-policy data distribution, optimizing the GCSL objective J GCSL ( π ) optimizes a lower bound on the desired objective, J ( π ) .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let J GCSL and J be as deﬁned above.
Revised: Let J GCSL and J be as deﬁned above.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Where α = max s,g,h D TV ( π ( ·| s, g, h ) (cid:107) π old ( ·| s, g, h )) and C is a constant independent of π .
Revised: Where α = max s,g,h D TV ( π ( ·| s, g, h ) (cid:107) π old ( ·| s, g, h )) and C is a constant independent of π .

Original: The proof is in Appendix B.1.
Revised: The proof is in Appendix B.1.

Original: This theorem provides a lower-bound on the goal-reaching objective with equality for the optimal policy; akin to many proofs for direct policy search methods, the strongest guarantees are provided under on-policy data collection ( α = 0 ).
Revised: This theorem provides a lower-bound on the goal-reaching objective with equality for the optimal policy; akin to many proofs for direct policy search methods, the strongest guarantees are provided under on-policy data collection ( α = 0 ).

Original: The analysis raises two questions: can we quantify the tightness of the bound given by Theorem 3.1, and what does an optimal solution to the GCSL objective imply about performance on the true objective?
Revised: The analysis raises two questions: can we quantify the tightness of the bound given by Theorem 3.1, and what does an optimal solution to the GCSL objective imply about performance on the true objective?

Original: The tightness of the bound depends on two choices in the algorithm: how off-policy data from π old is used to optimize the objective, and how the relabeling step adjusts the exact distribution of data being trained on.
Revised: The tightness of the bound depends on two choices in the algorithm: how off-policy data from π old is used to optimize the objective, and how the relabeling step adjusts the exact distribution of data being trained on.

Original: We ﬁnd that the looseness induced by the relabeling can be controlled by two factors: 1) the proportion of data that must be relabeled, and 2) the distance between the distribution of trajectories that needed to be relabeled and the distribution of trajectories that achieved the desired goal and were not relabeled.
Revised: We ﬁnd that the looseness induced by the relabeling can be controlled by two factors: 1) the proportion of data that must be relabeled, and 2) the distance between the distribution of trajectories that needed to be relabeled and the distribution of trajectories that achieved the desired goal and were not relabeled.

Original: If either of these quantities is minimized to zero, the looseness of the bound that stems from relabeling also goes to zero.
Revised: If either of these quantities is minimized to zero, the looseness of the bound that stems from relabeling also goes to zero.

Original: We present this analysis formally in Appendix B.2.
Revised: We present this analysis formally in Appendix B.2.

Original: Even when data is collected from an off-policy distribution, optimizing the GCSL objective over the full state space can provide guarantees on the performance of the learned policy.
Revised: Even when data is collected from an off-policy distribution, optimizing the GCSL objective over the full state space can provide guarantees on the performance of the learned policy.

Original: We write π ∗ to denote a policy that maximizes the true performance J ( π ) , and ˜ π ∗ to denote the policy that maximizes the GCSL objective J GCSL ( π ) over the set of all policies.
Revised: We write π ∗ to denote a policy that maximizes the true performance J ( π ) , and ˜ π ∗ to denote the policy that maximizes the GCSL objective J GCSL ( π ) over the set of all policies.

Original: The following theorem provides such a performance guarantee for deterministic environments (proof in Appendix B.3):
Revised: The following theorem provides such a performance guarantee for deterministic environments (proof in Appendix B.3):

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Consider an environment with deterministic dynamics and a data-collection policy π old with full support.
Revised: Consider an environment with deterministic dynamics and a data-collection policy π old with full support.

Original: If max s,g,h D TV ( π ( a | s, g, h ) , ˜ π ∗ ( a | s, g, h )) ≤ (cid:15) , then J ( π ∗ ) − J ( π ) < (cid:15)T .
Revised: If max s,g,h D TV ( π ( a | s, g, h ) , ˜ π ∗ ( a | s, g, h )) ≤ (cid:15) , then J ( π ∗ ) − J ( π ) < (cid:15)T .

Original: This theorem states that in an environment with deterministic transitions, the policy that maximizes the GCSL objective J GCSL ( π ) also maximizes the true performance J ( π ) .
Revised: This theorem states that in an environment with deterministic transitions, the policy that maximizes the GCSL objective J GCSL ( π ) also maximizes the true performance J ( π ) .

Original: Furthermore, if the GCSL loss is approximately minimized, then performance guarantees can be given as a function of the error across the full state space.
Revised: Furthermore, if the GCSL loss is approximately minimized, then performance guarantees can be given as a function of the error across the full state space.

Original: Whereas Theorem 3.1 shows that GCSL always optimizes a lower bound on the RL objective when iteratively re-collecting data with the updated policy, Theorem 3. shows that in certain environments, simply optimizing the GCSL objective from any off-policy data distribution without iterative data collection can also lead to convergence.
Revised: Whereas Theorem 3.1 shows that GCSL always optimizes a lower bound on the RL objective when iteratively re-collecting data with the updated policy, Theorem 3. shows that in certain environments, simply optimizing the GCSL objective from any off-policy data distribution without iterative data collection can also lead to convergence.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work studies the problem of goal-conditioned RL (Kaelbling, 1993) from sparse goal-reaching rewards.
Revised: Our work studies the problem of goal-conditioned RL (Kaelbling, 1993) from sparse goal-reaching rewards.

Original: To maximize data-efﬁciency in the presence of sparse rewards, value function methods use off-policy hindsight relabeling methods such as hindsight experience replay (Andrychowicz et al., 2017) to relabel rewards and transitions retroactively (Schaul et al., 2015; Pong et al., 2018).
Revised: To maximize data-efﬁciency in the presence of sparse rewards, value function methods use off-policy hindsight relabeling methods such as hindsight experience replay (Andrychowicz et al., 2017) to relabel rewards and transitions retroactively (Schaul et al., 2015; Pong et al., 2018).

Original: Despite the potential for learning with hindsight, optimization of goal-conditioned value functions suffers from instability due to challenging critic estimation.
Revised: Despite the potential for learning with hindsight, optimization of goal-conditioned value functions suffers from instability due to challenging critic estimation.

Original: Rauber et al.
Revised: Rauber et al.

Original: (2017) extends hindsight relabelling to policy gradient methods, but is hampered by high-variance importance weights that emerge from relabelling.
Revised: (2017) extends hindsight relabelling to policy gradient methods, but is hampered by high-variance importance weights that emerge from relabelling.

Original: Our method also relabels trajectories in hindsight, but does so in a completely different way: to supervise an imitation learning primitive to learn the optimal policy.
Revised: Our method also relabels trajectories in hindsight, but does so in a completely different way: to supervise an imitation learning primitive to learn the optimal policy.

Original: Unlike these methods,
Revised: Unlike these methods,

Original: GCSL does not maintain or estimate a value function, enabling a more stable learning problem, and more easily allowing the algorithm to incorporate off-policy data.
Revised: GCSL does not maintain or estimate a value function, enabling a more stable learning problem, and more easily allowing the algorithm to incorporate off-policy data.

Original: GCSL is inspired by supervised imitation learning (Billard et al., 2008; Hussein et al., 2017) via behavioral cloning (Pomerleau, 1989).
Revised: GCSL is inspired by supervised imitation learning (Billard et al., 2008; Hussein et al., 2017) via behavioral cloning (Pomerleau, 1989).

Original: Recent works have also considered imitation learning with goal relabeling for learning from human play data (Lynch et al., 2019; Gupta et al., 2019) or demonstrations (Ding et al., 2019).
Revised: Recent works have also considered imitation learning with goal relabeling for learning from human play data (Lynch et al., 2019; Gupta et al., 2019) or demonstrations (Ding et al., 2019).

Original: While GCSL is procedurally similar to Lynch et al.
Revised: While GCSL is procedurally similar to Lynch et al.

Original: and
Revised: and

Original: Ding et al.
Revised: Ding et al.

Original: , it differs crucially on the type of data used to train the policy — GCSL is trained on data collected by the agent itself from scratch , not from an expert or (noisy) optimal supervisor.
Revised: , it differs crucially on the type of data used to train the policy — GCSL is trained on data collected by the agent itself from scratch , not from an expert or (noisy) optimal supervisor.

Original: The fact that the same algorithmic procedure for training on optimal demonstrations can be applied iteratively using data from a sub-optimal agent to learn from scratch is non-trivial and constitutes one of our contributions.
Revised: The fact that the same algorithmic procedure for training on optimal demonstrations can be applied iteratively using data from a sub-optimal agent to learn from scratch is non-trivial and constitutes one of our contributions.

Original: GCSL has strong connections to direct policy search and self-imitation algorithms.
Revised: GCSL has strong connections to direct policy search and self-imitation algorithms.

Original: Direct policy search methods (Mannor et al., 2003; Peters & Schaal, 2007; Theodorou et al., 2010; Goschin et al., 2013; Norouzi et al., 2016; Nachum et al., 2016) selectively weight policies or trajectories by their performance during learning, as measured by the environment’s reward function or a learned value function, and maximize the likelihood of these trajectories using supervised learning.
Revised: Direct policy search methods (Mannor et al., 2003; Peters & Schaal, 2007; Theodorou et al., 2010; Goschin et al., 2013; Norouzi et al., 2016; Nachum et al., 2016) selectively weight policies or trajectories by their performance during learning, as measured by the environment’s reward function or a learned value function, and maximize the likelihood of these trajectories using supervised learning.

Original: Similar algorithmic procedures have also been studied in the context of learning models for planning (Pathak et al., 2018;
Revised: Similar algorithmic procedures have also been studied in the context of learning models for planning (Pathak et al., 2018;

Original: Savinov et al., 2018; Eysenbach et al., 2019).
Revised: Savinov et al., 2018; Eysenbach et al., 2019).

Original: GCSL is also closely related to selfimitation learning, where a small subset of trajectories are chosen to be imitated alongside an RL objective (Oh et al., 2018;
Revised: GCSL is also closely related to selfimitation learning, where a small subset of trajectories are chosen to be imitated alongside an RL objective (Oh et al., 2018;

Original: Hao et al., 2019), often measured using a well-shaped reward function.
Revised: Hao et al., 2019), often measured using a well-shaped reward function.

Original: However, GCSL neither relies on a hand-shaped reward function nor chooses a select group of elites, instead using goal relabeling to imitate every previously collected trajectory for higher data re-use and sample efﬁciency.
Revised: However, GCSL neither relies on a hand-shaped reward function nor chooses a select group of elites, instead using goal relabeling to imitate every previously collected trajectory for higher data re-use and sample efﬁciency.

Original: Goal-conditioned self-imitation learning methods when combined with a meta-controller, have been shown by Ecoffet et al.
Revised: Goal-conditioned self-imitation learning methods when combined with a meta-controller, have been shown by Ecoffet et al.

Original: (2020) to learn policies much faster in a single-task sparse reward setting.
Revised: (2020) to learn policies much faster in a single-task sparse reward setting.

Original: This line of work presents an avenue for using GCSL to aid learning in the single-task setting as well.
Revised: This line of work presents an avenue for using GCSL to aid learning in the single-task setting as well.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: In our experiments, we comparatively evaluate GCSL on a number of goal-conditioned tasks.
Revised: In our experiments, we comparatively evaluate GCSL on a number of goal-conditioned tasks.

Original: We focus on answering the following questions:
Revised: We focus on answering the following questions:

Original: Does GCSL effectively learn goal-conditioned policies from scratch?
Revised: Does GCSL effectively learn goal-conditioned policies from scratch?

Original: Can GCSL learn behaviors more effectively than standard RL methods? 3.
Revised: Can GCSL learn behaviors more effectively than standard RL methods? 3.

Original: Is GCSL less sensitive to hyperparameters than value-based methods? 4.
Revised: Is GCSL less sensitive to hyperparameters than value-based methods? 4.

Original: Can GCSL incorporate demonstration data more effectively than value-based methods?5.
Revised: Can GCSL incorporate demonstration data more effectively than value-based methods?

Original:  E XPERIMENTAL F RAMEWORK
Revised: 5. E XPERIMENTAL F RAMEWORK

Original: We evaluate GCSL on ﬁve simulated control environments for goal-reaching: 2D room navigation, object pushing with a robotic arm, the classic Lunar Lander game, opening a door with a robotic arm, and object manipulation with a dexterous 9 DoF robotic hand (referred to as claw manipulation), shown in Figure 2 (Environments from Nair et al., 2018; Ghosh et al., 2019; Ahn et al., 2019, details in Appendix A.3).
Revised: We evaluate GCSL on ﬁve simulated control environments for goal-reaching: 2D room navigation, object pushing with a robotic arm, the classic Lunar Lander game, opening a door with a robotic arm, and object manipulation with a dexterous 9 DoF robotic hand (referred to as claw manipulation), shown in Figure 2 (Environments from Nair et al., 2018; Ghosh et al., 2019; Ahn et al., 2019, details in Appendix A.3).

Original: These tasks allow us to study the performance of our method under a variety of system dynamics, in settings with both easy and difﬁcult exploration.
Revised: These tasks allow us to study the performance of our method under a variety of system dynamics, in settings with both easy and difﬁcult exploration.

Original: For each task, the target goal distribution corresponds to a uniform distribution over reachable conﬁgurations.
Revised: For each task, the target goal distribution corresponds to a uniform distribution over reachable conﬁgurations.

Original: Performance is quantiﬁed by the distance of the agent to the goal at the last timestep.
Revised: Performance is quantiﬁed by the distance of the agent to the goal at the last timestep.

Original: We present details about the environments, evaluation protocol, hyperparameters, and an extended set of results in Appendix A.
Revised: We present details about the environments, evaluation protocol, hyperparameters, and an extended set of results in Appendix A.

Revised: We have additionally open-sourced our implementation at https://github.com/dibyaghosh/gcsl .

Original: For the practical implementation of GCSL, we parameterize the policy as a neural network that takes in state, goal as input, ignoring the horizon, and outputs a distribution over actions.
Revised: For the practical implementation of GCSL, we parameterize the policy as a neural network that takes in state, goal as input, ignoring the horizon, and outputs a distribution over actions.

Original: Although in general, the optimal policy does vary with the horizon, in environments where it is possible to stay at the goal, a Markovian policy that reaches, then stays at the goal can be near-optimal.
Revised: Although in general, the optimal policy does vary with the horizon, in environments where it is possible to stay at the goal, a Markovian policy that reaches, then stays at the goal can be near-optimal.

Original: Empirically, we also ﬁnd that Markovian policies exhibit more coherent exploratory behavior than horizon-varying policies.
Revised: Empirically, we also ﬁnd that Markovian policies exhibit more coherent exploratory behavior than horizon-varying [Figure]

Original:  In our main results, the GCSL policy network ignores the horizon provided to the policy, a design decision we ablate in Section 5.3.
Revised: GCSL PPO TD3-HER policies. In our main results, the GCSL policy network ignores the horizon provided to the policy, a design decision we ablate in Section 5.3.

Original: Implementation details for GCSL are in Appendix A.1.
Revised: Implementation details for GCSL are in Appendix A.1.

Original: 5. L EARNING G OAL -C ONDITIONED P OLICIES
Revised: L EARNING G OAL -C ONDITIONED P OLICIES

Original: We ﬁrst evaluate the effectiveness of GCSL for reaching goals on the domains visualized in Figure 2, covering a variety of control problems spanning robotics and video games.
Revised: We ﬁrst evaluate the effectiveness of GCSL for reaching goals on the domains visualized in Figure 2, covering a variety of control problems spanning robotics and video games.

Original: To better understand the performance of our algorithm, we provide comparisons to value-based methods utilizing hindsight experience replay (HER) (Andrychowicz et al., 2017), and policy-gradient methods, two well 2.0 1e 0. D i s t an c e Four Rooms 0.
Revised: To better understand the performance of our algorithm, we provide comparisons to value-based methods utilizing hindsight experience replay (HER) (Andrychowicz et al., 2017), and policy-gradient methods, two well established families of RL algorithms for solving goal-conditioned tasks.

Original: 1.01e 0.

Original: Sawyer Pushing 0.

Original: 2.0 1e

Original: Lunar Lander 0.

Original: 1.0 Timesteps 1e 0.

Original: D i s t an c e

Original: [Figure]

Original: [Figure]

Original: Timesteps 1e 0.

Original: [Figure]

Original: GCSL

Original: PPO [Figure] established families of RL algorithms for solving goal-conditioned tasks. In particular, we compare against TD3-HER, an off-policy temporal difference RL algorithm that combines TD3 (Fujimoto et al., 2018)
Revised: In particular, we compare against TD3-HER, an off-policy temporal difference RL algorithm that combines TD3 (Fujimoto et al., 2018)

Original: (an improvement on the DDPG method used by Andrychowicz et al.
Revised: (an improvement on the DDPG method used by Andrychowicz et al.

Original: with HER.
Revised: with HER.

Original: TD3-HER requires signiﬁcantly more machinery than GCSL: while GCSL only maintains a policy,
Revised: TD3-HER requires signiﬁcantly more machinery than GCSL: while GCSL only maintains a policy,

Original: TD3-HER maintains a policy, a value function, a target policy, and a target value function, all of which are necessary for good performance.
Revised: TD3-HER maintains a policy, a value function, a target policy, and a target value function, all of which are necessary for good performance.

Original: We also compare with PPO (Schulman et al., 2017), a state-of-the-art on-policy policy gradient algorithm that does not leverage data relabeling, but is known to provide more stable optimization than off-policy methods and perform well on typical benchmark problems.
Revised: We also compare with PPO (Schulman et al., 2017), a state-of-the-art on-policy policy gradient algorithm that does not leverage data relabeling, but is known to provide more stable optimization than off-policy methods and perform well on typical benchmark problems.

Original: Details for the training procedure for these comparisons, hyperparameter and architectural choices, as well as some additional comparisons are presented in Appendix A.2.
Revised: Details for the training procedure for these comparisons, hyperparameter and architectural choices, as well as some additional comparisons are presented in Appendix A.2.

Original: The results in Figure 3 show that GCSL generally performs as well or better than the best performing prior RL method on each task, only losing out slightly to PPO on the door opening task, where exploration is less of a challenge.
Revised: The results in Figure 3 show that GCSL generally performs as well or better than the best performing prior RL method on each task, only losing out slightly to PPO on the door opening task, where exploration is less of a challenge.

Original: GCSL outperforms both methods by a large margin on the pushing and claw tasks, and by a small margin on the lunar lander task.
Revised: GCSL outperforms both methods by a large margin on the pushing and claw tasks, and by a small margin on the lunar lander task.

Original: These empirical results suggest that GCSL, despite its simplicity, represents a stable and appealing alternative to signiﬁcantly more complex RL methods, without the need for separate critics, policy gradients, or target networks.
Revised: These empirical results suggest that GCSL, despite its simplicity, represents a stable and appealing alternative to signiﬁcantly more complex RL methods, without the need for separate critics, policy gradients, or target networks.

Original: 5. A NALYSIS OF L EARNING P ROGRESS AND L EARNED B EHAVIORS
Revised: 5. A NALYSIS OF L EARNING P ROGRESS AND L EARNED B EHAVIORS

Original: [Figure] To analyze GCSL, we evaluate its performance in a number of scenarios, varying the quality and quantity of data, the policy class, and the relabeling technique (Figure 4).
Revised: To analyze GCSL, we evaluate its performance in a number of scenarios, varying the quality and quantity of data, the policy class, and the relabeling technique (Figure 5).

Original: Full details for these scenarios and results for all domains are in Appendix A.4.
Revised: Full details for these scenarios and results for all domains are in Appendix A.4.

Original: First, we study how varying the policy class can affect the performance of GCSL.
Revised: First, we study how varying the policy class can affect the performance of GCSL.

Original: In Section 5.1, we hypothesized that GCSL with a Markovian policy would outperform a time-varying policy.
Revised: In Section 5.1, we hypothesized that GCSL with a Markovian policy would outperform a time-varying policy.

Original: Indeed, allowing policies to be time-varying (“TimeVarying Policy” in Figure 4) speeds up training on domains like Lunar Lander; on domains requiring more exploration like the Sawyer pushing task, exploration using time-varying policies is ineffective and degrades performance. 
Revised: Indeed, allowing policies to be time-varying (“Time-Varying Policy” in Figure 5) speeds up training on domains like Lunar Lander; on domains requiring more exploration like the Sawyer pushing task, exploration using time-varying policies is ineffective and degrades performance. [Figure]

Original: To investigate the impact of the data-collection policy, we consider variations that collect data using a ﬁxed policy or train only on on-policy data.
Revised: To investigate the impact of the data-collection policy, we consider variations that collect data using a ﬁxed policy or train only on on-policy data.

Original: When collecting data using a ﬁxed policy (“Fixed Data Collection” in Figure 4), the algorithm learns much slower, suggesting that iterative data collection is crucial for GCSL.
Revised: When collecting data using a ﬁxed policy (“Fixed Data Collection” in Figure 5) , the algorithm learns much slower, suggesting that iterative data collection is crucial for GCSL.

Original: By forcing the data to be on-policy (“OnPolicy” in Figure 4), the algorithm cannot utilize all data seen during training.
Revised: By forcing the data to be on-policy (“On-Policy” in Figure 5), the algorithm cannot utilize all data seen during training.

Original: GCSL still makes progress in this case, but more slowly.
Revised: GCSL still makes progress in this case, but more slowly.

Original: We additionally consider limited-horizon relabeling, in which only states and goals that are at most 3 steps apart are relabeled, similar to proposals in prior work (Pathak et al., 2018;
Revised: We additionally consider limited-horizon relabeling, in which only states and goals that are at most 3 steps apart are relabeled, similar to proposals in prior work (Pathak et al., 2018;

Original: Savinov et al., 2018).
Revised: Savinov et al., 2018).

Original: Limiting the horizon degrades performance (“Limited relabeling” in Figure 4), indicating that multi-horizon relabeling is important.
Revised: Limiting the horizon degrades performance (“Limited relabeling” in Figure 5), indicating that multi-horizon relabeling is important.

Original: [Figure]
Revised: [Figure]

Original: Finally, we discuss the concern that since GCSL uses ﬁnal-timestep optimality, it may provide signiﬁcantly different behaviors than shortest-path optimality.
Revised: Finally, we discuss the concern that since GCSL uses ﬁnal-timestep optimality, it may provide signiﬁcantly different behaviors than shortest-path optimality.

Original: While in theory, GCSL can learn round-about trajectories or otherwise exhibit pathological behavior, we ﬁnd that on our empirical benchmarks, GCSL learns fairly direct goal-reaching behaviors (visualized in Appendix C.2).
Revised: While in theory, GCSL can learn round-about trajectories or otherwise exhibit pathological behavior, we ﬁnd that on our empirical benchmarks, GCSL learns fairly direct goalreaching behaviors (visualized in Appendix C.2).

Original: Since even the time-varying policy shares network parameters for different horizons, we hypothesize that the policy is constrained to produce behaviors that are roughly consistent through time, resulting in directed behaviors that resemble shortest-path optimality.
Revised: Since even the time-varying policy shares network parameters for different horizons, we hypothesize that the policy is constrained to produce behaviors that are roughly consistent through time, resulting in directed behaviors that resemble shortest-path optimality.

Original: 5. R OBUSTNESS TO H YPERPARAMETERS
Revised: 5. R OBUSTNESS TO H YPERPARAMETERS

Original: Our next experiment tests the hypothesis that GCSL is more robust to hyperparameters than valuebased RL methods like TD3-HER.
Revised: Our next experiment tests the hypothesis that GCSL is more robust to hyperparameters than value-based RL methods like TD3-HER.

Original: The intuition is that, while dynamic programming methods are known to be quite sensitive to hyperparameters (Henderson et al., 2018), supervised learning techniques seem more robust.
Revised: The intuition is that, while dynamic programming methods are known to be quite sensitive to hyperparameters (Henderson et al., 2018), supervised learning techniques seem more robust.

Original: We ran a sweep across nine hyperparameter conﬁgurations, varying network capacity (size of the hidden layers in [250 , 500 , 1000] ) and frequency of gradient updates (gradient updates per environment step in [1 , 2 , 4] ).
Revised: We ran a sweep across nine hyperparameter conﬁgurations, varying network capacity (size of the hidden layers in [250 , 500 , 1000] ) and frequency of gradient updates (gradient updates per environment step in [1 , 2 , 4] ).

Original: We compared both GCSL and TD3-HER and plotted the distribution of ﬁnal timestep performance across all possible conﬁgurations in Fig.
Revised: We compared both GCSL and TD3-HER and plotted the distribution of ﬁnal timestep performance across all possible conﬁgurations in Fig.

Original: We observe that the distribution of performance for GCSL is more tightly clustered than for TD3-HER, indicating lower sensitivity to hyperparameters.
Revised: We observe that the distribution of performance for GCSL is more tightly clustered than for TD3-HER, indicating lower sensitivity to hyperparameters.

Original: We emphasize that GCSL has fewer hyperparameters than TD3-HER; since GCSL does not learn a value function, it does not require parameters for the value function architecture, target update frequency, discount factor, or actor update frequency.
Revised: We emphasize that GCSL has fewer hyperparameters than TD3-HER; since GCSL does not learn a value function, it does not require parameters for the value function architecture, target update frequency, discount factor, or actor update frequency.

Original: 5. I NITIALIZING WITH D EMONSTRATIONS
Revised: 5. I NITIALIZING WITH D EMONSTRATIONS

Original: As GCSL can relabel and imitate trajectories from arbitrary sources, the algorithm is amenable to initialization from logs of previously collected trajectories or from demonstration data collected by an expert.
Revised: As GCSL can relabel and imitate trajectories from arbitrary sources, the algorithm is amenable to initialization from logs of previously collected trajectories or from demonstration data collected by an expert.

Original: In this section, we compare the performance of GCSL bootstrapped from expert demonstrations to TD3-HER.
Revised: In this section, we compare the performance of GCSL bootstrapped from expert demonstrations to TD3-HER.

Original: Both methods can in principle utilize off-policy demonstrations; however, our results in Figure 6 show that GCSL beneﬁts substantially more from these demonstrations.
Revised: Both methods can in principle utilize off-policy demonstrations; however, our results in Figure 6 show that GCSL beneﬁts substantially more from these demonstrations.

Original: While value-based RL methods are known to struggle with data that is far off-policy (Kumar et al., 2019a), the simple supervised learning procedure in GCSL can take advantage of such data easily.
Revised: While value-based RL methods are known to struggle with data that is far off-policy (Kumar et al., 2019a), the simple supervised learning procedure in GCSL can take advantage of such data easily.

Original: [Figure]
Revised: [Figure]

Original: In this experiment, we provide the agent with a set of demonstration trajectories, each for reaching a different goal.
Revised: In this experiment, we provide the agent with a set of demonstration trajectories, each for reaching a different goal.

Original: GCSL adds this data to the initial dataset, without any other modiﬁcations to the algorithm.
Revised: GCSL adds this data to the initial dataset, without any other modiﬁcations to the algorithm.

Original: For TD3-HER, we incorporate demonstrations following the setup of Vecerik et al.
Revised: For TD3-HER, we incorporate demonstrations following the setup of Vecerik et al.

Original: Even with these measures, the value function in TD3-HER still suffers degraded performance and error accumulation during pre-training.
Revised: Even with these measures, the value function in TD3-HER still suffers degraded performance and error accumulation during pre-training.

Original: When expert demonstrations are provided for the robotic pushing environment (Figure 6), GCSL progressively improves faster than when from scratch, but TD3 is unable to improve substantially beyond the original behavioral-cloned policy.
Revised: When expert demonstrations are provided for the robotic pushing environment (Figure 6), GCSL progressively improves faster than when from scratch, but TD3 is unable to improve substantially beyond the original behavioral-cloned policy.

Original: We hypothesize that the difference in performance largely occurs because of the instability and optimism bias present when training value functions using demonstrations.
Revised: We hypothesize that the difference in performance largely occurs because of the instability and optimism bias present when training value functions using demonstrations.

Original:  D ISCUSSIONAND F UTURE W ORK
Revised:  D ISCUSSION

Original: We proposed GCSL, a simple algorithm that uses supervised learning on its own previously collected data to iteratively learn goal-reaching policies from scratch.
Revised: We proposed GCSL, a simple algorithm that uses supervised learning on its own previously collected data to iteratively learn goal-reaching policies from scratch.

Original: GCSL lifts several limitations of previous goal-reaching methods: it does not require a hand-deﬁned reward, expert demonstrations, or the need to learn a value function.
Revised: GCSL lifts several limitations of previous goal-reaching methods: it does not require a hand-deﬁned reward, expert demonstrations, or the need to learn a value function.

Original: GCSL often outperforms more complex RL algorithms, is robust to hyperparameters, uses off-policy data and can incorporate expert demonstrations when they are available.
Revised: GCSL often outperforms more complex RL algorithms, is robust to hyperparameters, uses off-policy data, and can incorporate expert demonstrations when they are available.

Original: The current instantiation of GCSL is limited in exploration, since it relies primarily on the stochasticity of the policy to explore; a promising future direction would be to selectively reweight the sampled rollouts to promote novelty-seeking exploration.
Revised: The current instantiation of GCSL is limited in exploration, since it relies primarily on the stochasticity of the policy to explore; a promising future direction would be to selectively reweight the sampled rollouts to promote novelty-seeking exploration.

Original: Nonetheless, GCSL is simple, scalable, and readily applicable — a step towards the fully autonomous learning of goal-directed agents 
Revised: Nonetheless, GCSL is simple, scalable, and readily applicable — a step towards the fully autonomous learning of goal-directed agents A CKNOWLEDGEMENTS

Revised: This research was supported by an NSF graduate fellowship, Berkeley DeepDrive, the National

Revised: Science Foundation, the Ofﬁce of Naval Research and support from Google, Amazon, and NVIDIA.

Revised: We thank Karol Hausman, Ignasi Clavera, Aviral Kumar, Marvin Zhang, Vikash Kumar for thoughtful discussions, insights and feedback on paper drafts

Original: A E XPERIMENTAL D ETAILS A. G OAL -C ONDITIONED S UPERVISED L EARNING (GCSL)
Revised: A E XPERIMENTAL D ETAILS A. G OAL -C ONDITIONED S UPERVISED L EARNING (GCSL)

Original: GCSL iteratively performs maximum likelihood estimation using a dataset of relabeled trajectories that have been previously collected by the agent.
Revised: GCSL iteratively performs maximum likelihood estimation using a dataset of relabeled trajectories that have been previously collected by the agent.

Original: Here we present details about the policy class, data collection procedure, and other design choices.
Revised: Here we present details about the policy class, data collection procedure, and other design choices.

Original: We parameterize a time-invariant policy using a neural network which takes as input state and goal (not the horizon), and returns probabilities for a discretized grid of actions of the action space.
Revised: We parameterize a time-invariant policy using a neural network which takes as input state and goal (not the horizon), and returns probabilities for a discretized grid of actions of the action space.

Original: The neural network concatenates the state and goal together, and passes the concatenated input into a feedforward network with two hidden layers of size 400 and 300 respectively, outputting logits for each discretized action.
Revised: The neural network concatenates the state and goal together, and passes the concatenated input into a feedforward network with two hidden layers of size 400 and 300 respectively, outputting logits for each discretized action.

Original: Empirically, we have found GCSL to perform much better with larger choices of neural networks; however, we use this two-layer neural network for fair comparisons to TD3-HER.
Revised: Empirically, we have found GCSL to perform much better with larger choices of neural networks; however, we use this two-layer neural network for fair comparisons to TD3-HER.

Original: The GCSL loss is optimized using the Adam optimizer with learning rate α = 5 × 10 − 4 , with a batch size of 256 , taking one gradient step for every step in the environment.
Revised: The GCSL loss is optimized using the Adam optimizer with learning rate α = 5 × 10 − 4 , with a batch size of 256 , taking one gradient step for every step in the environment.

Original: When executing in the environment, the ﬁrst 10000 environment steps are taken according to uniform random action selection, after which the data-collection policy is the greedy policy: a = arg max a π ( a | s, g ) .
Revised: When executing in the environment, the ﬁrst 10000 environment steps are taken according to uniform random action selection, after which the data-collection policy is the greedy policy: a = arg max a π ( a | s, g ) .

Original: The replay buffer stores trajectories and relabels on the ﬂy, with the size of the buffer subject only to memory constraints.
Revised: The replay buffer stores trajectories and relabels on the ﬂy, with the size of the buffer subject only to memory constraints.

Original: To clarify, instead of explicitly relabeling and storing all (cid:0) T 2 (cid:1) possible tuples from a trajectory, we instead save the trajectory and relabel at training time.
Revised: To clarify, instead of explicitly relabeling and storing all (cid:0) T 2 (cid:1) possible tuples from a trajectory, we instead save the trajectory and relabel at training time.

Original: When sampling from the dataset, a trajectory is chosen at random, a start index t and goal index t (cid:48) >
Revised: When sampling from the dataset, a trajectory is chosen at random, a start index t and goal index t (cid:48) >

Original: t are sampled uniformly at random, and the tuple corresponding to this state and goal are relabelled and sampled.
Revised: t are sampled uniformly at random, and the tuple corresponding to this state and goal are relabelled and sampled.

Original: A. RL C OMPARISONS
Revised: A. RL C OMPARISONS

Original: We provide comparisons to two goal-reaching RL methods: TD3-HER and PPO.
Revised: We provide comparisons to two goal-reaching RL methods: TD3-HER and PPO.

Original: While GCSL is able to efﬁciently optimize the last-timestep objective, RL methods that learn value functions perform poorly when optimizing this objective.
Revised: While GCSL is able to efﬁciently optimize the last-timestep objective, RL methods that learn value functions perform poorly when optimizing this objective.

Original: We include results of RL methods optimizing the last-timestep objective for completeness in Figure 9, but our main RL comparisons instead optimize the classical discounted sum of returns:
Revised: We include results of RL methods optimizing the last-timestep objective for completeness in Figure 9, but our main RL comparisons instead optimize the classical discounted sum of returns:

Original: [Equation]
Revised: [Equation]

Original: where reaching the goal leads to termination, the reward function remains the same, r g ( s )
Revised: where reaching the goal leads to termination, the reward function remains the same, r g ( s )

Original: = 1 ( s = g )
Revised: = 1 ( s = g )

Original: and we use γ = 0 .
Revised: and we use γ = 0 .

Original: We log other metrics for these algorithms in Figure 12, and have included a variety of possible metrics.
Revised: We log other metrics for these algorithms in Figure 12, and have included a variety of possible metrics.

Original: TD3-HER (Fujimoto et al., 2018; Andrychowicz et al., 2017): To efﬁciently learn a value function, we use hindsight relabelling.
Revised: TD3-HER (Fujimoto et al., 2018; Andrychowicz et al., 2017): To efﬁciently learn a value function, we use hindsight relabelling.

Original: Speciﬁcally, a transition (( s, g ) , a, ( s (cid:48) , g )) gets relabeled to (( s, g (cid:48) ) , a, ( s (cid:48) , g (cid:48) )) , where g (cid:48)
Revised: Speciﬁcally, a transition (( s, g ) , a, ( s (cid:48) , g )) gets relabeled to (( s, g (cid:48) ) , a, ( s (cid:48) , g (cid:48) )) , where g (cid:48)

Original: =
Revised: =

Original: g with probability 0 .
Revised: g with probability 0 .

Original: 1 , g (cid:48)
Revised: 1 , g (cid:48)

Original: =
Revised: =

Original: s (cid:48) with probability 0 .
Revised: s (cid:48) with probability 0 .

Original: 5 , and g (cid:48) =
Revised: 5 , and g (cid:48) =

Original: s t for some future state in the trajectory s t with probability 0 .
Revised: s t for some future state in the trajectory s t with probability 0 .

Original: As described in Section 2 , the agent receives a reward of 1 and the trajectory ends if the transition is relabeled to g (cid:48)
Revised: As described in Section 2 , the agent receives a reward of 1 and the trajectory ends if the transition is relabeled to g (cid:48)

Original: =
Revised: =

Original: s (cid:48) , and 0 otherwise.
Revised: s (cid:48) , and 0 otherwise.

Original: Under this formalism, the optimal value function, V ∗ ( s, g ) ∝ γ T ( s,g ) , where T ( s, g ) is the minimum expected time to go from s to g .
Revised: Under this formalism, the optimal value function, V ∗ ( s, g ) ∝ γ T ( s,g ) , where T ( s, g ) is the minimum expected time to go from s to g .

Original: Both the Q-function and the actor for TD3 are parametrized as neural networks, with the same architecture (except ﬁnal layers) for state-based domains as those for GCSL.
Revised: Both the Q-function and the actor for TD3 are parametrized as neural networks, with the same architecture (except ﬁnal layers) for state-based domains as those for GCSL.

Original: We found the default values of learning rate, target update period, and number of critic updates to be the best amongst our hyperparameter search across the domains (single set of hyperparameters for all domains).
Revised: We found the default values of learning rate, target update period, and number of critic updates to be the best amongst our hyperparameter search across the domains (single set of hyperparameters for all domains).

Original: Since GCSL uses discretized actions, we additionally compared to a version of TD3-HER that also uses discretized actions (results in Figure 12).
Revised: Since GCSL uses discretized actions, we additionally compared to a version of TD3-HER that also uses discretized actions (results in Figure 12).

Original: The performance of TD3-HER w/ discretized actions decreases performance on the Claw task, increases on the Door Opening task, and has the same performance on the other three environments.
Revised: The performance of TD3-HER w/ discretized actions decreases performance on the Claw task, increases on the Door Opening task, and has the same performance on the other three environments.

Original: PPO (Schulman et al., 2017): Because PPO is an on-policy RL algorithm, we cannot relabel goals, unlike in GCSL or TD3-HER.
Revised: PPO (Schulman et al., 2017): Because PPO is an on-policy RL algorithm, we cannot relabel goals, unlike in GCSL or TD3-HER.

Original: Instead, we provide a surrogate (cid:15)
Revised: Instead, we provide a surrogate (cid:15)

Original: -ball indicator reward function: r ( s, g ) =
Revised: -ball indicator reward function: r ( s, g ) =

Original: 1( d ( s, g ) <
Revised: 1( d ( s, g ) <

Original: (cid:15) ) , where (cid:15) is chosen appropriately for each environment.
Revised: (cid:15) ) , where (cid:15) is chosen appropriately for each environment.

Original: We emphasize that this reward function makes the policy optimization problem for PPO much easier, since while the other two methods only can check if two states are exactly equal, PPO also has access to the distance
Revised: We emphasize that this reward function makes the policy optimization problem for PPO much easier, since while the other two methods only can check if two states are exactly equal, PPO also has access to the distance

Original: (within an (cid:15) -ball).
Revised: (within an (cid:15) -ball).

Original: To maximize the data efﬁciency of PPO, we performed a coarse hyperparameter sweep over the batch size for the algorithm.
Revised: To maximize the data efﬁciency of PPO, we performed a coarse hyperparameter sweep over the batch size for the algorithm.

Original: Just as with TD3, we mimic the same neural network architecture for the parametrizations of the policies as GCSL.
Revised: Just as with TD3, we mimic the same neural network architecture for the parametrizations of the policies as GCSL.

Original: A. T ASK D ESCRIPTIONS
Revised: A. T ASK D ESCRIPTIONS

Original: For each environment, the goal space is identical to the state space; each trajectory in the environment lasts for 50 timesteps.
Revised: For each environment, the goal space is identical to the state space; each trajectory in the environment lasts for 50 timesteps.

Original: 2D Room Navigation (Ghosh et al., 2019)
Revised: 2D Room Navigation (Ghosh et al., 2019)

Original: This environment requires an agent to navigate to points in an environment with four rooms that connect to adjacent rooms.
Revised: This environment requires an agent to navigate to points in an environment with four rooms that connect to adjacent rooms.

Original: The state space has two dimensions, consisting of the cartesian coordinates of the agent.
Revised: The state space has two dimensions, consisting of the cartesian coordinates of the agent.

Original: The agent has acceleration control, and the action space has two dimensions.
Revised: The agent has acceleration control, and the action space has two dimensions.

Original: The distribution of goals p ( g ) is uniform on the state space, and the agent starts in a ﬁxed location in the bottom left room.
Revised: The distribution of goals p ( g ) is uniform on the state space, and the agent starts in a ﬁxed location in the bottom left room.

Original: Robotic Pushing (Ghosh et al., 2019)
Revised: Robotic Pushing (Ghosh et al., 2019)

Original: This environment requires a Sawyer manipulator to move a freely moving block in an enclosed play area with dimensions 40 cm × 20 cm.
Revised: This environment requires a Sawyer manipulator to move a freely moving block in an enclosed play area with dimensions 40 cm × 20 cm.

Original: The state space is 4 -dimensional, consisting of the Cartesian coordinates of the end-effector of the sawyer agent and the Cartesian coordinates of the block.
Revised: The state space is 4 -dimensional, consisting of the Cartesian coordinates of the end-effector of the sawyer agent and the Cartesian coordinates of the block.

Original: The Sawyer is controlled via end-effector position control with a three-dimensional action space.
Revised: The Sawyer is controlled via end-effector position control with a three-dimensional action space.

Original: The distribution of goals p ( g ) is uniform on the state space (uniform block location and uniform end-effector location), and the agent starts with the block and end-effector both in the bottom-left corner of the play area.
Revised: The distribution of goals p ( g ) is uniform on the state space (uniform block location and uniform end-effector location), and the agent starts with the block and end-effector both in the bottom-left corner of the play area.

Original: Lunar Lander (Brockman et al., 2016)
Revised: Lunar Lander (Brockman et al., 2016)

Original: This environment requires a rocket to land in a speciﬁed region.
Revised: This environment requires a rocket to land in a speciﬁed region.

Original: The state space includes the normalized position of the rocket, the angle of the rocket, whether the legs of the rocket are touching the ground, and velocity information.
Revised: The state space includes the normalized position of the rocket, the angle of the rocket, whether the legs of the rocket are touching the ground, and velocity information.

Original: Goals are sampled uniformly along the landing region, either touching the ground or hovering slightly above, with zero velocity.
Revised: Goals are sampled uniformly along the landing region, either touching the ground or hovering slightly above, with zero velocity.

Original: Door Opening: (Nair et al., 2018)
Revised: Door Opening: (Nair et al., 2018)

Original: This environment requires a Sawyer manipulator to open a small cabinet door, initially shut closed, sitting on a table to a speciﬁed angle.
Revised: This environment requires a Sawyer manipulator to open a small cabinet door, initially shut closed, sitting on a table to a speciﬁed angle.

Original: The state space consists of the Cartesian coordinates of the Sawyer end-effector and the door’s angle.
Revised: The state space consists of the Cartesian coordinates of the Sawyer end-effector and the door’s angle.

Original: As in the Robotic Pushing task, the three-dimensional action space controls the position of the end-effector.
Revised: As in the Robotic Pushing task, the three-dimensional action space controls the position of the end-effector.

Original: The distribution of goals p ( g ) is uniform on door angles from 0 (completely closed) to 0.83 radians.
Revised: The distribution of goals p ( g ) is uniform on door angles from 0 (completely closed) to 0.83 radians.

Original: Claw Manipulation: (Ahn et al., 2019)
Revised: Claw Manipulation: (Ahn et al., 2019)

Original: A 9-DOF ”claw”-like robot is required to turn a valve to various positions .
Revised: A 9-DOF ”claw”-like robot is required to turn a valve to various positions .

Original: The state space includes the positions of each joint of each claw (3 joints onclaws) and embeds the current angle of the valve in Cartesian coordinate ( θ (cid:55)→ (sin θ, cos θ ) ).
Revised: The state space includes the positions of each joint of each claw (3 joints onclaws) and embeds the current angle of the valve in Cartesian coordinate ( θ (cid:55)→ (sin θ, cos θ ) ).

Original: The robot is controlled via joint angle control.
Revised: The robot is controlled via joint angle control.

Original: The goal space consists only of the claw angle, which is sampled uniformly from the unit circle.
Revised: The goal space consists only of the claw angle, which is sampled uniformly from the unit circle.

Original: A. A BLATIONS
Revised: A. A BLATIONS

Original: In Section 5.3, we analyzed the performance of the following variants of GCSL (Figure 8).
Revised: In Section 5.3, we analyzed the performance of the following variants of GCSL (Figure 7).

Original: Limited relabeling - This model relabels only states and goals that are at most three steps apart: { ( s t , a t , s t
Revised: Limited relabeling - This model relabels only states and goals that are at most three steps apart: { ( s t , a t , s t

Original: + h , h ) : t > 0 , h ≤ 3 } 2.
Revised: + h , h ) : t > 0 , h ≤ 3 } 2.

Original: On-Policy Only the most recent 10000 transitions are stored and trained on. 3.
Revised: On-Policy Only the most recent 10000 transitions are stored and trained on. 3.

Original: Fixed Data Collection Data is collected according to a uniform policy over actions.
Revised: Fixed Data Collection Data is collected according to a uniform policy over actions.

Original: Time-Varying Policy Policies are conditioned on the remaining horizon.
Revised: Time-Varying Policy Policies are conditioned on the remaining horizon.

Original: Alongside the state and goal, the policy gets a reverse temperature encoding of the horizon as input.
Revised: Alongside the state and goal, the policy gets a reverse temperature encoding of the horizon as input.

Original: [Figure]
Revised: [Figure]

Original: In addition
Revised: In addition

Original: [Figure]
Revised: [Figure]

Original: we also performed an ablation to measure how the performance of GCSL depends on the ratio of gradient steps to environment steps being collected.
Revised: we also performed an ablation to measure how the performance of GCSL depends on the ratio of gradient steps to environment steps being collected.

Original: A larger number of gradient updates indicate higher levels of data re-use.
Revised: A larger number of gradient updates indicate higher levels of data re-use.

Original: Our results indicate that GCSL is robust to this hyperparameter
Revised: Our results indicate that GCSL is robust to this hyperparameter

Original: [Figure]
Revised: [Figure]

Original: Finally
Revised: Finally

Original: [Figure]
Revised: [Figure]

Original: we compared the performance of our RL comparisons, TD3-HER and PPO, when optimizing for the ﬁnal-timestep objective in Equation 2 compared to optimizing the discounted return objective in Equation 4.
Revised: we compared the performance of our RL comparisons, TD3-HER and PPO, when optimizing for the ﬁnal-timestep objective in Equation 2 compared to optimizing the discounted return objective in Equation 4.

Original: To faithfully optimize the ﬁnal-timestep objective, the policy and value networks for
Revised: To faithfully optimize the ﬁnal-timestep objective, the policy and value networks for

Original: TD3-HER and PPO also take the remaining horizon as input.
Revised: TD3-HER and PPO also take the remaining horizon as input.

Original: Our results indicate that TD3-HER and
Revised: Our results indicate that TD3-HER and

Original: PPO learn slower (and sometimes not at all) when optimizing the ﬁnal-timestep objective than the discounted return objective.
Revised: PPO learn slower (and sometimes not at all) when optimizing the ﬁnal-timestep objective than the discounted return objective.

Original: Therefore, for the most fair evaluation, we compare the performance of
Revised: Therefore, for the most fair evaluation, we compare the performance of

Original: GCSL to the RL methods that optimize the discounted return objective
Revised: GCSL to the RL methods that optimize the discounted return objective

Original: [Figure]
Revised: [Figure]

Original: A. I NITIALIZING WITH D EMONSTRATIONS
Revised: A. I NITIALIZING WITH D EMONSTRATIONS

Original: We train an expert policy for robotic pushing using TRPO with a shaped dense reward function, and collect a dataset of 200 trajectories, each corresponding to a different goal.
Revised: We train an expert policy for robotic pushing using TRPO with a shaped dense reward function, and collect a dataset of 200 trajectories, each corresponding to a different goal.

Original: To train GCSL using these demonstrations, we simply populate the replay buffer with these trajectories at the beginning of training, and optimize the GCSL objective using these trajectories to warm-start the algorithm.
Revised: To train GCSL using these demonstrations, we simply populate the replay buffer with these trajectories at the beginning of training, and optimize the GCSL objective using these trajectories to warm-start the algorithm.

Original: Initializing a value function method using demonstrates requires signiﬁcantly more attention: we perform the following procedure.
Revised: Initializing a value function method using demonstrates requires signiﬁcantly more attention: we perform the following procedure.

Original: First, we perform goal-conditioned behavior cloning to learn an initial policy π BC .
Revised: First, we perform goal-conditioned behavior cloning to learn an initial policy π BC .

Original: Next, we collect 200 new trajectories in the environment using a uniform data collection scheme.
Revised: Next, we collect 200 new trajectories in the environment using a uniform data collection scheme.

Original: Using this dataset of 400 trajectories, we perform policy evaluation on π BC to learn Q π BC using policy evaluation via bootstrapping.
Revised: Using this dataset of 400 trajectories, we perform policy evaluation on π BC to learn Q π BC using policy evaluation via bootstrapping.

Original: Having trained such an estimate of the
Revised: Having trained such an estimate of the

Original: Q-function, we initialize the policy and Q-function to these estimates, and run the appropriate value function RL algorithm.
Revised: Q-function, we initialize the policy and Q-function to these estimates, and run the appropriate value function RL algorithm.

Original: A. H INDSIGHT P OLICY G RADIENTS
Revised: A. H INDSIGHT P OLICY G RADIENTS

Original: [Figure]
Revised: [Figure]

Original: We compared GCSL to hindsight policy gradients (HPG) (Rauber et al., 2017) on the Fetch-Reach task from Andrychowicz et al.
Revised: We compared GCSL to hindsight policy gradients (HPG) (Rauber et al., 2017) on the Fetch-Reach task from Andrychowicz et al.

Original: Whereas GCSL uses relabelling in conjunction with supervised imitation learning,
Revised: Whereas GCSL uses relabelling in conjunction with supervised imitation learning,

Original: HPG performs reward relabeling with a policy gradient algorithm, introducing importance weights to correct for the distribution shift.
Revised: HPG performs reward relabeling with a policy gradient algorithm, introducing importance weights to correct for the distribution shift.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 10, GCSL solves the task faster than HPG and converges to a policy that gets closer to the goal.
Revised: 10, GCSL solves the task faster than HPG and converges to a policy that gets closer to the goal.

Original: HPG required more than 1 million samples to run reliably on Fetch-Reach, a task far easier to solve than any of our benchmark environments (Andrychowicz et al., 2017)
Revised: HPG required more than 1 million samples to run reliably on Fetch-Reach, a task far easier to solve than any of our benchmark environments (Andrychowicz et al., 2017)

Original: A. S IMPLE E XPLORATION S TRATEGIES
Revised: A. S IMPLE E XPLORATION S TRATEGIES

Original: [Figure]
Revised: [Figure]

Original: (2020), GCSL explores more and learns a more accurate goal-reaching policy when navigating a 2x larger four-rooms domain.
Revised: 2020), GCSL explores more and learns a more accurate goal-reaching policy when navigating a 2x larger four-rooms domain.

Original: Since GCSL learns via self-imitation, like many other selfimitation methods, it faces challenges of poor exploration and converging to sub-optimal points (Oh et al., 2018).
Revised: Since GCSL learns via self-imitation, like many other selfimitation methods, it faces challenges of poor exploration and converging to sub-optimal points (Oh et al., 2018).

Original: However, since GCSL can learn with off-policy data, we can deal with exploration challenges by using a directed exploration strategy or by injecting noise into the data collection process.
Revised: However, since GCSL can learn with off-policy data, we can deal with exploration challenges by using a directed exploration strategy or by injecting noise into the data collection process.

Original: As a simple demonstration, we combined GCSL with an exploration strategy akin to Go-Explore (Ecoffet et al., 2020).
Revised: As a simple demonstration, we combined GCSL with an exploration strategy akin to Go-Explore (Ecoffet et al., 2020).

Original: This exploration strategy takes greedy actions with the policy for the ﬁrst 90% of the trajectory, and then takes random actions for the remainder.
Revised: This exploration strategy takes greedy actions with the policy for the ﬁrst 90% of the trajectory, and then takes random actions for the remainder.

Original: Intuitively, this can be seen as a crude implementation of ﬁrst getting to the fringe states, and then randomly exploring beyond the fringe.
Revised: Intuitively, this can be seen as a crude implementation of ﬁrst getting to the fringe states, and then randomly exploring beyond the fringe.

Original: The main difference with this intuition is that we do not actually adjust the goal sampling strategy to only reach goals within the fringe.
Revised: The main difference with this intuition is that we do not actually adjust the goal sampling strategy to only reach goals within the fringe.

Original: We tested this exploration strategy on a 2x larger version of the four rooms domain, where greedy exploration is insufﬁcient to reach all regions in the space.
Revised: We tested this exploration strategy on a 2x larger version of the four rooms domain, where greedy exploration is insufﬁcient to reach all regions in the space.

Original: GCSL with the default exploration strategy is unable to reach all the goals, but when combined with this new exploration strategy, is able to learn faster in this environment and reach 15% more goals.
Revised: GCSL with the default exploration strategy is unable to reach all the goals, but when combined with this new exploration strategy, is able to learn faster in this environment and reach 15% more goals.

Original: [Figure]
Revised: [Figure]

Original: B T HEORETICAL A NALYSIS B. P ROOF OF T HEOREM 4. 
Revised: B T HEORETICAL A NALYSIS B. P ROOF OF T HEOREM 4.

Original: We will assume a discrete state space in this proof, and denote a trajectory as τ = { s 0 , a 0 , . . .
Revised: We will assume a discrete state space in this proof, and denote a trajectory as τ = { s 0 , a 0 , . . .

Original: , s T , a T } .
Revised: , s T , a T } .

Original: Let the notation G ( τ ) =
Revised: Let the notation G ( τ ) =

Original: s T denote the ﬁnal state of a trajectory, which represents the goal that the trajectory reached.
Revised: s T denote the ﬁnal state of a trajectory, which represents the goal that the trajectory reached.

Original: As there can be multiple paths to a goal, we let τ g = { τ : G ( τ ) =
Revised: As there can be multiple paths to a goal, we let τ g = { τ : G ( τ ) =

Original: g } denote the set of trajectories that reach a particular goal g .
Revised: g } denote the set of trajectories that reach a particular goal g .

Original: We abbreviate a policy’s trajectory distribution as π ( τ | g )
Revised: We abbreviate a policy’s trajectory distribution as π ( τ | g )

Original: = p ( s 0 ) (cid:81) Tt =

Revised: = p ( s 0 ) (cid:81) Tt =

Original: π ( a t | s t , g ) T ( s t +1 | s t , a t ) .
Revised: π ( a t | s t , g ) T ( s t +1 | s t , a t ) .

Original: The target goalreaching objective we wish to optimize is the probability of reaching a commanded goal, when goals are sampled from a pre-speciﬁed distribution p ( g ) .
Revised: The target goalreaching objective we wish to optimize is the probability of reaching a commanded goal, when goals are sampled from a pre-speciﬁed distribution p ( g ) .

Original: [Equation]
Revised: [Equation]

Original: GCSL optimizes the following objective, where the log-likelihood of the actions conditioned on the goals actually reached by the policy, G ( τ ) .
Revised: GCSL optimizes the following objective, where the log-likelihood of the actions conditioned on the goals actually reached by the policy, G ( τ ) .

Original: The distribution of trajectories used to optimize the objective is collected through a different policy, π old .
Revised: The distribution of trajectories used to optimize the objective is collected through a different policy, π old .

Original: We write π old ( τ ) =
Revised: We write π old ( τ ) =

Original: E g ∼ p
Revised: E g ∼ p

Original: ( g ) [ π old ( τ | g )] to concisely represent the marginalized distribution of trajectories from π old .
Revised: ( g ) [ π old ( τ | g )] to concisely represent the marginalized distribution of trajectories from π old .

Original: [Equation]
Revised: [Equation]

Original: To analyze how this objective relates to J ( π ) , we ﬁrst analyze the relationship between J ( π ) and a surrogate objective , given by
Revised: To analyze how this objective relates to J ( π ) , we ﬁrst analyze the relationship between J ( π ) and a surrogate objective , given by

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 from Schulman et al.
Revised: Theorem 1 from Schulman et al.

Original: (2015) states that
Revised: (2015) states that

Original: [Equation]
Revised: [Equation]

Original: where γ is a discount factor, (cid:15) is the maximum advantage over all states and actions, and α is the total variation distance between π and π old .
Revised: where γ is a discount factor, (cid:15) is the maximum advantage over all states and actions, and α is the total variation distance between π and π old .

Original: It is straightforward to show that the bound can be rewritten in the ﬁnite-horizon undiscounted case in terms of the horizon T , following Kakade & Langford (2002); Ross et al.
Revised: It is straightforward to show that the bound can be rewritten in the ﬁnite-horizon undiscounted case in terms of the horizon T , following Kakade & Langford (2002); Ross et al.

Original: (2011), to obtain the bound
Revised: (2011), to obtain the bound

Original: [Equation]
Revised: [Equation]

Original: where T is the horizon of the task.
Revised: where T is the horizon of the task.

Original: In the setting where data is collected from multiple policies, for example with a replay buffer, the bound cannot rely on the distance between policies at each state, but rather more generally the total variation distance between the trajectory distributions,
Revised: In the setting where data is collected from multiple policies, for example with a replay buffer, the bound cannot rely on the distance between policies at each state, but rather more generally the total variation distance between the trajectory distributions,

Original: [Equation]
Revised: [Equation]

Original: Since our reward function is 1 [ G ( τ )
Revised: Since our reward function is 1 [ G ( τ )

Original: =
Revised: =

Original: g ] , the return for any trajectory is bounded between 0 and 1 , allowing us to bound (cid:15) above by 1 .
Revised: g ] , the return for any trajectory is bounded between 0 and 1 , allowing us to bound (cid:15) above by 1 .

Original: This leaves α , which is the total variation divergence between π and π old .
Revised: This leaves α , which is the total variation divergence between π and π old .

Original: This divergence may be high if the data collection policy is very far from the current policy, but is low if the data was collected via a recent policy.
Revised: This divergence may be high if the data collection policy is very far from the current policy, but is low if the data was collected via a recent policy.

Original: We can now lower-bound the surrogate objective with the GCSL objective via the following:
Revised: We can now lower-bound the surrogate objective with the GCSL objective via the following:

Original: [Equation]
Revised: [Equation]

Original: g ]

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal line is our goal-relabeling objective: we train the policy to reach goals we reached.
Revised: The ﬁnal line is our goal-relabeling objective: we train the policy to reach goals we reached.

Original: The inequality holds since log π ( τ ) is always negative.
Revised: The inequality holds since log π ( τ ) is always negative.

Original: The inequality is loose by a term related to the probability of not reaching the commanded goal, which we analyze in the section below.
Revised: The inequality is loose by a term related to the probability of not reaching the commanded goal, which we analyze in the section below.

Original: Since the initial state and transition probabilities do not depend on the policy, we can simplify log π ( τ |G ( τ )) as (by absorbing non π -dependent terms into C 2 ):
Revised: Since the initial state and transition probabilities do not depend on the policy, we can simplify log π ( τ |G ( τ )) as (by absorbing non π -dependent terms into C 2 ):

Original: [Equation]
Revised: [Equation]

Original: Combining this result with the bound on the expected return completes the proof:
Revised: Combining this result with the bound on the expected return completes the proof:

Original: [Equation]
Revised: [Equation]

Original: Note that in order for J ( π ) and J GCSL ( π ) to be vacuously zero, the probability of reaching a goal under π old must be non-zero.
Revised: Note that in order for J ( π ) and J GCSL ( π ) to be vacuously zero, the probability of reaching a goal under π old must be non-zero.

Original: This assumption is reasonable, and matches the assumptions on ”exploratory data-collection” and full-support policies that are required by Q-learning and policy gradient convergence guarantees.
Revised: This assumption is reasonable, and matches the assumptions on ”exploratory data-collection” and full-support policies that are required by Q-learning and policy gradient convergence guarantees.

Original: B. Q UANTIFYING THE Q UALITY OF THE A PPROXIMATION
Revised: B. Q UANTIFYING THE Q UALITY OF THE A PPROXIMATION

Original: The tightness of the bound presented above is controlled from two locations: the off-policyness of π old with respect to π and the bound introduced by the lower bound in the theorem.
Revised: The tightness of the bound presented above is controlled from two locations: the off-policyness of π old with respect to π and the bound introduced by the lower bound in the theorem.

Original: The ﬁrst is well-studied in policy gradient methods; in particular, when the data is on-policy, the gap between
Revised: The ﬁrst is well-studied in policy gradient methods; in particular, when the data is on-policy, the gap between

Original: J surr ( π ) and J ( π ) is known to be a policy-independent constant.
Revised: J surr ( π ) and J ( π ) is known to be a policy-independent constant.

Original: We seek to better understand the gap introduced by Equation 6 in the analysis above.
Revised: We seek to better understand the gap introduced by Equation 6 in the analysis above.

Original: We deﬁne P π old ( G ( τ ) (cid:54) =
Revised: We deﬁne P π old ( G ( τ ) (cid:54) =

Original: g ) to be the probability of failure under π old , and additionally deﬁne p wrong ( τ ) and p right ( τ ) to be the conditional distribution of trajectories under π old given that it did not reach and did the commanded goal respectively.
Revised: g ) to be the probability of failure under π old , and additionally deﬁne p wrong ( τ ) and p right ( τ ) to be the conditional distribution of trajectories under π old given that it did not reach and did the commanded goal respectively.

Original: In the following section, we show that the gap introduced by Equation 6 can be controlled by the probability of making a mistake, P π old ( G ( τ ) (cid:54) = g ) , and D TV ( p wrong ( τ ) , p right ( τ )) , a measure of the difference between the distribution of trajectories that must be relabeled and those not.
Revised: In the following section, we show that the gap introduced by Equation 6 can be controlled by the probability of making a mistake, P π old ( G ( τ ) (cid:54) = g ) , and D TV ( p wrong ( τ ) , p right ( τ )) , a measure of the difference between the distribution of trajectories that must be relabeled and those not.

Original: We rewrite Equation 6 as follows:
Revised: We rewrite Equation 6 as follows:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne D to be the Radon-Nikodym derivative of p wrong ( τ ) wrt
Revised: Deﬁne D to be the Radon-Nikodym derivative of p wrong ( τ ) wrt

Original: π old ( τ )
Revised: π old ( τ )

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is afﬁne with respect to the GCSL loss, so the second term is the error we seek to understand.
Revised: The ﬁrst term is afﬁne with respect to the GCSL loss, so the second term is the error we seek to understand.

Original: [Equation]
Revised: [Equation]

Original: The inequality is maintained because of the nonpositivity of log π ( τ ) , and the ﬁnal step holds because π old ( τ ) is a mixture of p wrong ( τ ) and p right ( τ ) .
Revised: The inequality is maintained because of the nonpositivity of log π ( τ ) , and the ﬁnal step holds because π old ( τ ) is a mixture of p wrong ( τ ) and p right ( τ ) .

Original: This derivation shows that the gap between J surr and
Revised: This derivation shows that the gap between J surr and

Original: J GCSL (up to afﬁne consideration) can be controlled by (1) the probability of reaching the wrong goal and (2) the divergence between the conditional distribution of trajectories which did reach the commanded goal (do not need to be relabeled) and those which did not reach the commanded goal (must be relabeled).
Revised: J GCSL (up to afﬁne consideration) can be controlled by (1) the probability of reaching the wrong goal and (2) the divergence between the conditional distribution of trajectories which did reach the commanded goal (do not need to be relabeled) and those which did not reach the commanded goal (must be relabeled).

Original: As either term goes to 0 , this bound becomes tight.
Revised: As either term goes to 0 , this bound becomes tight.

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: In this section, we now prove that sufﬁciently optimizing the GCSL objective over the full state space causes the probability of reaching the wrong goal to be bounded close to 0 , and thus bounds the gap close to 0 .
Revised: In this section, we now prove that sufﬁciently optimizing the GCSL objective over the full state space causes the probability of reaching the wrong goal to be bounded close to 0 , and thus bounds the gap close to 0 .

Original: Suppose we collect trajectories from a policy π old .
Revised: Suppose we collect trajectories from a policy π old .

Original: Following the notation from before, we deﬁne π old ( τ ) =
Revised: Following the notation from before, we deﬁne π old ( τ ) =

Original: E g ∼ p
Revised: E g ∼ p

Original: ( g )
Revised: ( g )

Original: [ π data ( τ | g )] .
Revised: [ π data ( τ | g )] .

Original: For convenience, we deﬁne π ∗ ( a t | s t , g ) ∝ (cid:82) τ \ a t π data ( τ )1( G ( τ )
Revised: For convenience, we deﬁne π ∗ ( a t | s t , g ) ∝ (cid:82) τ \ a t π data ( τ )1( G ( τ )

Original: = g )

Revised: = g )

Original: ( s t ( τ ) =
Revised: ( s t ( τ ) =

Original: s t ) to be the conditional distribution of actions for a given state given that the goal g is reached at the end of the trajectory.
Revised: s t ) to be the conditional distribution of actions for a given state given that the goal g is reached at the end of the trajectory.

Original: If this conditional distribution is not deﬁned, we let π ∗ ( a t | s t , g ) be uniform, so that π ∗ ( a t | s t , g ) is well-deﬁned for all states, goals, and timesteps.
Revised: If this conditional distribution is not deﬁned, we let π ∗ ( a t | s t , g ) be uniform, so that π ∗ ( a t | s t , g ) is well-deﬁned for all states, goals, and timesteps.

Original: The notation for π ∗ is suggestive: in fact, it can be easily shown that under the assumptions of the theorem, full data coverage and deterministic dynamics, the induced policy π ∗ is in fact the optimal policy for maximizing the probability of reaching the goal.
Revised: The notation for π ∗ is suggestive: in fact, it can be easily shown that under the assumptions of the theorem, full data coverage and deterministic dynamics, the induced policy π ∗ is in fact the optimal policy for maximizing the probability of reaching the goal.

Original: To show that the GCSL policy also incurs low error, we provide a coupling argument, similar to Schulman et al.
Revised: To show that the GCSL policy also incurs low error, we provide a coupling argument, similar to Schulman et al.

Original: ; Kakade & Langford (2002); Ross et al.
Revised: ; Kakade & Langford (2002); Ross et al.

Original: Because
Revised: Because

Original: D TV ( π ( a t | s t , g ) , π ∗ ( a t | s t , g )) ≤ (cid:15) , we can deﬁne a (1 − (cid:15) )
Revised: D TV ( π ( a t | s t , g ) , π ∗ ( a t | s t , g )) ≤ (cid:15) , we can deﬁne a (1 − (cid:15) )

Original: -coupled policy pair ( π, π ∗ ) , which take differing actions with probability (cid:15) .
Revised: -coupled policy pair ( π, π ∗ ) , which take differing actions with probability (cid:15) .

Original: By a union bound over all timesteps, the probability that π and π ∗ take any different actions throughout the trajectory is bounded by (cid:15)T , and because of the assumptions of deterministic dynamics, take the same trajectory with probability 1 − (cid:15)T .
Revised: By a union bound over all timesteps, the probability that π and π ∗ take any different actions throughout the trajectory is bounded by (cid:15)T , and because of the assumptions of deterministic dynamics, take the same trajectory with probability 1 − (cid:15)T .

Original: Now, since the two policies take different trajectories with probability at most (cid:15)T , a simple bound shows that the probability that π GCSL reaches the goal is at most (cid:15)T less than π ∗ , leading to our result that the performance gap J ( π ∗ ) − J ( π ) < (cid:15)T .
Revised: Now, since the two policies take different trajectories with probability at most (cid:15)T , a simple bound shows that the probability that π GCSL reaches the goal is at most (cid:15)T less than π ∗ , leading to our result that the performance gap J ( π ∗ ) − J ( π ) < (cid:15)T .

Original: In environments in which every state is reachable from every other state in the desired horizon, this provides a global performance bound indicating that the optimal GCSL policy will reach the goal with probability at least 1 − (cid:15)T .
Revised: In environments in which every state is reachable from every other state in the desired horizon, this provides a global performance bound indicating that the optimal GCSL policy will reach the goal with probability at least 1 − (cid:15)T .

Original: C D IRECTNESS OF P OLICIES L EARNED BY GCSL C. Q UANTITATIVE A NALYSIS
Revised: C D IRECTNESS OF P OLICIES L EARNED BY GCSL C. Q UANTITATIVE A NALYSIS

Original: We measure how direct the policies learned by GCSL are.
Revised: We measure how direct the policies learned by GCSL are.

Original: To do so, we evaluate the following objective:
Revised: To do so, we evaluate the following objective:

Original: [Equation]
Revised: [Equation]

Original: In other words, if the policy reaches the goal in t timesteps, then the performance is γ t , and if the agent does not, then the performance is 0 ( t = ∞ ).
Revised: In other words, if the policy reaches the goal in t timesteps, then the performance is γ t , and if the agent does not, then the performance is 0 ( t = ∞ ).

Original: We choose this metric, as compared to exactly measuring the time taken to reach the goal, because it more cleanly handles the situation when the goal is not reached by the agent.
Revised: We choose this metric, as compared to exactly measuring the time taken to reach the goal, because it more cleanly handles the situation when the goal is not reached by the agent.

Original: We use γ = 0 . 99 .
Revised: We use γ = 0 . 99 .

Original: We additionally train an oracle agent using PPO to maximize a shaped reward r ( s, g ) = − d ( s, g ) for 3 million timesteps.
Revised: We additionally train an oracle agent using PPO to maximize a shaped reward r ( s, g ) = − d ( s, g ) for 3 million timesteps.

Original: While the performance of this oracle agent and GCSL aren’t directly comparable, since the oracle agent has privileged distance information, it serves as an rough upper bound on how fast an agent can reach the goal.
Revised: While the performance of this oracle agent and GCSL aren’t directly comparable, since the oracle agent has privileged distance information, it serves as an rough upper bound on how fast an agent can reach the goal.

Original: It may seem surprising that the GCSL policy learns to reach the goal so quickly, especially given that it optimizes the ﬁnal-timestep optimality objective, and not the shortest-path objective.
Revised: It may seem surprising that the GCSL policy learns to reach the goal so quickly, especially given that it optimizes the ﬁnal-timestep optimality objective, and not the shortest-path objective.

Original: This is due to two factors.
Revised: This is due to two factors.

Original: First, all the environments support the ”reach the goal and stay at goal” behavior – if it were not possible to remain at the goal, then GCSL would not exhibit this type of behavior.
Revised: First, all the environments support the ”reach the goal and stay at goal” behavior – if it were not possible to remain at the goal, then GCSL would not exhibit this type of behavior.

Original: Second, the policy we train ignores the horizon, and at a state, takes the same action at any timestep.
Revised: Second, the policy we train ignores the horizon, and at a state, takes the same action at any timestep.

Original: This inductive bias on the policy prevents it from learning behaviors that remain still for a long period of time, and then go to the goal right before the time horizon is reached.
Revised: This inductive bias on the policy prevents it from learning behaviors that remain still for a long period of time, and then go to the goal right before the time horizon is reached.

Original: [Table]
Revised: [Table]

Original: C. Q UALITATIVE A NALYSIS
Revised: C. Q UALITATIVE A NALYSIS

Original: Figure 13 below shows parts of the state along trajectories produced by GCSL.
Revised: Figure 13 below shows parts of the state along trajectories produced by GCSL.

Original: In Lunar Lander, this state is captured by the rocket’s position[Table] and in 2D Room Navigation it is the agent’s position.
Revised: In Lunar Lander, this state is captured by the rocket’s position and in 2D Room Navigation it is the agent’s position.

Original: While these trajectories do not always take the shortest path to the goal, they do often take fairly direct paths to the goal from the initial position avoiding very roundabout trajectories.
Revised: While these trajectories do not always take the shortest path to the goal, they do often take fairly direct paths to the goal from the initial position avoiding very roundabout trajectories.