Original: L EARNING G OAL -C ONDITIONED V ALUE F UNCTIONS WITH ONE - STEP P ATH REWARDS RATHER THAN G OALR EWARDS
Revised: L EARNING G OAL -C ONDITIONED V ALUE F UNCTIONS WITH ONE - STEP P ATH REWARDS RATHER THAN G OALR EWARDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-goal reinforcement learning (MGRL) addresses tasks where the desired goal state can change for every trial.
Revised: Multi-goal reinforcement learning (MGRL) addresses tasks where the desired goal state can change for every trial.

Original: State-of-the-art algorithms model these problems such that the reward formulation depends on the goals, to associate them with high reward.
Revised: State-of-the-art algorithms model these problems such that the reward formulation depends on the goals, to associate them with high reward.

Original: This dependence introduces additional goal reward resampling steps in algorithms like Hindsight Experience Replay (HER) that reuse trials in which the agent fails to reach the goal by recomputing rewards as if reached states were psuedo-desired goals.
Revised: This dependence introduces additional goal reward resampling steps in algorithms like Hindsight Experience Replay (HER) that reuse trials in which the agent fails to reach the goal by recomputing rewards as if reached states were psuedo-desired goals.

Original: We propose a reformulation of goal-conditioned value functions for MGRL that yields a similar algorithm, while removing the dependence of reward functions on the goal.
Revised: We propose a reformulation of goal-conditioned value functions for MGRL that yields a similar algorithm, while removing the dependence of reward functions on the goal.

Original: Our formulation thus obviates the requirement of reward-recomputation that is needed by HER and its extensions.
Revised: Our formulation thus obviates the requirement of reward-recomputation that is needed by HER and its extensions.

Original: We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning, from tabular domains to deep neural networks for use as a baseline.
Revised: We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning, from tabular domains to deep neural networks for use as a baseline.

Original: Our results are competitive with HER while substantially improving sampling efﬁciency in terms of reward computation.
Revised: Our results are competitive with HER while substantially improving sampling efﬁciency in terms of reward computation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many tasks in robotics require the speciﬁcation of a goal for every trial.
Revised: Many tasks in robotics require the speciﬁcation of a goal for every trial.

Original: For example, a robotic arm can be tasked to move an object to an arbitrary goal position on a table (Gu et al., 2017); a mobile robot can be tasked to navigate to an arbitrary goal landmark on a map (Zhu et al., 2017).
Revised: For example, a robotic arm can be tasked to move an object to an arbitrary goal position on a table (Gu et al., 2017); a mobile robot can be tasked to navigate to an arbitrary goal landmark on a map (Zhu et al., 2017).

Original: The adaptation of reinforcement learning to such goal-conditioned tasks where goal locations can change is called Multi-Goal Reinforcement Learning (MGRL)
Revised: The adaptation of reinforcement learning to such goal-conditioned tasks where goal locations can change is called Multi-Goal Reinforcement Learning (MGRL)

Original: (Plappert et al., 2018).
Revised: (Plappert et al., 2018).

Original: State-of-theart MGRL algorithms (Andrychowicz et al., 2017; Pong et al., 2018) work by estimating goalconditioned value functions (GCVF) which are deﬁned as expected cumulative rewards from start states with speciﬁed goals.
Revised: State-of-theart MGRL algorithms (Andrychowicz et al., 2017; Pong et al., 2018) work by estimating goalconditioned value functions (GCVF) which are deﬁned as expected cumulative rewards from start states with speciﬁed goals.

Original: GCVFs, in turn, are used to compute policies that determine the actions to take at every state.
Revised: GCVFs, in turn, are used to compute policies that determine the actions to take at every state.

Original: State-of-the-art MGRL algorithms use goal-rewards to associate the achievement of goals with high reward.
Revised: To learn GCVFs, MGRL algorithms use goal-reward , deﬁned as the relatively higher reward recieved on reaching the desired goal state.

Original: This conditions reward functions on the desired goals.
Revised: This makes the reward function dependent on the desired goal.

Original: For example, in the Fetch-Push task (Plappert et al., 2018) of moving a block to a given location on a table, every movement incurs a “-1” reward while reaching the desired goal returns a “0” reward.
Revised: For example, in the Fetch-Push task (Plappert et al., 2018) of moving a block to a given location on a table, every movement incurs a “-1” reward while reaching the desired goal returns a “0” goal-reward.

Original: This dependence introduces additional reward resampling steps in algorithms like Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), where trials in which the agent failed to reach the goal are reused by recomputing rewards as if the reached states were pseudo-desired goals.
Revised: This dependence introduces additional reward resampling steps in algorithms like Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), where trials in which the agent failed to reach the goal are reused by recomputing rewards as if the reached states were pseudodesired goals.

Original: Due to the dependence of the reward function on the goal, the relabelling of every pseudo-goal requires an independent reward-recomputation step, which can be expensive.
Revised: Due to the dependence of the reward function on the goal, the relabelling of every pseudo-goal requires an independent reward-recomputation step, which can be expensive.

Original: In this paper, we demonstrate that goal-rewards are avoidable.
Revised: In this paper, we demonstrate that goal-rewards are not needed to learn GCVFs .

Revised: For the Fetch-Push example, the “0” goal-reward does not need to be achieved to learn its GCVF.

Revised: Speciﬁcally, the agent continues to receive “-1” reward even when the block is in the given goal location.

Revised: This reward formulation is atypical in conventional RL because high reward is used to specify the desired goal location.

Original: Let us consider an example to motivate this.
Revised: However, this goal-reward is not necessary in goal-conditioned RL because the goal is already speciﬁed at the start of every episode. We use this idea, to propose a goal-conditioned RL algorithm which learns to reach goals without goal-rewards. This is a counter-intuitive result which is important for understanding goal-conditioned RL. [Figure]

Revised: Dextrous

Revised: We use these for our

Revised: Let us consider another example to motivate the redundancy of goal-rewards.

Original: Consider a student who has moved to a new university.
Revised: Consider a student who has moved to a new campus.

Original: To learn about the campus, the student explores it randomly with no speciﬁc goal in mind.
Revised: To learn about the campus, the student explores it randomly with no speciﬁc goal in mind.

Original: The key intuition here is that the student is not incentivized to ﬁnd speciﬁc goal locations (i.e. no goal-rewards) but is aware of the effort required to travel between points around the university.
Revised: The key intuition here is that the student is not incentivized to ﬁnd speciﬁc goal locations (i.e. no goal-rewards) but is aware of the effort required to travel between points around the university.

Original: When tasked with ﬁnding a goal classroom, the student can chain together these path efforts to ﬁnd the least-effort path to the classroom.
Revised: When tasked with ﬁnding a goal classroom, the student can chain together these path efforts to ﬁnd the least-effort path to the classroom.

Original: Based on this intuition of least-effort paths, we redeﬁne GCVFs to be the expected path-reward that is learned for all possible start-goal pairs.We introduce a one-step loss that assumes one-step paths [Figure] HandManipulateEggFull
Revised: Based on this intuition of least-effort paths, we redeﬁne GCVFs to be the expected path-reward that is learned for all possible start-goal pairs.

Original: HandManipulatePenRotate Dextrous We use to be the paths of maximum reward between pairs wherein the state and goal are adjacent.
Revised: We introduce a one-step loss that assumes one-step paths to be the paths of maximum reward between pairs wherein the state and goal are adjacent.

Original: Under this interpretation, the Bellman equation chooses and chains together one-step paths to ﬁnd longer maximum reward paths.
Revised: Under this interpretation, the Bellman equation chooses and chains together one-step paths to ﬁnd longer maximum reward paths.

Original: Experimentally, we show how this simple reinterpretation, which does not use goal rewards, performs as well as HER while outperforming it in terms of reward computation.
Revised: Experimentally, we show how this simple reinterpretation, which does not use goal rewards, performs as well as HER while outperforming it in terms of reward computation.

Original: We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning (FWRL) (Dhiman et al., 2018) to use parametric function approximators instead of tabular functions.
Revised: We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993) (also called Dynamic Goal Reinforcement learning) to use parametric function approximators instead of tabular functions.

Original: Similar to our re-deﬁnition of GCVFs, FWRL learns a goal-conditioned Floyd-Warshall function that represents path-rewards instead of future-rewards.
Revised: Similar to our re-deﬁnition of GCVFs, FWRL learns a goal-conditioned Floyd-Warshall function that represents path-rewards instead of future-rewards.

Original: We translate FWRL’s compositionality constraints in the space of GCVFs to introduce additional loss terms to the objective.
Revised: We translate FWRL’s compositionality constraints in the space of GCVFs to introduce additional loss terms to the objective.

Original: However, these additional loss terms do not show improvement over the baseline.
Revised: However, these additional loss terms do not show improvement over the baseline.

Original: We conjecture that the compositionality constraints are already captured by other loss terms.
Revised: We conjecture that the compositionality constraints are already captured by other loss terms.

Original: In summary, the contributions of this work are twofold.
Revised: In summary, the contributions of this work are twofold.

Original: Firstly, we reinterpret goal-conditioned value functions as expected path-rewards and introduce one-step loss, thereby removing the dependency of GCVFs on goal-rewards and reward resampling.
Revised: Firstly, we reinterpret goal-conditioned value functions as expected path-rewards and introduce one-step loss, thereby removing the dependency of GCVFs on goal-rewards and reward resampling.

Original: We showcase our algorithm’s improved sample efﬁciency (in terms of reward computation).
Revised: We showcase our algorithm’s improved sample efﬁciency (in terms of reward computation).

Original: We thus extend algorithms like HER to domains where reward recomputation is expensive or infeasible.
Revised: We thus extend algorithms like HER to domains where reward recomputation is expensive or infeasible.

Original: Secondly, we extend the tabular Floyd-Warshal
Revised: Secondly, we extend the tabular Floyd-Warshal

Original: Reinforcement Learning to use deep neural networks.
Revised: Reinforcement Learning to use deep neural networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Goal-conditioned tasks in reinforcement learning have been approached in two ways, depending upon whether the algorithm explicitly separates state and goal representations.
Revised: Goal-conditioned tasks in reinforcement learning have been approached in two ways, depending upon whether the algorithm explicitly separates state and goal representations.

Original: The ﬁrst approach is to use vanilla reinforcement learning algorithms that do not explicitly make this separation (Mirowski et al., 2016; Dosovitskiy & Koltun, 2016; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Mirowski et al., 2018).
Revised: The ﬁrst approach is to use vanilla reinforcement learning algorithms that do not explicitly make this separation (Mirowski et al., 2016; Dosovitskiy & Koltun, 2016; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Mirowski et al., 2018).

Original: These algorithms depend upon neural network architectures to carry the burden of learning the separated representations.
Revised: These algorithms depend upon neural network architectures to carry the burden of learning the separated representations.

Original: The second approach makes this separation explicit via the use of goal-conditioned value functions (Foster & Dayan, 2002;
Revised: The second approach makes this separation explicit via the use of goal-conditioned value functions (Foster & Dayan, 2002;

Original: Sutton et al., 2011).
Revised: Sutton et al., 2011).

Original: Universal Value Function Appoximators (Schaul et al., 2015) propose a network architecture and a factorization technique that separately encodes states and goals, taking advantage of correlations in their representations.
Revised: Universal Value Function Appoximators (Schaul et al., 2015) propose a network architecture and a factorization technique that separately encodes states and goals, taking advantage of correlations in their representations.

Original: Temporal Difference Models combine model-free and model-based RL to gain advantages from both realms by deﬁning and learning a horizon-dependent GCVF.
Revised: Temporal Difference Models combine model-free and model-based RL to gain advantages from both realms by deﬁning and learning a horizon-dependent GCVF.

Original: All these works require the use of goal-dependent reward functions and deﬁne GCVFs as future-rewards instead of path-rewards, contrasting them from our contribution.
Revised: All these works require the use of goal-dependent reward functions and deﬁne GCVFs as future-rewards instead of path-rewards, contrasting them from our contribution.

Original: Unlike our approach, Andrychowicz et al.
Revised: Unlike our approach, Andrychowicz et al.

Original: (2017) propose Hindsight Experience Replay , a technique for resampling state-goal pairs from failed experiences; which leads to faster learning in the presence of sparse rewards.
Revised: (2017) propose Hindsight Experience Replay , a technique for resampling state-goal pairs from failed experiences; which leads to faster learning in the presence of sparse rewards.

Original: In addition to depending on goal rewards, HER also requires the repeated recomputation of the reward function.
Revised: In addition to depending on goal rewards, HER also requires the repeated recomputation of the reward function.

Original: In contrast, we show how removing goal-rewards removes the need for such recomputations.
Revised: In contrast, we show how removing goal-rewards removes the need for such recomputations.

Original: We utilize HER as a baseline in our work.
Revised: We utilize HER as a baseline in our work.

Original: Dhiman et al. (2018) also use the structure of the space of GCVFs to learn.
Revised: Kaelbling (1993) also use the structure of the space of GCVFs to learn.

Original: This work employs compositionality constraints in the space of these functions to accelerate learning in a tabular domain.
Revised: This work employs compositionality constraints in the space of these functions to accelerate learning in a tabular domain.

Original: While their deﬁnition of GCVFs is similar to ours, they still require goal-rewards and do not employ one-step loss.
Revised: While their deﬁnition of GCVFs is similar to ours, the terminal condition is different. We describe this difference in Section 4.

Original: We  extend their work to deep neural networks.
Revised: We also extend their tabular formulation to deep neural networks and evaluate it against the baselines.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A reinforcement learning (RL) problem is formalized as a Markov Decision Process (MDP) (Sutton et al., 1998).
Revised: A reinforcement learning (RL) problem is formalized as a Markov Decision Process (MDP) (Sutton et al., 1998).

Original: A MDP is deﬁned by a ﬁve tuple ( S , A , T, R, γ ) , that governs a sequence of stateaction-reward triples [( s 0 , a 0 , r 0 ) , . . .
Revised: A MDP is deﬁned by a ﬁve tuple ( S , A , T, R, γ ) , that governs a sequence of stateaction-reward triples [( s 0 , a 0 , r 0 ) , . . .

Original: , ( s T , a T , r T )] .
Revised: , ( s T , a T , r T )] .

Original: S is the state space,
Revised: S is the state space,

Original: A is the action space,
Revised: A is the action space,

Original: T ( s, a ) : S ×
Revised: T ( s, a ) : S ×

Original: A → S is the system dynamics, R ( s, a ) : S ×
Revised: A → S is the system dynamics, R ( s, a ) : S ×

Original: A → R is the reward function and γ is the discount factor.
Revised: A → R is the reward function and γ is the discount factor.

Original: In a typical RL problem the transition function T is not given but is known to be static.
Revised: In a typical RL problem the transition function T is not given but is known to be static.

Original: In RL, the objective is to ﬁnd a policy π ( s ) :
Revised: In RL, the objective is to ﬁnd a policy π ( s ) :

Original: S →
Revised: S →

Original: A that maximizes the expected cumulative reward over time, R t = (cid:80) ∞ k = t γ k − t r k , called the return .
Revised: A that maximizes the expected cumulative reward over time, R t = (cid:80) ∞ k = t γ k − t r k , called the return .

Original: The discount factor, γ < 1 , forces the return to be ﬁnite for inﬁnite horizons.
Revised: The discount factor, γ < 1 , forces the return to be ﬁnite for inﬁnite horizons.

Original: Reinforcement learning is typically formulated in single-goal contexts.
Revised: Reinforcement learning is typically formulated in single-goal contexts.

Original: More recently there has been interest in multi-goal problems (Andrychowicz et al., 2017;
Revised: More recently there has been interest in multi-goal problems (Andrychowicz et al., 2017;

Original: Pong et al., 2018; Plappert et al., 2018), which is the focus of this work.
Revised: Pong et al., 2018; Plappert et al., 2018), which is the focus of this work.

Original: 3. D EEP R EINFORCEMENT LEARNING
Revised: 3. D EEP R EINFORCEMENT LEARNING

Original: A number of reinforcement learning algorithms use parametric function approximators to estimate the return in the form of an action-value function, Q ( s, a ) :
Revised: A number of reinforcement learning algorithms use parametric function approximators to estimate the return in the form of an action-value function, Q ( s, a ) :

Original: [Equation]
Revised: [Equation]

Original: where T is the episode length.
Revised: where T is the episode length.

Original: When the policy π is optimal, the Q -function satisﬁes the Bellman equation (Bellman, 1954).
Revised: When the policy π is optimal, the Q -function satisﬁes the Bellman equation (Bellman, 1954).

Original: [Equation]
Revised: [Equation]

Revised: The Q ∗ () function can be learned using Q-learning algorithm (Watkins & Dayan, 1992).

Original: The optimal policy can be computed from Q ∗ greedily, π ∗ ( s t ) =
Revised: The optimal policy can be computed from Q ∗ greedily, π ∗ ( s t ) =

Original: arg max a ∈A Q ∗ ( s t , a ) .
Revised: arg max a ∈A Q ∗ ( s t , a ) .

Original: In Deep Q-Networks (DQN), Mnih et al. (2013) formulate a loss function based on the Bellman equation to approximate the optimal Q ∗ -function using a deep neural network, Q m :
Revised: In Deep Q-Networks (DQN), Mnih et al. (2013) formulate a loss function based on the Bellman equation to approximate the optimal Q ∗ -function using a deep neural network, Q m :

Original: [Equation]
Revised: [Equation]

Original: where y t =
Revised: where y t =

Original: R ( s t , a t )+max a γQ tgt ( s t +1 , a ; θ Q tgt ) , is the target and Q tgt is the target network (Mnih et al., 2015a).
Revised: R ( s t , a t )+max a γQ tgt ( s t +1 , a ; θ Q tgt ) , is the target and Q tgt is the target network (Mnih et al., 2015a).

Original: The target network is a slower-changing copy of the main network that stabilizes learning.
Revised: The target network is a slower-changing copy of the main network that stabilizes learning.

Original: Mnih et al.
Revised: Mnih et al.

Original: (2015a) also employ replay buffers  that store transitions from episodes.
Revised: (2015a) also employ replay buffers (Lin, 1993) that store transitions from episodes.

Original: During training, these transitions are sampled out of order to train the networks in an off-policy manner, avoiding correlation in the samples and thus leading to faster learning.
Revised: During training, these transitions are sampled out of order to train the networks in an off-policy manner, avoiding correlation in the samples and thus leading to faster learning.

Original: In this work, we use an extension of DQN to continuous action spaces called deep deterministic policy-gradients (DDPG) (Lillicrap et al., 2015).
Revised: In this work, we use an extension of DQN to continuous action spaces called deep deterministic policy-gradients (DDPG) (Lillicrap et al., 2015).

Original: DDPG approximates the policy with a policy network π tgt ( s ; θ π ) that replaces the max operator in y t .
Revised: DDPG approximates the policy with a policy network π tgt ( s ; θ π ) that replaces the max operator in y t .

Original: The target thus becomes y t =
Revised: The target thus becomes y t =

Original: R ( s t , a t ) + γQ tgt ( s t +1 , π tgt ( s t +1 ; θ π ); θ Q tgt ) and the loss function changes accordingly:
Revised: [Equation] ) and the loss function changes accordingly:

Original: [Equation]
Revised: [Equation]

Original: 3. M ULTI - GOAL R EINFORCEMENT LEARNING
Revised: 3. M ULTI - GOAL R EINFORCEMENT LEARNING

Original: Multi-Goal Reinforcement Learning (Plappert et al., 2018) focuses on problems where the desired goal state can change for every episode.
Revised: Multi-Goal Reinforcement Learning (Plappert et al., 2018) focuses on problems where the desired goal state can change for every episode.

Original: State-of-the-art MGRL algorithms learn a goal-conditioned value function (GCVF), Q ( s, a, g ) , which is deﬁned similar to the Q -function (5), but with an additional dependence on the desired goal speciﬁcation g ∈ G :
Revised: State-of-the-art MGRL algorithms learn a goal-conditioned value function (GCVF), Q ( s, a, g ) , which is deﬁned similar to the Q -function (5), but with an additional dependence on the desired goal speciﬁcation g ∈ G :

Original: [Equation]
Revised: [Equation]

Original: The structure of the goal speciﬁcation, g ∈ G , can be arbitrary.
Revised: The structure of the goal speciﬁcation, g ∈ G , can be arbitrary.

Original: For example, in a robotic arm experiment, possible goal speciﬁcations include the desired position of the end-effector and the desired joint angles of the robot.
Revised: For example, in a robotic arm experiment, possible goal speciﬁcations include the desired position of the end-effector and the desired joint angles of the robot.

Original: The states and achieved goals are assumed to be an observable part of the Goal-MDP to enable the agent to learn the correspondences between them, [( s 0 , a 0 , g 0 , r 0 ) , . . .
Revised: The states and achieved goals are assumed to be an observable part of the Goal-MDP to enable the agent to learn the correspondences between them, [( s 0 , a 0 , g 0 , r 0 ) , . . .

Original: , ( s T , a T , g T , r T )] .
Revised: , ( s T , a T , g T , r T )] .

Original: Consequently, this Goal-MDP is fully governed by the six tuple ( S , A , G , T, R, γ ) .
Revised: Consequently, this Goal-MDP is fully governed by the six tuple ( S , A , G , T, R, γ ) .

Original: The reward, R ( s, a, g ) : S ×
Revised: The reward, R ( s, a, g ) : S ×

Original: A × G → R , and policy π ( s, g ) : S × G → A are also in turn conditioned on goal information.
Revised: A × G → R , and policy π ( s, g ) : S × G → A are also in turn conditioned on goal information.

Original: Hindsight Experience Replay HER (Andrychowicz et al., 2017) builds upon this deﬁnition of
Revised: [Table] HER (Andrychowicz et al., 2017) builds upon this deﬁnition of

Original: GCVFs (5).
Revised: GCVFs (5).

Original: The main insight of HER is that there is no valuable feedback from the environment when the agent does not reach the goal.
Revised: The main insight of HER is that there is no valuable feedback from the environment when the agent does not reach the goal.

Original: This is further exacerbated when goals are sparse in the state-space.
Revised: This is further exacerbated when goals are sparse in the state-space.

Original: HER solves this problem by reusing these failed experiences for learning.
Revised: HER solves this problem by reusing these failed experiences for learning.

Original: It recomputes a reward for each reached state by relabeling them as pseudo-goals.
Revised: It recomputes a reward for each reached state by relabeling them as pseudo-goals.

Original: In our experiments, we employ HER’s future strategy for pseudo-goal sampling.
Revised: In our experiments, we employ HER’s future strategy for pseudo-goal sampling.

Original: More speciﬁcally, two transitions from the same episode in the replay buffer for times t and t + f are sampled.
Revised: More speciﬁcally, two transitions from the same episode in the replay buffer for times t and t + f are sampled.

Original: The achieved goal g t + f is then assumed to be the pseudo-goal.
Revised: The achieved goal g t + f is then assumed to be the pseudo-goal.

Original: The algorithm generates a new transition for the time step t with the reward re-computed as if g t + f was the desired goal, ( s t , a t , s t +1 , R ( s t , a t , g t + f )) .
Revised: The algorithm generates a new transition for the time step t with the reward re-computed as if g t + f was the desired goal, ( s t , a t , s t +1 , R ( s t , a t , g t + f )) .

Original: HER uses this new transition as a sample.
Revised: HER uses this new transition as a sample.

Original:  P ATH R EWARD -B ASED GCVF S
Revised:  P ATH R EWARD -B ASED GCVF S

Original: In our deﬁnition of the GCVF, instead of making the reward function depend upon the goal, we count accumulated rewards over a path, path-rewards , only if the goal is reached.
Revised: In our deﬁnition of the GCVF, instead of making the reward function depend upon the goal, we count accumulated rewards over a path, path-rewards , only if the goal is reached.

Original: This makes the dependence on the goal explicit instead of implicit to the reward formulation.
Revised: This makes the dependence on the goal explicit instead of implicit to the reward formulation.

Original: Mathematically,
Revised: Mathematically,

Original: [Equation]
Revised: [Equation]

Original: ∃ l such that g l = g 
Revised: ∃ l such that g l = g ∗

Original: [Equation]
Revised: [Equation]

Revised: 6a)

Original: 6a) −∞ otherwise, (6b) where l is the time step when the agent reaches the goal.
Revised: [Equation] otherwise, (6b [Equation] where l is the time step when the agent reaches the goal.

Original: If the agent does not reach the goal, the
Revised: If the agent does not reach the goal, the

Original: GCVF is deﬁned to be negative inﬁnity.
Revised: GCVF is deﬁned to be negative inﬁnity.

Original: This ﬁrst term (6a) is the expected cumulative reward over paths from a given start state to the goal.
Revised: This ﬁrst term (6a) is the expected cumulative reward over paths from a given start state to the goal.

Original: This imposes the constraint that cyclical paths in the state space must have negative cumulative reward for (6a) to yield ﬁnite values.
Revised: This imposes the constraint that cyclical paths in the state space must have negative cumulative reward for (6a) to yield ﬁnite values.

Original: For most practical physical problems, this constraints naturally holds if reward is taken to be some measure of negative energy expenditure.
Revised: For most practical physical problems, this constraints naturally holds if reward is taken to be some measure of negative energy expenditure.

Original: For example, in the robot arm experiment, moving the arm must expend energy (negative reward).
Revised: For example, in the robot arm experiment, moving the arm must expend energy (negative reward).

Original: Achieving a positive reward cycle would translate to generating inﬁnite energy .
Revised: Achieving a positive reward cycle would translate to generating inﬁnite energy .

Original: In all our experiments with this formulation, we use a constant reward of -1 for all states, R P ( s, a ) = − 1 ∀ s, a .
Revised: In all our experiments with this formulation, we use a constant reward of -1 for all states, R P ( s, a ) = − 1 ∀ s, a .

Original: For the cases when the agent does reach the goal at time step l (6a), the Bellman equation takes the following form:
Revised: For the cases when the agent does reach the goal at time step l (6a), the Bellman equation takes the following form:

Original: [Equation]
Revised: [Equation]

Original: Notice that terminal step in this equation is the step to reach the goal.
Revised: Notice that terminal step in this equation is the step to reach the goal.

Original: This differs from Equation (3), where the terminal step is the step at which the episode ends.
Revised: This differs from Equation (3), where the terminal step is the step at which the episode ends.

Original: This formulation is equivalent to the end of episode occuring immediately when the goal is reached.
Revised: This formulation is equivalent to the end of episode occuring immediately when the goal is reached.

Original: This reformulation does not require goal-rewards, which in turn obviates the requirement for pseudo-goals and reward recomputation.
Revised: This reformulation does not require goal-rewards, which in turn obviates the requirement for pseudo-goals and reward recomputation.

Original: One-Step Loss
Revised: One-Step Loss

Original: To enable algorithms like HER to work under this reformulation we need to recognize when the goal is reached (7b).
Revised: To enable algorithms like HER to work under this reformulation we need to recognize when the goal is reached (7b).

Original: This recognition is usually done by the reception of high goal reward.
Revised: This recognition is usually done by the reception of high goal reward.

Original: Instead, we use (7b) as a one-step loss that serves this purpose which is one of our main contributions:
Revised: Instead, we use (7b) as a one-step loss that serves this purpose which is one of our main contributions:

Original: [Equation]
Revised: [Equation]

Original: This loss is based on the assumption that one-step reward is the highest reward between adjacent start-goal states and allows us to estimate the one-step reward between them.
Revised: This loss is based on the assumption that one-step reward is the highest reward between adjacent start-goal states and allows us to estimate the one-step reward between them.

Original: Once learned, it serves as a proxy for the reward to the last step to the goal (7b).
Revised: Once learned, it serves as a proxy for the reward to the last step to the goal (7b).

Original: The Bellman equation (7a), serves as a one-step rollout to combine rewards to ﬁnd maximum reward paths to the goal.
Revised: The Bellman equation (7), serves as a one-step rollout to combine rewards to ﬁnd maximum reward paths to the goal.

Revised: One-step loss is different from the terminal step of Q-Learning because one-step loss is applicable to every transition unlike the terminal step.

Revised: However, one-step loss can be thought of as Q-Learning where every transition is a one-step episode where the achieved goal is the pseudo goal.

Revised: One-step loss also different from the terminal condition in Kaelbling (1993).

Revised: Kaelbling (1993) deﬁnes Q P ∗ ( ., ., . )

Original:  We modify an implementation of HER to include the step-loss term and disable goal rewards for our experiments.
Revised: similar to Eq (7) except the terminal condition is deﬁned as Q P ∗ ( s t , a, g ∗ ) =when g t = g ∗ . Under the stated assumptions, the two deﬁnitions are equivalent but one-step loss is advantageous as it can be applied to every transition unlike the Kaelbling (1993) terminal condition which can be only applied when g t = g ∗ . We modify an implementation of HER to include the step-loss term and disable goal rewards for our experiments.

Original: As in HER, we use the DDPG loss L ddpg while using the “future” goal sampling strategy described in the paper.
Revised: As in HER, we use the DDPG loss L ddpg while using the “future” goal sampling strategy described in the paper.

Original: The details of the resulting algorithm are shown as psuedo-code in
Revised: The details of the resulting algorithm are shown as pseudo-code in

Original: Algorithm 1 in the Appendix.
Revised: Algorithm 1 in the Appendix.

Original: 4. D EEP F LOYD -W ARSHALL R EINFORCEMENT L EARNING
Revised: 4. D EEP F LOYD -W ARSHALL R EINFORCEMENT L EARNING

Original: The GCVF redeﬁnition and one step-loss introduced in this paper are inspired by the tabular formulation of Floyd-Warshall Reinforcement Learning (FWRL) (Dhiman et al., 2018).
Revised: The GCVF redeﬁnition and one step-loss introduced in this paper are inspired by the tabular formulation of Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993).

Original: We extend this algorithm for use with deep neural networks.
Revised: We extend this algorithm for use with deep neural networks.

Original: Unfortunately, the algorithm itself does not show signiﬁcant improvement over the baselines.
Revised: Unfortunately, the algorithm itself does not show signiﬁcant improvement over the baselines.

Original: However, the intuitions gained in its implementation led to the contributions of this paper.
Revised: However, the intuitions gained in its implementation led to the contributions of this paper.

Original: The core contribution of FWRL is a compositionality constraint in the space of GCVFs.
Revised: The core contribution of FWRL is a compositionality constraint in the space of GCVFs.

Original: This constraint states that the optimal Q ∗ value from any state s t to any goal g t + f is greater than or equal to the sum of optimal Q ∗ values via any intermediate state-goal pair ( s w , g w ) :
Revised: This constraint states that the optimal Q ∗ value from any state s t to any goal g t + f is greater than or equal to the sum of optimal Q ∗ values via any intermediate state-goal pair ( s w , g w ) :

Original: [Equation]
Revised: [Equation]

Original: We translate these constraints into loss terms and add them to the DDPG loss L ddpg and one-step loss L step .
Revised: We translate these constraints into loss terms and add them to the DDPG loss L ddpg and one-step loss L step .

Original: Taking cue from Mnih et al.
Revised: Taking cue from Mnih et al.

Original: (2015b)
Revised: (2015b)

Original: , we do not repeat the the main online network Q m in the loss term.
Revised: , we do not repeat the the main online network Q m in the loss term.

Original: We use a target network Q tgt and split the constraint into two loss terms.
Revised: We use a target network Q tgt and split the constraint into two loss terms.

Original: One loss term acts as a lower bound L lo and the other acts as an upper bound L up : 
Revised: One loss term acts as a lower bound L lo and the other acts as an upper bound L up : [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that the above terms differ only by choice of the target and main network.
Revised: Note that the above terms differ only by choice of the target and main network.

Original: FWRL Sampling We augment HER sampling to additionally get the intermediate state-goal pair ( s w , g w ) .
Revised: FWRL Sampling We augment HER sampling to additionally get the intermediate state-goal pair ( s w , g w ) .

Original: Once a transition ( s t , a t , r t , s t +1 ) and a future goal g t + f have been sampled from the same episode, we sample another intermediate state and goal pair ( s w , g w ) such that t ≤ w ≤ t + f . 
Revised: Once a transition ( s t , a t , r t , s t +1 ) and a future goal g t + f have been sampled from the same episode, we sample another intermediate state and goal pair ( s w , g w ) such that t ≤ w ≤ t + f . F e t c h R ea c h   150 .0 . Epochs D i s t an c e f r o m goa l ( m )

Original: E XPERIMENTS
Revised: OursHERFWRL     5 · 104 RewardComputations OursHERFWRL   150 . Epochs S u cc e ss R a t e ( t e s t )

Revised: OursHERFWRL     5 · 104 RewardComputations

Revised: OursHERFWRL

Original: {+F e t c h P u s h    
+}
Revised: {+F e t c h P u s h    
+}

Revised: [Table]

Revised: [Table]

Revised: D i s t an c e f r o m goa l ( m )

Revised: OursHERFWRL

Revised: [Table]

Revised: [Table]

Revised: OursHERFWRL

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Epochs

Revised: S u cc e ss R a t e ( t e s t )

Revised: OursHERFWRL

Revised: [Table]

Revised: [Table]

Revised: RewardComputations

Revised: OursHERFWRL

Revised: F e t c h P

Revised: i ck

Revised: A nd P l a c e

Revised: [Table]

Revised: [Table]

Revised: D i s t an c e f r o m goa l ( m )

Revised: OursHERFWRL     5 ·

Revised: [Table]

Revised: RewardComputations

Revised: OursHERFWRL

Revised: [Table]

Revised: [Table]

Revised: S u

Revised: [Table]

Revised: e

Revised: [Table]

Revised: R a t e ( t e s t )

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: OursHERFWRL

Revised: F e t c h S li de

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: D i s t an c e f r o m goa l ( m )

Original: {+OursHERFWRL 0 .0 .0 .0 .
+}
Revised: {+OursHERFWRL 0 .0 .0 .0 .
+}

Revised: · 106 RewardComputations

Revised: OursHERFWRL    2005 · 10 −0 .0 .

Revised: Epochs

Revised: S u cc e ss R a t e ( t e s t )

Original: {+OursHERFWRL 0 .0 .0 .0 .
+}
Revised: {+OursHERFWRL 0 .0 .0 .0 .
+}

Revised: · 106 RewardComputations

Original:  We use the environments introduced in Plappert et al.
Revised: OursHERFWRL  E XPERIMENTS We use the environments introduced in Plappert et al.

Original: (2018) for our experiments.
Revised: (2018) for our experiments.

Original: Broadly the environments fall in two categories, Fetch and Hand tasks.
Revised: Broadly the environments fall in two categories, Fetch and Hand tasks.

Original: Our results show that learning is possible across all environments without the requirement of goal-reward.
Revised: Our results show that learning is possible across all environments without the requirement of goal-reward.

Revised: More speciﬁcally, the learning happens even when reward given to our algorithm is agent is always “-1” as opposed to the HER formulation where a special goal-reward of “0” is needed for learning to happen.

Original: The Fetch tasks involve a simulation of the Fetch robot’s 7-DOF robotic arm.
Revised: The Fetch tasks involve a simulation of the Fetch robot’s 7-DOF robotic arm.

Original: The four tasks are
Revised: The four tasks are

Original: Reach, Push, Slide and PickAndPlace.
Revised: Reach, Push, Slide and PickAndPlace.

Original: In the Reach task the arm’s end-effector is tasked to reach the a particular 3D coordinate.
Revised: In the Reach task the arm’s end-effector is tasked to reach the a particular 3D coordinate.

Original: In the Push task a block on a table needs to be pushed to a given point on it.
Revised: In the Push task a block on a table needs to be pushed to a given point on it.

Original: In the Slide task a puck must be slid to a desired location.
Revised: In the Slide task a puck must be slid to a desired location.

Original: In the PickAndPlace task a block on a table must be picked up and moved to a 3D coordinate.
Revised: In the PickAndPlace task a block on a table must be picked up and moved to a 3D coordinate.

Original: The Hand tasks use a simulation of the Shadow’s Dexterous Hand to manipulate objects of different shapes and sizes.
Revised: The Hand tasks use a simulation of the Shadow’s Dexterous Hand to manipulate objects of different shapes and sizes.

Original: These tasks are HandReach, HandManipulateBlockRotateXYZ, HandManipulateEggFull and HandManipulatePenRotate.
Revised: These tasks are HandReach, HandManipulateBlockRotateXYZ, HandManipulateEggFull and HandManipulatePenRotate.

Original: In HandReach the hand’s ﬁngertips need to reach a given conﬁguration.
Revised: In HandReach the hand’s ﬁngertips need to reach a given conﬁguration.

Original: In the HandManipulateBlockRotateXYZ, the hand needs to rotate a cubic block to a desired orientation.
Revised: In the HandManipulateBlockRotateXYZ, the hand needs to rotate a cubic block to a desired orientation.

Original: In HandManipulateEggFull, the hand repeats this orientation task with an egg, and in HandManipulatePenRotate, it does so with a pen.
Revised: In HandManipulateEggFull, the hand repeats this orientation task with an egg, and in HandManipulatePenRotate, it does so with a pen.

Original: Snapshots of all these tasks can be found in Figure 1.
Revised: Snapshots of all these tasks can be found in Figure 1.

Original: Note that these tasks use joint angles, not visual input.
Revised: Note that these tasks use joint angles, not visual input.

Original: [Figure]

Original: 5. M ETRICS
Revised: 5. M ETRICS

Original: Similar to prior work, we evaluate all experiments on two metrics: the success rate and the average distance to the goal.
Revised: Similar to prior work, we evaluate all experiments on two metrics: the success rate and the average distance to the goal.

Original: The success rate is deﬁned as the fraction of episodes in which the agent is able to reach the goal within a pre-deﬁned threshold region.
Revised: The success rate is deﬁned as the fraction of episodes in which the agent is able to reach the goal within a pre-deﬁned threshold region.

Original: The metric distance of the goal is the euclidean distance between the achieved goal and the desired goal in meters.
Revised: The metric distance of the goal is the euclidean distance between the achieved goal and the desired goal in meters.

Original: These metrics are plotted against a standard progress measure, the number of training epochs, showing comparable results of our method to the baselines.
Revised: These metrics are plotted against a standard progress measure, the number of training epochs, showing comparable results of our method to the baselines.

Original: To emphasize that our method does not require goal-reward and reward re-computation, we plot these metrics against another progress measure, the number of reward computations used during training.
Revised: To emphasize that our method does not require goal-reward and reward re-computation, we plot these metrics against another progress measure, the number of reward computations used during training.

Original: This includes both the episode rollouts and the reward recomputations during HER sampling.
Revised: This includes both the episode rollouts and the reward recomputations during HER sampling.

Original: 5. H YPER - PARAMETERS CHOICES
Revised: 5. H YPER - PARAMETERS CHOICES

Original: Unless speciﬁed, all our hyper-parameters are identical to the ones used in the HER implementation (Dhariwal et al., 2017).
Revised: Unless speciﬁed, all our hyper-parameters are identical to the ones used in the HER implementation (Dhariwal et al., 2017).

Original: We note two main changes to HER to make the comparison more fair.
Revised: We note two main changes to HER to make the comparison more fair.

Original: Firstly, we use a smaller distance-threshold .
Revised: Firstly, we use a smaller distance-threshold .

Original: The environment used for HER and FWRL returns the goal-reward when the achieved goal is within this threshold of the desired goal.
Revised: The environment used for HER and FWRL returns the goal-reward when the achieved goal is within this threshold of the desired goal.

Original: Because of the absence of goal-rewards, the distance-threshold information is not used by our method.
Revised: Because of the absence of goal-rewards, the distance-threshold information is not used by our method.

Original: We reduce the it to 1cm which is reduction by a factor of 5 compared to HER.
Revised: We reduce the threshold to 1cm which is reduction by a factor of 5 compared to HER.

Original: Secondly, we run all experiments on 6 cores each, while HER uses 19.
Revised: Secondly, we run all experiments on 6 cores each, while HER uses 19.

Original: The batch size used is a function of the number of cores and hence this parameter has a signiﬁcant effect on learning.
Revised: The batch size used is a function of the number of cores and hence this parameter has a signiﬁcant effect on learning.

Original: To ensure fair comparison, all experiments are run with the same hyper-parameters and random seeds to ensure that variations in performance are purely due to differences between the algorithms.[Figure] nd [Figure]
Revised: To ensure fair comparison, all experiments are run with the same hyper-parameters and random seeds to ensure that variations in performance are purely due to differences between the algorithms.

Original: n

Original: [Figure]

Original: t a t e

Original: [Figure]

Original: l

Original: [Figure]

Original: t

Original: [Figure]

Original: s t )

Original: [Figure]

Original: method (red) against HER (blue) (Andrychowicz et al., 2016) and FWRL (green) (Dhiman et al., 2018) for the distance-from-goal and success rate metrics.

Original: Furthermore, both metrics are plotted against two progress measures, the number of training epochs and the number of reward computations.

Original: Measured by distance from the goal, our method performs comparable to or better than the baselines for both progress measurements.

Original: For the success rate, our method underperforms against the baselines.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: All our experimental results are described below, highlighting the strengths and weaknesses of our algorithm.
Revised: All our experimental results are described below, highlighting the strengths and weaknesses of our algorithm.

Original: Across all our experiments, the distance-to-the-goal metric achieves comparable performance to HER without requiring goal-rewards .
Revised: Across all our experiments, the distance-to-the-goal metric achieves comparable performance to HER without requiring goal-rewards .

Original: Fetch Tasks
Revised: Fetch Tasks

Original: The experimental results for Fetch tasks are shown in Figure 2.
Revised: The experimental results for Fetch tasks are shown in Figure 2.

Original: For the Fetch Reach and Push tasks, our method achieves comparable performance to the baselines across both metrics in terms of training epochs and outperforms them in terms of reward recomputations.
Revised: For the Fetch Reach and Push tasks, our method achieves comparable performance to the baselines across both metrics in terms of training epochs and outperforms them in terms of reward recomputations.

Original: Notably, the
Revised: Notably, the

Original: Fetch Pick and Place task trains in signiﬁcantly fewer epochs.
Revised: Fetch Pick and Place task trains in signiﬁcantly fewer epochs.

Original: For the Fetch Slide task the opposite is true.
Revised: For the Fetch Slide task the opposite is true.

Original: We conjecture that Fetch Slide is more sensitive to the distance threshold information, which our method is unable to use.
Revised: We conjecture that Fetch Slide is more sensitive to the distance threshold information, which our method is unable to use.

Original: Hand Tasks For the Hand tasks, the distance to the goal and the success rate show different trends.
Revised: Hand Tasks For the Hand tasks, the distance to the goal and the success rate show different trends.

Original: We show the results in Figure 3.
Revised: We show the results in Figure 3.

Original: When the distance metric is plotted against epochs, we get comparable performance for all tasks; when plotted against reward computations, we outperform all baselines on all tasks except Hand Reach.
Revised: When the distance metric is plotted against epochs, we get comparable performance for all tasks; when plotted against reward computations, we outperform all baselines on all tasks except Hand Reach.

Original: The baselines perform well enough on this task, leaving less scope for signiﬁcant improvement.
Revised: The baselines perform well enough on this task, leaving less scope for signiﬁcant improvement.

Original: These trends do not hold for the success rate metric, on which our method consistently under-performs compared to the baselines across tasks.
Revised: These trends do not hold for the success rate metric, on which our method consistently under-performs compared to the baselines across tasks.

Original: This is surprising, as all algorithms average equally on the distance-from-goal metric.
Revised: This is surprising, as all algorithms average equally on the distance-from-goal metric.

Original: We conjecture that this might be the result of high-distance failure cases of the baselines, i.e. when the baselines fail, they do so at larger distances from the goal.
Revised: We conjecture that this might be the result of high-distance failure cases of the baselines, i.e. when the baselines fail, they do so at larger distances from the goal.

Original: In contrast, we assume our method’s success and failiure cases are closer together.[Figure] [Figure] r a t
Revised: In contrast, we assume our method’s success and failure cases are closer together.

Original: [Figure]

Original: t

Original: [Figure]

Original:  A NALYSIS
Revised:  A NALYSIS

Original: To gain a deeper understanding of the method we perform three additional experiments on different tasks.
Revised: To gain a deeper understanding of the method we perform three additional experiments on different tasks.

Original: We ask the following questions: (a) How important is the step loss?
Revised: We ask the following questions: (a) How important is the step loss?

Original: (b) What happens when the goal-reward is also available to our method?
Revised: (b) What happens when the goal-reward is also available to our method?

Original: (c) How sensitive is HER and our method to the distance-threshold?
Revised: (c) How sensitive is HER and our method to the distance-threshold?

Original: How important is the step loss?
Revised: How important is the step loss?

Original: We choose the Fetch-Push task for this experiment.
Revised: We choose the Fetch-Push task for this experiment.

Original: We run our algorithm with no goal reward and without the step loss on this task.
Revised: We run our algorithm with no goal reward and without the step loss on this task.

Original: Results show that our algorithm fails to reach the goal when the step-loss is removed (Fig. 4a) showing it’s necessity. 
Revised: Results show that our algorithm fails to reach the goal when the step-loss is removed (Fig. 4a) showing its necessity. H and R ea c h    40   · 10 − Epochs

Original: What happens when the goal-reward is also available to our method?
Revised: D i s t an c e f r o m goa l ( m ) OursHERFWRL 0 . 5 · 105 RewardComputations OursHERFWRL    400 . 2 0 .0 .

Revised: Epochs

Revised: S u cc e ss R a t e ( t e s t )

Revised: OursHERFWRL 0 .

Revised: 5 · 105 RewardComputations

Revised: OursHERFWRL

Revised: M an i pu l a t e B l o ck R o t a t e XY

Revised: Z    40 0 .0 .0 .

Revised: Epochs

Revised: D i s t an c e f r o m goa l ( m )

Revised: OursHERFWRL 0 .

Revised: 5 · 105 RewardComputations

Revised: OursHERFWRL    400 .0 .0 .

Revised: Epochs

Revised: S u cc e ss R a t e ( t e s t )

Revised: OursHERFWRL 0 .

Revised: 5 · 105 RewardComputations

Revised: OursHERFWRL

Revised: M an i pu l a t e E gg F u ll    40 0 .0 .0 .

Revised: Epochs

Revised: D i s t an c e f r o m goa l ( m )

Revised: OursHERFWRL 0 .

Revised: 5 · 105 RewardComputations

Revised: OursHERFWRL    400 .0 .

Revised: Epochs

Revised: S u cc e ss R a t e ( t e s t )

Revised: OursHERFWRL 0 . 1 .

Revised: [Figure]

Revised: [Figure]

Revised: P en R o t a t e

Revised: [Figure]

Revised: D i s t an c e f r o m goa l ( m )

Revised: OursHERFWRL 0 .

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: S u

Revised: [Figure]

Revised: e ss R a t e ( t e s t )

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: ( (cid:15) ) with respect to the success-rate and distance-from-goal metrics.

Revised: Both algorithms success-rate is sensitive the threshold while only HER’s distance-from-goal is affected by it.

Revised: What happens when the goal-reward is also available to our method?

Original: We run this experiment on the Fetch PickAndPlace task.
Revised: We run this experiment on the Fetch PickAndPlace task.

Original: We ﬁnd that goal-rewards do not effect the performance of our algorithm further solidifying the avoidability of goal-reward (Fig 4b).
Revised: We ﬁnd that goal-rewards do not affect the performance of our algorithm further solidifying the avoidability of goal-reward (Fig 4b).

Original: How sensitive is HER and our method to the distance-threshold?
Revised: How sensitive is HER and our method to the distance-threshold?

Original: In the absence of goalrewards, our algorithm is not to able capture distance threshold information that decides whether the agent has reached the goal or not.
Revised: In the absence of goalrewards, our algorithm is not to able capture distance threshold information that decides whether the agent has reached the goal or not.

Original: This information is available to HER.
Revised: This information is available to HER.

Original: To understand the sensitivity of our algorithm and HER on this parameter, we vary it over 0.05 (the original HER value), 0.01 and 0.001 meters (Fig. 5).
Revised: To understand the sensitivity of our algorithm and HER on this parameter, we vary it over 0.05 (the original HER value), 0.01 and 0.001 meters (Fig. 5).

Original: Results show that for the success-rate metric, which is itself a function of this parameter, both algorithms are affected equally (Fig. 5a).
Revised: Results show that for the success-rate metric, which is itself a function of this parameter, both algorithms are affected equally (Fig. 5a).

Original: For the distance-from-goal, only HER is affected (Fig. 5b).
Revised: For the distance-from-goal, only HER is affected (Fig. 5b).

Original: This ﬁts our expectations as set up in section 5.2.
Revised: This ﬁts our expectations as set up in section 5.2.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we pose a reinterpretation of goal-conditioned value functions and show that under this paradigm learning is possible in the absence of goal reward.
Revised: In this work we pose a reinterpretation of goal-conditioned value functions and show that under this paradigm learning is possible in the absence of goal reward.

Original: This is a suprising result that runs counter to intuitions that underly most reinforcement learning algorithms.
Revised: This is a surprising result that runs counter to intuitions that underlie most reinforcement learning algorithms.

Original: In future work, we will augment our method to incorporate the distance-threshold information to make the task easier to learn when the threshold is high.
Revised: In future work, we will augment our method to incorporate the distance-threshold information to make the task easier to learn when the threshold is high.

Original: We hope that the experiments and results presented in this paper lead to a broader discussion about the assumptions actually required for learning multi-goal tasks.
Revised: We hope that the experiments and results presented in this paper lead to a broader discussion about the assumptions actually required for learning multi-goal tasks.

Original:  A PPENDIX
Revised:  A PPENDIX

Original: /* By default all states are unreachable */ Initialize networks Q m ( s i , a
Revised: Our algorithm 1 is different from HER Andrychowicz et al.

Original: i , g j ; θ Q ) and π ( s i , s g ; θ π ) ; Copy the main network to target network Q tgt ( s i , a i , g j ; θ Q ) ← Q m ( s i , a i , g j ; θ Q ) ;
Revised: (2016) because it contains additional step-loss term L step at line number 17 which allows the algorithm to learn even when the rewards received are independent of desired goal.

Original: Initialize replay memory M ; for e ← 1 to E do Sample g e ∈ G ; Set t ← 0 ; Observe state s t and achieved goal g t ; /
Revised: Also in HER sampling (line 13), the algorithm recomputes the rewards because the goal is replaced with a pseudo-goal.

Original: * Episode rollout
Revised: Our algorithm does not need reward recomputation because the reward formulation does not depend on the goal and is not affected by choice of pseudo-goal.

Original: */ for t ← 1 to T do Take action a t ← (cid:15) -greedy ( π m ( s t , g ; θ π )) ; Observe s t +1 , g t +1 , r t ; Store ( s t , g t , a t , s t +1 , g t +1 , r t ; g e ) in memory M [ e ] ; / * Train * / for t ← 1 to T do HER sample batch
Revised: Our algorithm is also different from Floyd-Warshall Reinforcement learning because it does not contain L up and L lo terms and contains the additional L step .

Original: B =
Revised: B =

Original: [( s i , g i , a i , s i +1 , g i +1 , r i ; g i + f i ) , . . .
Revised: [( s i , g i , a i , s i +1 , g i +1 , r i ; g i + f i ) , . . .

Original: , ( s b , g b , a b , s b +1 , g b +1 , r b ; g b + f b )] from M ;L ( . . . )
Revised: , ( s b , g b , a b , s b +1 , g b +1 , r b ; g b + f b )] from M ;

Original: = 0 ; for b ∈ 1 to | B | do ( s b , g b , a b , s b +1 , g b +1 , r b , g b + f b
Revised: L

Original: [Equation]
Revised: [Equation]

Original: = B [ b ] ; / *

Original: Step loss

Original: [Equation]
Revised: [Equation]

Original: DDPG loss

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Update gradients 
Revised: Update gradients for

Original: for Q m and π m using loss L ( . . . )

Original: Result: Q m , π m
Revised: Result: Q m , π m