Original: H
Revised: H

Original: INDSIGHT POLICY GRADIENTS
Revised: INDSIGHT POLICY GRADIENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy.
Revised: A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy.

Original: In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals.
Revised: In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals.

Original: In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efﬁcient learning.
Revised: In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efﬁcient learning.

Original: However, reinforcement learning agents have only recently been endowed with such capacity for hindsight.
Revised: However, reinforcement learning agents have only recently been endowed with such capacity for hindsight.

Original: In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms.
Revised: In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms.

Original: Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efﬁciency.
Revised: Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In a traditional reinforcement learning setting, an agent interacts with an environment in a sequence of episodes, observing states and acting according to a policy that ideally maximizes expected cumulative reward.
Revised: In a traditional reinforcement learning setting, an agent interacts with an environment in a sequence of episodes, observing states and acting according to a policy that ideally maximizes expected cumulative reward.

Original: If an agent is required to pursue different goals across episodes, its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal.
Revised: If an agent is required to pursue different goals across episodes, its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal.

Original: This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent.
Revised: This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent.

Original: Learning such goal-conditional behavior has received signiﬁcant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012;
Revised: Learning such goal-conditional behavior has received signiﬁcant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012;

Original: Kupcsik et al., 2013; Deisenroth et al., 2014; Schaul et al., 2015; Zhu et al., 2017; Kober et al., 2012; Ghosh et al., 2018; Mankowitz et al., 2018; Pathak et al., 2018).
Revised: Kupcsik et al., 2013; Deisenroth et al., 2014; Schaul et al., 2015; Zhu et al., 2017; Kober et al., 2012; Ghosh et al., 2018; Mankowitz et al., 2018; Pathak et al., 2018).

Original: Consequently, developing goal-based curricula to facilitate learning has also attracted considerable interest (Fabisch & Metzen, 2014;
Revised: Consequently, developing goal-based curricula to facilitate learning has also attracted considerable interest (Fabisch & Metzen, 2014;

Original: Florensa et al., 2017; Sukhbaatar et al., 2018; Srivastava et al., 2013).
Revised: Florensa et al., 2017; Sukhbaatar et al., 2018; Srivastava et al., 2013).

Original: In hierarchical reinforcement learning, goal-conditional policies may enable agents to plan using subgoals, which abstracts the details involved in lower-level decisions (Oh et al., 2017;
Revised: In hierarchical reinforcement learning, goal-conditional policies may enable agents to plan using subgoals, which abstracts the details involved in lower-level decisions (Oh et al., 2017;

Original: Vezhnevets et al., 2017; Kulkarni et al., 2016; Levy et al., 2017).
Revised: Vezhnevets et al., 2017; Kulkarni et al., 2016; Levy et al., 2017).

Original: In a typical sparse-reward environment , an agent receives a non-zero reward only upon reaching a goal state .
Revised: In a typical sparse-reward environment , an agent receives a non-zero reward only upon reaching a goal state .

Original: Besides being natural, this task formulation avoids the potentially difﬁcult problem of reward shaping , which often biases the learning process towards suboptimal behavior (Ng et al., 1999).
Revised: Besides being natural, this task formulation avoids the potentially difﬁcult problem of reward shaping , which often biases the learning process towards suboptimal behavior (Ng et al., 1999).

Original: Unfortunately, sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms (Andrychowicz et al., 2017; Florensa et al., 2017).
Revised: Unfortunately, sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms (Andrychowicz et al., 2017; Florensa et al., 2017).

Original: For example, consider an agent tasked with traveling between cities.
Revised: For example, consider an agent tasked with traveling between cities.

Original: In a sparse-reward formulation, if reaching a desired destination by chance is unlikely, a learning agent will rarely obtain reward signals.
Revised: In a sparse-reward formulation, if reaching a desired destination by chance is unlikely, a learning agent will rarely obtain reward signals.

Original: At the same time, it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations.
Revised: At the same time, it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations.

Original: In this context, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight .
Revised: In this context, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight .

Original: This capacity was recently introduced by Andrychowicz et al.
Revised: This capacity was recently introduced by Andrychowicz et al.

Original: (2017) to off-policy reinforcement learning algorithms that rely on experience replay (Lin, 1992).
Revised: (2017) to off-policy reinforcement learning algorithms that rely on experience replay (Lin, 1992).

Original: In earlier work, Karkus et al.
Revised: In earlier work, Karkus et al.

Original: introduced hindsight to policy search based on Bayesian optimization (Metzen et al., 2015).
Revised: introduced hindsight to policy search based on Bayesian optimization (Metzen et al., 2015).

Original: In this paper, we demonstrate how hindsight can be introduced to policy gradient methods (Williams, 1986; 1992; Sutton et al., 1999a), generalizing this idea to a successful class of reinforcement learning algorithms (Peters & Schaal, 2008; Duan et al., 2016).
Revised: In this paper, we demonstrate how hindsight can be introduced to policy gradient methods (Williams, 1986; 1992; Sutton et al., 1999a), generalizing this idea to a successful class of reinforcement learning algorithms (Peters & Schaal, 2008; Duan et al., 2016).

Original: In contrast to previous work on hindsight, our approach relies on importance sampling (Bishop, 2013).
Revised: In contrast to previous work on hindsight, our approach relies on importance sampling (Bishop, 2013).

Original: In reinforcement learning, importance sampling has been traditionally employed in order to efﬁciently reuse information obtained by earlier policies during learning (Precup et al., 2000; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Thomas et al., 2015; Munos et al., 2016).
Revised: In reinforcement learning, importance sampling has been traditionally employed in order to efﬁciently reuse information obtained by earlier policies during learning (Precup et al., 2000; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Thomas et al., 2015; Munos et al., 2016).

Original: In comparison, our approach attempts to efﬁciently learn about different goals using information obtained by the current policy for a speciﬁc goal.
Revised: In comparison, our approach attempts to efﬁciently learn about different goals using information obtained by the current policy for a speciﬁc goal.

Original: This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient results.
Revised: This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient results.

Original: In comparison to conventional (goal-conditional) policy gradient estimators, our proposed estimators lead to remarkable sample efﬁciency on a diverse selection of sparse-reward environments.
Revised: In comparison to conventional (goal-conditional) policy gradient estimators, our proposed estimators lead to remarkable sample efﬁciency on a diverse selection of sparse-reward environments.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We denote random variables by upper case letters and assignments to these variables by corresponding lower case letters.
Revised: We denote random variables by upper case letters and assignments to these variables by corresponding lower case letters.

Original: We let Val( X ) denote the set of valid assignments to a random variable X .
Revised: We let Val( X ) denote the set of valid assignments to a random variable X .

Original: We also omit the subscript that typically relates a probability function to random variables when there is no risk of ambiguity.
Revised: We also omit the subscript that typically relates a probability function to random variables when there is no risk of ambiguity.

Original: For instance, we may use p ( x ) to denote p X ( x ) and p ( y ) to denote p Y ( y ) .
Revised: For instance, we may use p ( x ) to denote p X ( x ) and p ( y ) to denote p Y ( y ) .

Original: Consider an agent that interacts with its environment in a sequence of episodes, each of which lasts for exactly T time steps.
Revised: Consider an agent that interacts with its environment in a sequence of episodes, each of which lasts for exactly T time steps.

Original: The agent receives a goal g ∈ Val( G ) at the beginning of each episode.
Revised: The agent receives a goal g ∈ Val( G ) at the beginning of each episode.

Original: At every time step t , the agent observes a state s t ∈ Val( S t ) , receives a reward r ( s t , g ) ∈ R , and chooses an action a t ∈ Val( A t ) .
Revised: At every time step t , the agent observes a state s t ∈ Val( S t ) , receives a reward r ( s t , g ) ∈ R , and chooses an action a t ∈ Val( A t ) .

Original: For simplicity of notation, suppose that
Revised: For simplicity of notation, suppose that

Original: Val( G ) , Val( S t ) , and
Revised: Val( G ) , Val( S t ) , and

Original: Val( A t ) are ﬁnite for every t .
Revised: Val( A t ) are ﬁnite for every t .

Original: In our setting, a goal-conditional policy deﬁnes a probability distribution over actions for every combination of state and goal.
Revised: In our setting, a goal-conditional policy deﬁnes a probability distribution over actions for every combination of state and goal.

Original: The same policy is used to make decisions at every time step.
Revised: The same policy is used to make decisions at every time step.

Original: Let τ = s 1 , a 1 , s 2 , a 2 , . . .
Revised: Let τ = s 1 , a 1 , s 2 , a 2 , . . .

Original: , s T − 1 , a T − 1 , s T denote a trajectory.
Revised: , s T − 1 , a T − 1 , s T denote a trajectory.

Original: We assume that the probability p ( τ | g, θ ) of trajectory τ given goal g and a policy parameterized by θ ∈ Val( Θ ) is given by
Revised: We assume that the probability p ( τ | g, θ ) of trajectory τ given goal g and a policy parameterized by θ ∈ Val( Θ ) is given by

Original: [Equation]
Revised: [Equation]

Original: In contrast to a Markov decision process, this formulation allows the probability of a state transition given an action to change across time steps within an episode.
Revised: In contrast to a Markov decision process, this formulation allows the probability of a state transition given an action to change across time steps within an episode.

Original: More importantly, it implicitly states that the probability of a state transition given an action is independent of the goal pursued by the agent, which we denote by S t +1 ⊥⊥ G | S t , A t .
Revised: More importantly, it implicitly states that the probability of a state transition given an action is independent of the goal pursued by the agent, which we denote by S t +1 ⊥⊥ G | S t , A t .

Original: For every τ , g, and θ , we also assume that p ( τ | g, θ ) is non-zero and differentiable with respect to θ .
Revised: For every τ , g, and θ , we also assume that p ( τ | g, θ ) is non-zero and differentiable with respect to θ .

Original: Assuming that G ⊥⊥ Θ , the expected return η ( θ ) of a policy parameterized by θ is given by
Revised: Assuming that G ⊥⊥ Θ , the expected return η ( θ ) of a policy parameterized by θ is given by

Original: [Equation]
Revised: [Equation]

Original: The action-value function is given by Q θ t ( s, a, g ) =
Revised: The action-value function is given by Q θ t ( s, a, g ) =

Original: E (cid:104)(cid:80)
Revised: E (cid:104)(cid:80)

Original: Tt (cid:48)
Revised: Tt (cid:48)

Original: =
Revised: =

Original: t +1 r
Revised: t +1 r

Original: ( S t (cid:48) , g ) | S t = s, A t = a, g, θ (cid:105) , the value function by V θ t ( s, g ) =
Revised: ( S t (cid:48) , g ) | S t = s, A t = a, g, θ (cid:105) , the value function by V θ t ( s, g ) =

Original: E (cid:2) Q θ t ( s, A t , g ) | S t =
Revised: E (cid:2) Q θ t ( s, A t , g ) | S t =

Original: s, g, θ (cid:3) , and the advantage function by
Revised: s, g, θ (cid:3) , and the advantage function by

Original: A θ t ( s, a, g ) =
Revised: A θ t ( s, a, g ) =

Original: Q θ t ( s, a, g ) − V θ t
Revised: Q θ t ( s, a, g ) − V θ t

Original: ( s, g ) .
Revised: ( s, g ) .

Original:  G OAL - CONDITIONAL POLICY GRADIENTS
Revised:  G OAL - CONDITIONAL POLICY GRADIENTS

Original: This section presents results for goal-conditional policies that are analogous to well-known results for conventional policies (Peters & Schaal, 2008).
Revised: This section presents results for goal-conditional policies that are analogous to well-known results for conventional policies (Peters & Schaal, 2008).

Original: They establish the foundation for the results presented in the next section.
Revised: They establish the foundation for the results presented in the next section.

Original: The corresponding proofs are included in Appendix A for completeness.
Revised: The corresponding proofs are included in Appendix A for completeness.

Original: The objective of policy gradient methods is ﬁnding policy parameters that achieve maximum expected return.
Revised: The objective of policy gradient methods is ﬁnding policy parameters that achieve maximum expected return.

Original: When combined with Monte Carlo techniques (Bishop, 2013), the following result allows pursuing this objective using gradient-based optimization.
Revised: When combined with Monte Carlo techniques (Bishop, 2013), the following result allows pursuing this objective using gradient-based optimization.

Original: Theorem 3.1 (Goal-conditional policy gradient) .
Revised: Theorem 3.1 (Goal-conditional policy gradient) .

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: The following result allows employing a baseline to reduce the variance of the gradient estimator.
Revised: The following result allows employing a baseline to reduce the variance of the gradient estimator.

Original: Theorem 3.2 (Goal-conditional policy gradient, baseline formulation) .
Revised: Theorem 3.2 (Goal-conditional policy gradient, baseline formulation) .

Original: For every t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For every t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Appendix A.7 presents the constant baselines that minimize the (elementwise) variance of the corresponding estimator.
Revised: Appendix A.7 presents the constant baselines that minimize the (elementwise) variance of the corresponding estimator.

Original: However, such baselines are usually impractical to compute (or estimate), and the variance of the estimator may be reduced further by a baseline function that depends on state and goal.
Revised: However, such baselines are usually impractical to compute (or estimate), and the variance of the estimator may be reduced further by a baseline function that depends on state and goal.

Original: Although generally suboptimal, it is typical to let the baseline function b θ t approximate the value function V θ t
Revised: Although generally suboptimal, it is typical to let the baseline function b θ t approximate the value function V θ t

Original: (Greensmith et al., 2004).
Revised: (Greensmith et al., 2004).

Original: Lastly, actor-critic methods may rely on the following result for goal-conditional policies.
Revised: Lastly, actor-critic methods may rely on the following result for goal-conditional policies.

Original: Theorem 3.3 (Goal-conditional policy gradient, advantage formulation) .
Revised: Theorem 3.3 (Goal-conditional policy gradient, advantage formulation) .

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original:  H INDSIGHT POLICY GRADIENTS
Revised:  H INDSIGHT POLICY GRADIENTS

Original: This section presents the novel ideas that introduce hindsight to policy gradient methods.
Revised: This section presents the novel ideas that introduce hindsight to policy gradient methods.

Original: The corresponding proofs can be found in Appendix B.
Revised: The corresponding proofs can be found in Appendix B.

Original: Suppose that the reward r ( s, g ) is known for every combination of state s and goal g , as in previous work on hindsight (Andrychowicz et al., 2017; Karkus et al., 2016).
Revised: Suppose that the reward r ( s, g ) is known for every combination of state s and goal g , as in previous work on hindsight (Andrychowicz et al., 2017; Karkus et al., 2016).

Original: In that case, it is possible to evaluate a trajectory obtained while trying to achieve an original goal g (cid:48) for an alternative goal g .
Revised: In that case, it is possible to evaluate a trajectory obtained while trying to achieve an original goal g (cid:48) for an alternative goal g .

Original: Using importance sampling, this information can be exploited using the following central result.
Revised: Using importance sampling, this information can be exploited using the following central result.

Original: Theorem 4.1 (Every-decision hindsight policy gradient) .
Revised: Theorem 4.1 (Every-decision hindsight policy gradient) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: In the formulation presented above, every reward is multiplied by the ratio between the likelihood of the corresponding trajectory under an alternative goal and the likelihood under the original goal (see Eq. 1).
Revised: In the formulation presented above, every reward is multiplied by the ratio between the likelihood of the corresponding trajectory under an alternative goal and the likelihood under the original goal (see Eq. 1).

Original: Intuitively, every reward should instead be multiplied by a likelihood ratio that only considers the corresponding trajectory up to the previous action.
Revised: Intuitively, every reward should instead be multiplied by a likelihood ratio that only considers the corresponding trajectory up to the previous action.

Original: This intuition underlies the following important result, named after an analogous result for action-value functions by Precup et al.
Revised: This intuition underlies the following important result, named after an analogous result for action-value functions by Precup et al.

Original: Theorem 4.2 (Per-decision hindsight policy gradient) .
Revised: Theorem 4.2 (Per-decision hindsight policy gradient) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: The following lemma allows introducing baselines to hindsight policy gradients (see App.
Revised: The following lemma allows introducing baselines to hindsight policy gradients (see App.

Original: B.4).
Revised: B.4).

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: For every g (cid:48) , t, θ , and associated real-valued (baseline)
Revised: For every g (cid:48) , t, θ , and associated real-valued (baseline)

Original: [Equation]
Revised: [Equation]

Original: Appendix B.7 presents the constant baselines that minimize the (elementwise) variance of the corresponding gradient estimator.
Revised: Appendix B.7 presents the constant baselines that minimize the (elementwise) variance of the corresponding gradient estimator.

Original: By analogy with the conventional practice, we suggest letting the baseline function b θ t approximate the value function V θ t instead.
Revised: By analogy with the conventional practice, we suggest letting the baseline function b θ t approximate the value function V θ t instead.

Original: Importantly, the choice of likelihood ratio in Lemma 4.1 is far from unique.
Revised: Importantly, the choice of likelihood ratio in Lemma 4.1 is far from unique.

Original: However, besides leading to straightforward estimation, it also underlies the advantage formulation presented below.
Revised: However, besides leading to straightforward estimation, it also underlies the advantage formulation presented below.

Original: Theorem 4.3 (Hindsight policy gradient, advantage formulation) .
Revised: Theorem 4.3 (Hindsight policy gradient, advantage formulation) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by ∇
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by ∇

Original: [Equation]
Revised: [Equation]

Original: ( θ ) = (cid:88)
Revised: ( θ ) = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Fortunately, the following result allows approximating the advantage under a goal using a state transition collected while pursuing another goal (see App. D.4).
Revised: Fortunately, the following result allows approximating the advantage under a goal using a state transition collected while pursuing another goal (see App. D.4).

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: For every t and θ , the advantage function A θ t is given by
Revised: For every t and θ , the advantage function A θ t is given by

Original: [Equation]
Revised: [Equation]

Original:  H INDSIGHT GRADIENT ESTIMATORS
Revised:  H INDSIGHT GRADIENT ESTIMATORS

Original: This section details gradient estimation based on the results presented in the previous section.
Revised: This section details gradient estimation based on the results presented in the previous section.

Original: The corresponding proofs can be found in Appendix C.
Revised: The corresponding proofs can be found in Appendix C.

Original: Consider a dataset (batch)
Revised: Consider a dataset (batch)

Original: D
Revised: D

Original: = { ( τ ( i ) , g ( i ) ) } Ni =1 where each trajectory τ ( i ) is obtained using a policy parameterized by θ in an attempt to achieve a goal g ( i ) chosen by the environment.
Revised: = { ( τ ( i ) , g ( i ) ) } Ni =1 where each trajectory τ ( i ) is obtained using a policy parameterized by θ in an attempt to achieve a goal g ( i ) chosen by the environment.

Original: The following result points to a straightforward estimator based on Theorem 4.2.
Revised: The following result points to a straightforward estimator based on Theorem 4.2.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: The per-decision hindsight policy gradient estimator, given by
Revised: The per-decision hindsight policy gradient estimator, given by

Original: [Equation]
Revised: [Equation]

Original: is a consistent and unbiased estimator of the gradient ∇ η ( θ ) of the expected return.
Revised: is a consistent and unbiased estimator of the gradient ∇ η ( θ ) of the expected return.

Original: In preliminary experiments, we found that this estimator leads to unstable learning progress, which is probably due to its potential high variance.
Revised: In preliminary experiments, we found that this estimator leads to unstable learning progress, which is probably due to its potential high variance.

Original: The following result, inspired by weighted importance sampling (Bishop, 2013), represents our attempt to trade variance for bias.
Revised: The following result, inspired by weighted importance sampling (Bishop, 2013), represents our attempt to trade variance for bias.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: The weighted per-decision hindsight policy gradient estimator, given by
Revised: The weighted per-decision hindsight policy gradient estimator, given by

Original: N (cid:88)
Revised: N (cid:88)

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: is a consistent estimator of the gradient ∇ η ( θ ) of the expected return.
Revised: is a consistent estimator of the gradient ∇ η ( θ ) of the expected return.

Original: In simple terms, the likelihood ratio for every combination of trajectory, (alternative) goal, and time step is normalized across trajectories by this estimator.
Revised: In simple terms, the likelihood ratio for every combination of trajectory, (alternative) goal, and time step is normalized across trajectories by this estimator.

Original: In Appendix C.3, we present a result that enables the corresponding consistency-preserving weighted baseline .
Revised: In Appendix C.3, we present a result that enables the corresponding consistency-preserving weighted baseline .

Original: Consider a set G ( i )
Revised: Consider a set G ( i )

Original: =
Revised: =

Original: { g ∈ Val( G ) | exists a t such that r ( s ( i ) t , g ) (cid:54) = 0 } composed of so-called active goals during the i -th episode.
Revised: { g ∈ Val( G ) | exists a t such that r ( s ( i ) t , g ) (cid:54) = 0 } composed of so-called active goals during the i -th episode.

Original: The feasibility of the proposed estimators relies on the fact that only active goals correspond to non-zero terms inside the expectation over goals in Expressions 11 and 12.
Revised: The feasibility of the proposed estimators relies on the fact that only active goals correspond to non-zero terms inside the expectation over goals in Expressions 11 and 12.

Original: In many natural sparse-reward environments, active goals will correspond directly to states visited during episodes (for instance, the cities visited while trying to reach other cities), which enables computing said expectation exactly when the goal distribution is known.
Revised: In many natural sparse-reward environments, active goals will correspond directly to states visited during episodes (for instance, the cities visited while trying to reach other cities), which enables computing said expectation exactly when the goal distribution is known.

Original: The proposed estimators have remarkable properties that differentiate them from previous (weighted) importance sampling estimators for off-policy learning.
Revised: The proposed estimators have remarkable properties that differentiate them from previous (weighted) importance sampling estimators for off-policy learning.

Original: For instance, although a trajectory is often more likely under the original goal than under an alternative goal, in policies with strong optimal substructure, a high probability of a trajectory between the state a and the goal (state) c that goes through the state b may naturally allow for a high probability of the corresponding (sub)trajectory between the state a and the goal (state) b .
Revised: For instance, although a trajectory is often more likely under the original goal than under an alternative goal, in policies with strong optimal substructure, a high probability of a trajectory between the state a and the goal (state) c that goes through the state b may naturally allow for a high probability of the corresponding (sub)trajectory between the state a and the goal (state) b .

Original: In other cases, the (unnormalized) likelihood ratios may become very small for some (alternative) goals after a few time steps across all trajectories.
Revised: In other cases, the (unnormalized) likelihood ratios may become very small for some (alternative) goals after a few time steps across all trajectories.

Original: After normalization, in the worst case, this may even lead to equivalent ratios for such goals for a given time step across all trajectories.
Revised: After normalization, in the worst case, this may even lead to equivalent ratios for such goals for a given time step across all trajectories.

Original: In any case, it is important to note that only likelihood ratios associated to active goals for a given episode will affect the gradient estimate.
Revised: In any case, it is important to note that only likelihood ratios associated to active goals for a given episode will affect the gradient estimate.

Original: Additionally, an original goal will always have (unnormalized) likelihood ratios equal to one for the corresponding episode.
Revised: Additionally, an original goal will always have (unnormalized) likelihood ratios equal to one for the corresponding episode.

Original: Under mild additional assumptions, the proposed estimators also allow using a dataset containing goals chosen arbitrarily (instead of goals drawn from the goal distribution).
Revised: Under mild additional assumptions, the proposed estimators also allow using a dataset containing goals chosen arbitrarily (instead of goals drawn from the goal distribution).

Original: Although this feature is not required by our experiments, we believe that it may be useful to circumvent catastrophic forgetting during curriculum learning (McCloskey & Cohen, 1989;
Revised: Although this feature is not required by our experiments, we believe that it may be useful to circumvent catastrophic forgetting during curriculum learning (McCloskey & Cohen, 1989;

Original: Kirkpatrick et al., 2017).
Revised: Kirkpatrick et al., 2017).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section reports results of an empirical comparison between goal-conditional policy gradient estimators and hindsight policy gradient estimators.
Revised: This section reports results of an empirical comparison between goal-conditional policy gradient estimators and hindsight policy gradient estimators.

Original: 1 Because there are no well-established sparsereward environments intended to test agents under multiple goals, this comparison focuses on our own selection of environments.
Revised: 1 Because there are no well-established sparsereward environments intended to test agents under multiple goals, this comparison focuses on our own selection of environments.

Original: These environments are diverse in terms of stochasticity, state space dimensionality and size, relationship between goals and states, and number of actions.
Revised: These environments are diverse in terms of stochasticity, state space dimensionality and size, relationship between goals and states, and number of actions.

Original: In every one of these environments, the agent receives the remaining number of time steps plus one as a reward for reaching the goal state, which also ends the episode.
Revised: In every one of these environments, the agent receives the remaining number of time steps plus one as a reward for reaching the goal state, which also ends the episode.

Original: In every other situation, the agent receives no reward.
Revised: In every other situation, the agent receives no reward.

Original: Importantly, the weighted per-decision hindsight policy gradient estimator used in our experiments ( HPG ) does not precisely correspond to Expression 12.
Revised: Importantly, the weighted per-decision hindsight policy gradient estimator used in our experiments ( HPG ) does not precisely correspond to Expression 12.

Original: Firstly, the original estimator requires a constant number of time steps T , which would often require the agent to act after the end of an episode in the environments that we consider.
Revised: Firstly, the original estimator requires a constant number of time steps T , which would often require the agent to act after the end of an episode in the environments that we consider.

Original: Secondly, although it is feasible to compute Expression 12 exactly when the goal distribution is known (as explained in Sec. 5), we sometimes subsample the sets of active goals per episode.
Revised: Secondly, although it is feasible to compute Expression 12 exactly when the goal distribution is known (as explained in Sec. 5), we sometimes subsample the sets of active goals per episode.

Original: Furthermore, when including a baseline that approximates the value function, we again consider only active goals, which by itself generally results in an inconsistent estimator ( HPG+B ).
Revised: Furthermore, when including a baseline that approximates the value function, we again consider only active goals, which by itself generally results in an inconsistent estimator ( HPG+B ).

Original: As will become evident in the following sections, these compromised estimators still lead to remarkable sample efﬁciency.
Revised: As will become evident in the following sections, these compromised estimators still lead to remarkable sample efﬁciency.

Original: We assess sample efﬁciency through learning curves and average performance scores, which are obtained as follows.
Revised: We assess sample efﬁciency through learning curves and average performance scores, which are obtained as follows.

Original: After collecting a number of batches (composed of trajectories and goals), each of which enables one step of gradient ascent, an agent undergoes evaluation .
Revised: After collecting a number of batches (composed of trajectories and goals), each of which enables one step of gradient ascent, an agent undergoes evaluation .

Original: During evaluation, the agent interacts with the environment for a number of episodes, selecting actions with maximum probability according to its policy.
Revised: During evaluation, the agent interacts with the environment for a number of episodes, selecting actions with maximum probability according to its policy.

Original: A learning curve shows the average return obtained during each evaluation step, averaged across multiple runs (independent learning procedures).
Revised: A learning curve shows the average return obtained during each evaluation step, averaged across multiple runs (independent learning procedures).

Original: The curves presented in this text also include a 95% bootstrapped conﬁdence interval.
Revised: The curves presented in this text also include a 95% bootstrapped conﬁdence interval.

Original: The average performance is given by the average return across evaluation steps, averaged across runs.
Revised: The average performance is given by the average return across evaluation steps, averaged across runs.

Original: During both training and evaluation, goals are drawn uniformly at random.
Revised: During both training and evaluation, goals are drawn uniformly at random.

Original: Note that there is no held-out set of goals for evaluation, since we are interested in evaluating sample efﬁciency instead of generalization.
Revised: Note that there is no held-out set of goals for evaluation, since we are interested in evaluating sample efﬁciency instead of generalization.

Original: For every combination of environment and batch size, grid search is used to select hyperparameters for each estimator according to average performance scores (after the corresponding standard deviation across runs is subtracted, as suggested by Duan et al.
Revised: For every combination of environment and batch size, grid search is used to select hyperparameters for each estimator according to average performance scores (after the corresponding standard deviation across runs is subtracted, as suggested by Duan et al.

Original: Deﬁnitive results are obtained by using the best hyperparameters found for each estimator in additional runs.
Revised: Deﬁnitive results are obtained by using the best hyperparameters found for each estimator in additional runs.

Original: In this section, we discuss deﬁnitive results for small ( 2 ) and medium ( 16 ) batch sizes.
Revised: In this section, we discuss deﬁnitive results for small ( 2 ) and medium ( 16 ) batch sizes.

Original: More details about our experiments can be found in Appendices E.1 and E.2.
Revised: More details about our experiments can be found in Appendices E.1 and E.2.

Original: Appendix E.3 contains unabridged results and a supplementary empirical study of likelihood ratios (Appendix E.3.6).
Revised: Appendix E.3 contains unabridged results, a supplementary empirical study of likelihood ratios (Appendix E.3.6), and an empirical comparison with hindsight experience replay (Appendix E.3.7).

Original: 6. B IT FLIPPING ENVIRONMENTS
Revised: 6. B IT FLIPPING ENVIRONMENTS

Original: In a bit ﬂipping environment, the agent starts every episode in the same state ( 0 , represented by k bits), and its goal is to reach a randomly chosen state.
Revised: In a bit ﬂipping environment, the agent starts every episode in the same state ( 0 , represented by k bits), and its goal is to reach a randomly chosen state.

Original: The actions allow the agent to toggle (ﬂip) each bit individually.
Revised: The actions allow the agent to toggle (ﬂip) each bit individually.

Original: The maximum number of time steps is k + 1 .
Revised: The maximum number of time steps is k + 1 .

Original: Despite its apparent simplicity, this environment is an ideal testbed for reinforcement learning algorithms intended to deal with sparse rewards, since obtaining a reward by chance is unlikely even for a relatively small k .
Revised: Despite its apparent simplicity, this environment is an ideal testbed for reinforcement learning algorithms intended to deal with sparse rewards, since obtaining a reward by chance is unlikely even for a relatively small k .

Original: Andrychowicz et al.
Revised: Andrychowicz et al.

Original: (2017) employed a similar environment to evaluate their hindsight approach.
Revised: (2017) employed a similar environment to evaluate their hindsight approach.

Original: Figure 4 presents the learning curves for k = 8 .
Revised: Figure 4 presents the learning curves for k = 8 .

Original: Goal-conditional policy gradient estimators with and without an approximate value-function baseline ( GCPG+B and GCPG , respectively) obtain excellent policies and lead to comparable sample efﬁciency.
Revised: Goal-conditional policy gradient estimators with and without an approximate value function baseline ( GCPG+B and GCPG , respectively) obtain excellent policies and lead to comparable sample efﬁciency.

Original: HPG+B obtains excellent policies more than 400 batches earlier than these estimators, but its policies degrade upon additional training.
Revised: HPG+B obtains excellent policies more than 400 batches earlier than these estimators, but its policies degrade upon additional training.

Original: Recall that HPG+B focuses solely on active goals where the theory prescribes an expectation over all goals.
Revised: Recall that HPG+B focuses solely on active goals where the theory prescribes an expectation over all goals.

Original: Accordingly, we hypothesize that the observed instability is caused by a combination of the changes in the state visitation distribution during training with the difﬁculty of ﬁtting the baseline.
Revised: Accordingly, we hypothesize that the observed instability is caused by a combination of the changes in the state visitation distribution during training with the difﬁculty of ﬁtting the baseline.

Original: In comparison, HPG obtains excellent policies as early as HPG+B, but its policies remain remarkably stable upon additional training.
Revised: In comparison, HPG obtains excellent policies as early as HPG+B, but its policies remain remarkably stable upon additional training.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The learning curves for k = 16 are presented in Figure 5.
Revised: The learning curves for k = 16 are presented in Figure 5.

Original: Clearly, both GCPG and GCPG+B are unable to obtain policies that perform better than chance, which is explained by the fact that they rarely incorporate reward signals during training.
Revised: Clearly, both GCPG and GCPG+B are unable to obtain policies that perform better than chance, which is explained by the fact that they rarely incorporate reward signals during training.

Original: Conﬁrming the importance of hindsight, HPG leads to stable and sample efﬁcient learning.
Revised: Conﬁrming the importance of hindsight, HPG leads to stable and sample efﬁcient learning.

Original: Although HPG+B also obtains excellent policies, they deteriorate upon additional training.
Revised: Although HPG+B also obtains excellent policies, they deteriorate upon additional training.

Original: Similar results can be observed for a small batch size (see App. E.3.3).
Revised: Similar results can be observed for a small batch size (see App. E.3.3).

Original: The average performance results documented in Appendix E.3.5 conﬁrm that HPG leads to remarkable sample efﬁciency.
Revised: The average performance results documented in Appendix E.3.5 conﬁrm that HPG leads to remarkable sample efﬁciency.

Original: Importantly, Appendices E.3.1 and E.3.2 present hyperparameter sensitivity graphs suggesting that
Revised: Importantly, Appendices E.3.1 and E.3.2 present hyperparameter sensitivity graphs suggesting that

Original: HPG is less sensitive to hyperparameter settings than the other estimators.
Revised: HPG is less sensitive to hyperparameter settings than the other estimators.

Original: The same two appendices also document an ablation study where the likelihood ratios are removed from HPG, which notably promotes increased hyperparameter sensitivity.
Revised: The same two appendices also document an ablation study where the likelihood ratios are removed from HPG, which notably promotes increased hyperparameter sensitivity.

Original: This study conﬁrms the usefulness of the correction prescribed by importance sampling.
Revised: This study conﬁrms the usefulness of the correction prescribed by importance sampling.

Original: 6. G RID WORLD ENVIRONMENTS
Revised: 6. G RID WORLD ENVIRONMENTS

Original: In the grid world environments that we consider, the agent starts every episode in a (possibly random) position on an 11 × 11 grid, and its goal is to reach a randomly chosen (non-initial) position.
Revised: In the grid world environments that we consider, the agent starts every episode in a (possibly random) position on an 11 × 11 grid, and its goal is to reach a randomly chosen (non-initial) position.

Original: Some of the positions on the grid may contain impassable obstacles (walls).
Revised: Some of the positions on the grid may contain impassable obstacles (walls).

Original: The actions allow the agent to move in the four cardinal directions.
Revised: The actions allow the agent to move in the four cardinal directions.

Original: Moving towards walls causes the agent to remain in its current position.
Revised: Moving towards walls causes the agent to remain in its current position.

Original: A state or goal is represented by a pair of integers between 0 and 10 .
Revised: A state or goal is represented by a pair of integers between 0 and 10 .

Original: The maximum number of time steps is 32 .
Revised: The maximum number of time steps is 32 .

Original: In the empty room environment, the agent starts every episode in the upper left corner of the grid, and there are no walls.
Revised: In the empty room environment, the agent starts every episode in the upper left corner of the grid, and there are no walls.

Original: In the four rooms environment (Sutton et al., 1999b), the agent starts every episode in one of the four corners of the grid (see Fig. 1).
Revised: In the four rooms environment (Sutton et al., 1999b), the agent starts every episode in one of the four corners of the grid (see Fig. 1).

Original: There are walls that partition the grid into four rooms, such that each room provides access to two other rooms through single openings (doors).
Revised: There are walls that partition the grid into four rooms, such that each room provides access to two other rooms through single openings (doors).

Original: With probability 0 .
Revised: With probability 0 .

Original: 2 , the action chosen by the agent is ignored and replaced by a random action.
Revised: 2 , the action chosen by the agent is ignored and replaced by a random action.

Original: Figure 6 shows the learning curves for the empty room environment.
Revised: Figure 6 shows the learning curves for the empty room environment.

Original: Clearly, every estimator obtains excellent policies, although HPG and HPG+B improve sample efﬁciency by at least 200 batches.
Revised: Clearly, every estimator obtains excellent policies, although HPG and HPG+B improve sample efﬁciency by at least 200 batches.

Original: The learning curves for the four rooms environment are presented in Figure 7.
Revised: The learning curves for the four rooms environment are presented in Figure 7.

Original: In this surprisingly challenging environment, every estimator obtains unsatisfactory policies.
Revised: In this surprisingly challenging environment, every estimator obtains unsatisfactory policies.

Original: However, it is still clear that HPG and HPG+B improve sample efﬁciency.
Revised: However, it is still clear that HPG and HPG+B improve sample efﬁciency.

Original: In contrast to the experiments presented in the previous section, HPG+B does not give rise to instability, which we attribute to easier value function estimation.
Revised: In contrast to the experiments presented in the previous section, HPG+B does not give rise to instability, which we attribute to easier value function estimation.

Original: Similar results can be observed for a small batch size (see App. E.3.3).
Revised: Similar results can be observed for a small batch size (see App. E.3.3).

Original: HPG achieves the best average performance in every grid world experiment except for a single case, where the best average performance is achieved by HPG+B (see App. E.3.5).
Revised: HPG achieves the best average performance in every grid world experiment except for a single case, where the best average performance is achieved by HPG+B (see App. E.3.5).

Original: The hyperparameter sensitivity graphs presented in Appendices E.3.1 and E.3.2 once again suggest that HPG is less sensitive to hyperparameter choices, and that ignoring likelihood ratios promotes increased sensitivity (at least in the four rooms environment).
Revised: The hyperparameter sensitivity graphs presented in Appendices E.3.1 and E.3.2 once again suggest that HPG is less sensitive to hyperparameter choices, and that ignoring likelihood ratios promotes increased sensitivity (at least in the four rooms environment).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 6. M S . P AC - MAN ENVIRONMENT
Revised: 6. M S . P AC - MAN ENVIRONMENT

Original: The Ms. Pac-man environment is a variant of the homonymous game for ATARI 2600 (see Fig. 2).
Revised: The Ms. Pac-man environment is a variant of the homonymous game for ATARI 2600 (see Fig. 2).

Original: The agent starts every episode close to the center of the map, and its goal is to reach a randomly chosen (non-initial) position on a 14 × 19 grid deﬁned on the game screen.
Revised: The agent starts every episode close to the center of the map, and its goal is to reach a randomly chosen (non-initial) position on a 14 × 19 grid deﬁned on the game screen.

Original: The actions allow the agent to move in the four cardinal directions for 13 game ticks.
Revised: The actions allow the agent to move in the four cardinal directions for 13 game ticks.

Original: A state is represented by the result of preprocessing a sequence of game screens (images) as described in Appendix E.1.
Revised: A state is represented by the result of preprocessing a sequence of game screens (images) as described in Appendix E.1.

Original: A goal is represented by a pair of integers.
Revised: A goal is represented by a pair of integers.

Original: The maximum number of time steps is 28 , although an episode will also end if the agent is captured by an enemy.
Revised: The maximum number of time steps is 28 , although an episode will also end if the agent is captured by an enemy.

Original: In comparison to the grid world environments considered in the previous section, this environment is additionally challenging due to its high-dimensional states and the presence of enemies.
Revised: In comparison to the grid world environments considered in the previous section, this environment is additionally challenging due to its high-dimensional states and the presence of enemies.

Original: Figure 8 presents the learning curves for a medium batch size.
Revised: Figure 8 presents the learning curves for a medium batch size.

Original: Approximate value function baselines are excluded from this experiment due to the signiﬁcant cost of systematic hyperparameter search.
Revised: Approximate value function baselines are excluded from this experiment due to the signiﬁcant cost of systematic hyperparameter search.

Original: Although HPG obtains better policies during early training, GCPG obtains better ﬁnal policies.
Revised: Although HPG obtains better policies during early training, GCPG obtains better ﬁnal policies.

Original: However, for such a medium batch size, only 3 active goals per episode (out of potentially 28 ) are subsampled for HPG.
Revised: However, for such a medium batch size, only 3 active goals per episode (out of potentially 28 ) are subsampled for HPG.

Original: Although this harsh subsampling brings computational efﬁciency, it also appears to handicap the estimator.
Revised: Although this harsh subsampling brings computational efﬁciency, it also appears to handicap the estimator.

Original: This hypothesis is supported by the fact that HPG outperforms
Revised: This hypothesis is supported by the fact that HPG outperforms

Original: GCPG for a small batch size, when all active goals are used (see Apps.
Revised: GCPG for a small batch size, when all active goals are used (see Apps.

Original: E.3.3 and E.3.5).
Revised: E.3.3 and E.3.5).

Original: Policies obtained using each estimator are illustrated by videos included on the project website.
Revised: Policies obtained using each estimator are illustrated by videos included on the project website.

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: r n
Revised: r n

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 6. F ETCH P USH ENVIRONMENT
Revised: 6. F ETCH P USH ENVIRONMENT

Original: The FetchPush environment is a variant of the environment recently proposed by Plappert et al.
Revised: The FetchPush environment is a variant of the environment recently proposed by Plappert et al.

Original: (2018) to assess goal-conditional policy learning algorithms in a challenging task of practical interest (see
Revised: (2018) to assess goal-conditional policy learning algorithms in a challenging task of practical interest (see

Original: Fig. 3).
Revised: Fig. 3).

Original: In a simulation, a robotic arm with seven degrees of freedom is required to push a randomly placed object (block) towards a randomly chosen position.
Revised: In a simulation, a robotic arm with seven degrees of freedom is required to push a randomly placed object (block) towards a randomly chosen position.

Original: The arm starts every episode in the same conﬁguration.
Revised: The arm starts every episode in the same conﬁguration.

Original: In contrast to the original environment, the actions in our variant allow increasing the desired velocity of the gripper along each of two orthogonal directions by ± 0 .
Revised: In contrast to the original environment, the actions in our variant allow increasing the desired velocity of the gripper along each of two orthogonal directions by ± 0 .

Original: 1 or ± 1 , leading to a total of eight actions.
Revised: 1 or ± 1 , leading to a total of eight actions.

Original: A state is represented by a 28 -dimensional real vector that contains the following information: positions of the gripper and block; rotational and positional velocities of the gripper and block; relative position of the block with respect to the gripper; state of the gripper; and current desired velocity of the gripper along each direction.
Revised: A state is represented by a 28 -dimensional real vector that contains the following information: positions of the gripper and block; rotational and positional velocities of the gripper and block; relative position of the block with respect to the gripper; state of the gripper; and current desired velocity of the gripper along each direction.

Original: A goal is represented by three coordinates.
Revised: A goal is represented by three coordinates.

Original: The maximum number of time steps is 50 .
Revised: The maximum number of time steps is 50 .

Original: Figure 9 presents the learning curves for a medium batch size.
Revised: Figure 9 presents the learning curves for a medium batch size.

Original: HPG obtains good policies after a reasonable number of batches, in sharp contrast to GCPG.
Revised: HPG obtains good policies after a reasonable number of batches, in sharp contrast to GCPG.

Original: For such a medium batch size, onlyactive goals per episode (out of potentially 50 ) are subsampled for HPG, showing that subsampling is a viable alternative to reduce the computational cost of hindsight.
Revised: For such a medium batch size, onlyactive goals per episode (out of potentially 50 ) are subsampled for HPG, showing that subsampling is a viable alternative to reduce the computational cost of hindsight.

Original: Similar results are observed for a small batch size, when all active goals are used (see Apps.
Revised: Similar results are observed for a small batch size, when all active goals are used (see Apps.

Original: E.3.3 and E.3.5).
Revised: E.3.3 and E.3.5).

Original: Policies obtained using each estimator are illustrated by videos included on the project website.
Revised: Policies obtained using each estimator are illustrated by videos included on the project website.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced techniques that enable learning goal-conditional policies using hindsight.
Revised: We introduced techniques that enable learning goal-conditional policies using hindsight.

Original: In this context, hindsight refers to the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended.
Revised: In this context, hindsight refers to the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended.

Original: Prior to our work, hindsight has been limited to off-policy reinforcement learning algorithms that rely on experience replay (Andrychowicz et al., 2017) and policy search based on Bayesian optimization (Karkus et al., 2016).
Revised: Prior to our work, hindsight has been limited to off-policy reinforcement learning algorithms that rely on experience replay (Andrychowicz et al., 2017) and policy search based on Bayesian optimization (Karkus et al., 2016).

Original: In addition to the fundamental hindsight policy gradient, our technical results include its baseline and advantage formulations.
Revised: In addition to the fundamental hindsight policy gradient, our technical results include its baseline and advantage formulations.

Original: These results are based on a self-contained goal-conditional policy framework that is also introduced in this text.
Revised: These results are based on a self-contained goal-conditional policy framework that is also introduced in this text.

Original: Besides the straightforward estimator built upon the per-decision hindsight policy gradient, we also presented a consistent estimator inspired by weighted importance sampling, together with the corresponding baseline formulation.
Revised: Besides the straightforward estimator built upon the per-decision hindsight policy gradient, we also presented a consistent estimator inspired by weighted importance sampling, together with the corresponding baseline formulation.

Original: A variant of this estimator leads to remarkable comparative sample efﬁciency on a diverse selection of sparsereward environments, especially in cases where direct reward signals are extremely difﬁcult to obtain.
Revised: A variant of this estimator leads to remarkable comparative sample efﬁciency on a diverse selection of sparsereward environments, especially in cases where direct reward signals are extremely difﬁcult to obtain.

Original: This crucial feature allows natural task formulations that require just trivial reward shaping.
Revised: This crucial feature allows natural task formulations that require just trivial reward shaping.

Original: The main drawback of hindsight policy gradient estimators appears to be their computational cost, which is directly related to the number of active goals in a batch.
Revised: The main drawback of hindsight policy gradient estimators appears to be their computational cost, which is directly related to the number of active goals in a batch.

Original: This issue may be mitigated by subsampling active goals, which generally leads to inconsistent estimators.
Revised: This issue may be mitigated by subsampling active goals, which generally leads to inconsistent estimators.

Original: Fortunately, our experiments suggest that this is a viable alternative.
Revised: Fortunately, our experiments suggest that this is a viable alternative.

Original: Note that the success of hindsight experience replay also depends on an active goal subsampling heuristic (Andrychowicz et al., 2017, Sec. 4.5).
Revised: Note that the success of hindsight experience replay also depends on an active goal subsampling heuristic (Andrychowicz et al., 2017, Sec. 4.5).

Original: The inconsistent hindsight policy gradient estimator with a value-function baseline employed in our experiments sometimes leads to unstable learning, which is likely related to its unprincipled focus on active goals.
Revised: The inconsistent hindsight policy gradient estimator with a value function baseline employed in our experiments sometimes leads to unstable learning, which is likely related to the difﬁculty of ﬁtting such a value function without hindsight.

Original: This behavior requires more investigation.
Revised: This behavior requires more investigation.

Original: Further experiments are also required to evaluate hindsight on dense-reward environments.
Revised: Further experiments are also required to evaluate hindsight on dense-reward environments.

Original: There are many possibilities for future work besides integrating hindsight policy gradients into systems that rely on goal-conditional policies: deriving additional estimators; implementing and evaluating hindsight (advantage) actor-critic methods; assessing whether hindsight policy gradients can successfully circumvent catastrophic forgetting during curriculum learning of goal-conditional policies; approximating the reward function to reduce required supervision employing hindsight experience replay to train value-function baselines; analysing the variance of the proposed estimators studying the impact of active goal subsampling; evaluating every technique on continuous action spaces; and comparing with techniques based on hindsight experience replay.
Revised: There are many possibilities for future work besides integrating hindsight policy gradients into systems that rely on goal-conditional policies: deriving additional estimators; implementing and evaluating hindsight (advantage) actor-critic methods; assessing whether hindsight policy gradients can successfully circumvent catastrophic forgetting during curriculum learning of goal-conditional policies approximating the reward function to reduce required supervision employing hindsight experience replay to train value function baselines ; analysing the variance of the proposed estimators studying the impact of active goal subsampling and evaluating every technique on continuous action spaces.

Original: A G OAL - CONDITIONAL POLICY GRADIENTS
Revised: A G OAL - CONDITIONAL POLICY GRADIENTS

Original: This appendix contains proofs related to the results presented in Section 3: Theorem 3.1 (App. A.2),
Revised: This appendix contains proofs related to the results presented in Section 3: Theorem 3.1 (App. A.2),

Original: Theorem 3.2 (App.
Revised: Theorem 3.2 (App.

Original: A.4)
Revised: A.4)

Original: and Theorem 3.3 (App. A.6).
Revised: and Theorem 3.3 (App. A.6).

Original: Appendix A.7 presents optimal constant baselines for goal-conditional policies.
Revised: Appendix A.7 presents optimal constant baselines for goal-conditional policies.

Original: The remaining subsections contain auxiliary results.
Revised: The remaining subsections contain auxiliary results.

Original: A. T HEOREM A. 
Revised: A. T HEOREM A.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The partial derivative ∂η ( θ ) /∂θ
Revised: The partial derivative ∂η ( θ ) /∂θ

Original: j of the expected return η ( θ ) with respect to θ j is given by
Revised: j of the expected return η ( θ ) with respect to θ j is given by

Original: [Equation]
Revised: [Equation]

Original: The likelihood-ratio trick allows rewriting the previous equation as
Revised: The likelihood-ratio trick allows rewriting the previous equation as

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: A. T HEOREM 3. 
Revised: A. T HEOREM 3.

Original: Theorem 3.1 (Goal-conditional policy gradient) .
Revised: Theorem 3.1 (Goal-conditional policy gradient) .

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: [Equation]
Revised: [Equation]

Original: the partial derivative ∂η ( θ ) /∂θ j of η ( θ ) with respect to θ j is given by
Revised: the partial derivative ∂η ( θ ) /∂θ j of η ( θ ) with respect to θ j is given by

Original: [Equation]
Revised: [Equation]

Original: The previous equation can be rewritten as
Revised: The previous equation can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Let c denote an expectation inside Eq.
Revised: Let c denote an expectation inside Eq.

Original: 19 for t (cid:48) ≥ t .
Revised: 19 for t (cid:48) ≥ t .

Original: In that case, A t (cid:48) ⊥⊥ S t | S t (cid:48) , G, Θ , and so
Revised: In that case, A t (cid:48) ⊥⊥ S t | S t (cid:48) , G, Θ , and so

Original: [Equation]
Revised: [Equation]

Original: Reversing the likelihood-ratio trick,
Revised: Reversing the likelihood-ratio trick,

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the terms where t (cid:48) ≥ t can be dismissed from Eq. 19, leading to
Revised: the terms where t (cid:48) ≥ t can be dismissed from Eq. 19, leading to

Original: [Equation]
Revised: [Equation]

Original: The previous equation can be conveniently rewritten as
Revised: The previous equation can be conveniently rewritten as

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: A.

Original: L EMMA A.
Revised: L EMMA A.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For every j, t, θ , and associated real-valued (baseline) function b θ t ,
Revised: For every j, t, θ , and associated real-valued (baseline) function b θ t ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Letting c denote an expectation inside Eq.
Revised: Letting c denote an expectation inside Eq.

Original: [Equation]
Revised: [Equation]

Original: Reversing the likelihood-ratio trick,
Revised: Reversing the likelihood-ratio trick,

Original: [Equation]
Revised: [Equation]

Original: A. T HEOREM 3. 
Revised: A. T HEOREM 3.

Original: Theorem 3.2 (Goal-conditional policy gradient, baseline formulation) .
Revised: Theorem 3.2 (Goal-conditional policy gradient, baseline formulation) .

Original: For every t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For every t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result is obtained by subtracting Eq.
Revised: The result is obtained by subtracting Eq.

Original: 24 from Eq.
Revised: 24 from Eq.

Original: Importantly, for every combination of θ and t , it would also be possible to have a distinct baseline function for each parameter in θ .
Revised: Importantly, for every combination of θ and t , it would also be possible to have a distinct baseline function for each parameter in θ .

Original: A.
Revised: A.

Original: L EMMA A.
Revised: L EMMA A.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: 23 and rearranging terms,
Revised: 23 and rearranging terms,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:88) st +1: T p (
Revised: ) (cid:88) st +1: T p (

Original: [Equation]
Revised: [Equation]

Original: t +1: T
Revised: t +1: T

Original: [Equation]
Revised: [Equation]

Original: a t ,g, θ ) T (cid:88) t (cid:48) =
Revised: a t ,g, θ ) T (cid:88) t (cid:48) =

Original: t +1 r
Revised: t +1 r

Original: ( s t (cid:48) ,g ) .
Revised: ( s t (cid:48) ,g ) .

Original: By the deﬁnition of action-value function,
Revised: By the deﬁnition of action-value function,

Original: [Equation]
Revised: [Equation]

Original: A. T HEOREM 3. 
Revised: A. T HEOREM 3.

Original: Theorem 3.3 (Goal-conditional policy gradient, advantage formulation) .
Revised: Theorem 3.3 (Goal-conditional policy gradient, advantage formulation) .

Original: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: The gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result is obtained by choosing b θ t =
Revised: The result is obtained by choosing b θ t =

Original: V θ t and subtracting Eq.
Revised: V θ t and subtracting Eq.

Original: 24 from Eq.
Revised: 24 from Eq.

Original: A. T HEOREM A. 
Revised: A. T HEOREM A.

Original: For arbitrary j and θ , consider the following deﬁnitions of f and h .
Revised: For arbitrary j and θ , consider the following deﬁnitions of f and h .

Original: [Equation]
Revised: [Equation]

Original: For every b j ∈ R , using Theorem 3.1 and the fact that E [ h ( T , G ) | θ ] = 0 by Lemma A. 1 ,
Revised: For every b j ∈ R , using Theorem 3.1 and the fact that E [ h ( T , G ) | θ ] = 0 by Lemma A. 1 ,

Original: [Equation]
Revised: [Equation]

Original: Theorem A.2.
Revised: Theorem A.2.

Original: Assuming Var [ h ( T , G ) | θ ] > 0 , the (optimal constant baseline) b j that minimizes
Revised: Assuming Var [ h ( T , G ) | θ ] > 0 , the (optimal constant baseline) b j that minimizes

Original: Var [ f ( T , G ) − b j h ( T , G ) | θ ] is given by
Revised: Var [ f ( T , G ) − b j h ( T , G ) | θ ] is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result is an application of Lemma D.4.
Revised: The result is an application of Lemma D.4.

Original: B H INDSIGHT POLICY GRADIENTS
Revised: B H INDSIGHT POLICY GRADIENTS

Original: This appendix contains proofs related to the results presented in Section 4: Theorem 4.1 (App. B.1),
Revised: This appendix contains proofs related to the results presented in Section 4: Theorem 4.1 (App. B.1),

Original: Theorem 4.2 (App.
Revised: Theorem 4.2 (App.

Original: B.2), Lemma 4.1 (App. B.3), Theorem B.1 (App. B.4), and Theorem 4.3 (App.
Revised: B.2), Lemma 4.1 (App. B.3), Theorem B.1 (App. B.4), and Theorem 4.3 (App.

Original: B.6).
Revised: B.6).

Original: Appendix B.7 presents optimal constant baselines for hindsight policy gradients.
Revised: Appendix B.7 presents optimal constant baselines for hindsight policy gradients.

Original: Appendix B. contains an auxiliary result.
Revised: Appendix B. contains an auxiliary result.

Original: B. T HEOREM 4. 
Revised: B. T HEOREM 4.

Original: The following theorem relies on importance sampling, a traditional technique used to obtain estimates related to a random variable X ∼ p using samples from an arbitrary positive distribution q .
Revised: The following theorem relies on importance sampling, a traditional technique used to obtain estimates related to a random variable X ∼ p using samples from an arbitrary positive distribution q .

Original: This technique relies on the following equalities:
Revised: This technique relies on the following equalities:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1 (Every-decision hindsight policy gradient) .
Revised: Theorem 4.1 (Every-decision hindsight policy gradient) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from Theorem 3.1, importance sampling allows rewriting the partial derivative ∂η ( θ ) /∂θ
Revised: Starting from Theorem 3.1, importance sampling allows rewriting the partial derivative ∂η ( θ ) /∂θ

Original: j as ∂ ∂θ
Revised: j as ∂ ∂θ

Original: [Equation]
Revised: [Equation]

Original: Using Equation 1,
Revised: Using Equation 1,

Original: [Equation]
Revised: [Equation]

Original: B. T HEOREM 4. 
Revised: B. T HEOREM 4.

Original: Theorem 4.2 (Per-decision hindsight policy gradient) .
Revised: Theorem 4.2 (Per-decision hindsight policy gradient) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: [Equation]
Revised: [Equation]

Original: the partial derivative ∂η ( θ ) /∂θ
Revised: the partial derivative ∂η ( θ ) /∂θ

Original: j can be rewritten as
Revised: j can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: If we split every trajectory into states and actions before and after t (cid:48)
Revised: If we split every trajectory into states and actions before and after t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: then ∂η ( θ ) /∂θ
Revised: then ∂η ( θ ) /∂θ

Original: j is given by
Revised: j is given by

Original: [Equation]
Revised: [Equation]

Original: where z is deﬁned by
Revised: where z is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Using Lemma D.2 and canceling terms,
Revised: Using Lemma D.2 and canceling terms,

Original: [Equation]
Revised: [Equation]

Original: Using Lemma D.2 once again
Revised: Using Lemma D.2 once again

Original: [Equation]
Revised: [Equation]

Original: Using the fact that S t (cid:48) ⊥⊥ G | S 1: t
Revised: Using the fact that S t (cid:48) ⊥⊥ G | S 1: t

Original: [Equation]
Revised: [Equation]

Original: Θ , z
Revised: Θ , z

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Substituting z into Expression 38 and returning to an expectation over trajectories,
Revised: Substituting z into Expression 38 and returning to an expectation over trajectories,

Original: [Equation]
Revised: [Equation]

Original: t (cid:48)
Revised: t (cid:48)

Original: [Equation]
Revised: [Equation]

Original:   t (cid:48)
Revised:   t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:89) k
Revised: (cid:89) k

Original: = p ( a k | s k
Revised: = p ( a k | s k

Original: [Equation]
Revised: [Equation]

Original: g, θ ) p ( a k | s k , g (cid:48) , θ ) 
Revised: g, θ ) p ( a k | s k , g (cid:48) , θ ) 

Original:  r ( s t (cid:48) , g ) .
Revised:  r ( s t (cid:48) , g ) .

Original: B. L EMMA 4. 
Revised: B. L EMMA 4.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: For every g (cid:48) , t, θ , and associated real-valued (baseline) function b θ t
Revised: For every g (cid:48) , t, θ , and associated real-valued (baseline) function b θ t

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let c denote the j -th element of the vector in the left-hand side of Eq. 8, such that
Revised: Let c denote the j -th element of the vector in the left-hand side of Eq. 8, such that

Original: [Equation]
Revised: [Equation]

Original: Using Lemma D
Revised: Using Lemma D

Original: [Equation]
Revised: [Equation]

Original: 1 and writing the expectations explicitly,
Revised: 1 and writing the expectations explicitly,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Canceling terms, using Lemma D.1 once again
Revised: Canceling terms, using Lemma D.1 once again

Original: [Equation]
Revised: [Equation]

Original: and reversing the likelihood-ratio trick,
Revised: and reversing the likelihood-ratio trick,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Pushing constants outside the summation over actions at time step t ,
Revised: Pushing constants outside the summation over actions at time step t ,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: B. T HEOREM B. 
Revised: B. T HEOREM B.

Original: Theorem B.1 (Hindsight policy gradient, baseline formulation) .
Revised: Theorem B.1 (Hindsight policy gradient, baseline formulation) .

Original: For every g (cid:48) , t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For every g (cid:48) , t, θ , and associated real-valued (baseline) function b θ t , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result is obtained by subtracting Eq.
Revised: The result is obtained by subtracting Eq.

Original: 8 from Eq.
Revised: 8 from Eq.

Original: Importantly, for every combination of θ and t , it would also be possible to have a distinct baseline function for each parameter in θ .
Revised: Importantly, for every combination of θ and t , it would also be possible to have a distinct baseline function for each parameter in θ .

Original: B. L EMMA B. 
Revised: B. L EMMA B.

Original: Lemma B.1 (Hindsight policy gradient, action-value formulation) .
Revised: Lemma B.1 (Hindsight policy gradient, action-value formulation) .

Original: For an arbitrary goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by ∇
Revised: For an arbitrary goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by ∇

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: [Equation]
Revised: [Equation]

Original: the partial derivative ∂η ( θ ) /∂θ
Revised: the partial derivative ∂η ( θ ) /∂θ

Original: j can be written as
Revised: j can be written as

Original: [Equation]
Revised: [Equation]

Original: Using importance sampling
Revised: Using importance sampling

Original: [Equation]
Revised: [Equation]

Original: for an arbitrary goal g (cid:48)
Revised: for an arbitrary goal g (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: Using Lemma D
Revised: Using Lemma D

Original: [Equation]
Revised: [Equation]

Original: 1 and rewriting the previous equation using expectations,
Revised: 1 and rewriting the previous equation using expectations,

Original: [Equation]
Revised: [Equation]

Original: B. T HEOREM 4. 
Revised: B. T HEOREM 4.

Original: Theorem 4.3 (Hindsight policy gradient, advantage formulation) .
Revised: Theorem 4.3 (Hindsight policy gradient, advantage formulation) .

Original: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by
Revised: For an arbitrary (original) goal g (cid:48) , the gradient ∇ η ( θ ) of the expected return with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result is obtained by choosing b θ t =
Revised: The result is obtained by choosing b θ t =

Original: V θ t and subtracting Eq.
Revised: V θ t and subtracting Eq.

Original: 44 from Eq.
Revised: 44 from Eq.

Original: B. T HEOREM B. 
Revised: B. T HEOREM B.

Original: For arbitrary g (cid:48) , j, and θ , consider the following deﬁnitions of f and h .
Revised: For arbitrary g (cid:48) , j, and θ , consider the following deﬁnitions of f and h .

Original: [Equation]
Revised: [Equation]

Original: For every b j ∈ R , using Theorem 4.2 and the fact that E [ h ( T ) | g (cid:48) , θ ] = 0 by Lemma 4.1,
Revised: For every b j ∈ R , using Theorem 4.2 and the fact that E [ h ( T ) | g (cid:48) , θ ] = 0 by Lemma 4.1,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Theorem B.2.
Revised: Theorem B.2.

Original: Assuming Var [ h ( T ) | g (cid:48) , θ ] > 0 , the (optimal constant baseline) b j that minimizes
Revised: Assuming Var [ h ( T ) | g (cid:48) , θ ] > 0 , the (optimal constant baseline) b j that minimizes

Original: Var [ f ( T ) − b j h ( T ) | g (cid:48) , θ ] is given by
Revised: Var [ f ( T ) − b j h ( T ) | g (cid:48) , θ ] is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: The result is an application of Lemma D.4.
Revised: The result is an application of Lemma D.4.

Original: C H INDSIGHT GRADIENT ESTIMATORS
Revised: C H INDSIGHT GRADIENT ESTIMATORS

Original: This appendix contains proofs related to the estimators presented in Section 5: Theorem 5.1 (App. C.1) and Theorem 5.2 (App. C.2).
Revised: This appendix contains proofs related to the estimators presented in Section 5: Theorem 5.1 (App. C.1) and Theorem 5.2 (App. C.2).

Original: Appendix C.3 presents a result that enables a consistency-preserving weighted baseline .
Revised: Appendix C.3 presents a result that enables a consistency-preserving weighted baseline .

Original: In this appendix, we will consider a dataset D
Revised: In this appendix, we will consider a dataset D

Original: = { ( τ ( i ) , g ( i ) ) } Ni =1 where each trajectory τ ( i ) is obtained using a policy parameterized by θ in an attempt to achieve a goal g ( i ) chosen by the environment.
Revised: = { ( τ ( i ) , g ( i ) ) } Ni =1 where each trajectory τ ( i ) is obtained using a policy parameterized by θ in an attempt to achieve a goal g ( i ) chosen by the environment.

Original: Because D is an iid dataset given Θ ,
Revised: Because D is an iid dataset given Θ ,

Original: [Equation]
Revised: [Equation]

Original: C.
Revised: C.

Original: T HEOREM 5.
Revised: T HEOREM 5.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: The per-decision hindsight policy gradient estimator, given by
Revised: The per-decision hindsight policy gradient estimator, given by

Original: [Equation]
Revised: [Equation]

Original: is a consistent and unbiased estimator of the gradient ∇ η ( θ ) of the expected return.
Revised: is a consistent and unbiased estimator of the gradient ∇ η ( θ ) of the expected return.

Original: Proof.
Revised: Proof.

Original: Let I ( N ) j denote the j -th element of the estimator, which can be written as
Revised: Let I ( N ) j denote the j -th element of the estimator, which can be written as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: I ( τ , g
Revised: I ( τ , g

Original: [Equation]
Revised: [Equation]

Original: , θ
Revised: , θ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Using Theorem 4.2, the expected value E (cid:104) I ( N ) j | θ (cid:105) is given by
Revised: Using Theorem 4.2, the expected value E (cid:104) I ( N ) j | θ (cid:105) is given by

Original: [Equation]
Revised: [Equation]

Original: Therefore, I ( N ) j is an unbiased estimator of ∂η ( θ ) /∂θ j .
Revised: Therefore, I ( N ) j is an unbiased estimator of ∂η ( θ ) /∂θ j .

Original: Conditionally on Θ , the random variable I ( N ) j is an average of iid random variables with expected value ∂η ( θ ) /∂θ
Revised: Conditionally on Θ , the random variable I ( N ) j is an average of iid random variables with expected value ∂η ( θ ) /∂θ

Original: j (see Eq. 61).
Revised: j (see Eq. 61).

Original: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13),
Revised: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13),

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: Therefore, I ( N ) j is a consistent estimator of ∂η ( θ ) /∂θ j .
Revised: Therefore, I ( N ) j is a consistent estimator of ∂η ( θ ) /∂θ j .

Original: C. T HEOREM 5. 
Revised: C. T HEOREM 5.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: The weighted per-decision hindsight policy gradient estimator, given by
Revised: The weighted per-decision hindsight policy gradient estimator, given by

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: is a consistent estimator of the gradient ∇ η ( θ ) of the expected return.
Revised: is a consistent estimator of the gradient ∇ η ( θ ) of the expected return.

Original: Proof.
Revised: Proof.

Original: Let W ( N ) j denote the j -th element of the estimator, which can be written as
Revised: Let W ( N ) j denote the j -th element of the estimator, which can be written as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Consider the expected value E
Revised: Consider the expected value E

Original: [Equation]
Revised: [Equation]

Original: i ) , G ( i ) , g, t, t (cid:48) , θ ) j | θ (cid:105) , which is given by
Revised: i ) , G ( i ) , g, t, t (cid:48) , θ ) j | θ (cid:105) , which is given by

Original: E X i =
Revised: E X i =

Original: [Equation]
Revised: [Equation]

Original: Using the fact that t (cid:48) > t , Lemma D.1, and canceling terms, E X i can be written as
Revised: Using the fact that t (cid:48) > t , Lemma D.1, and canceling terms, E X i can be written as

Original: [Equation]
Revised: [Equation]

Original: Because S t (cid:48) ⊥⊥ G | S
Revised: Because S t (cid:48) ⊥⊥ G | S

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: − 1 , Θ
Revised: − 1 , Θ

Original: [Equation]
Revised: [Equation]

Original: Conditionally on Θ , the variable X ( g, t, t (cid:48) )
Revised: Conditionally on Θ , the variable X ( g, t, t (cid:48) )

Original: ( N ) j is an average of iid random variables with expected value E X i .
Revised: ( N ) j is an average of iid random variables with expected value E X i .

Original: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13),
Revised: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13),

Original: X ( g, t, t (cid:48) )
Revised: X ( g, t, t (cid:48) )

Original: ( N ) j a.s.
Revised: ( N ) j a.s.

Original: [Equation]
Revised: [Equation]

Original: E X i .
Revised: E X i .

Original: Using Lemma D.1, the expected value E Y i
Revised: Using Lemma D.1, the expected value E Y i

Original: = E (cid:104) Y ( T ( i ) , G ( i ) , g, t, t (cid:48) , θ ) j | θ (cid:105) is given by
Revised: = E (cid:104) Y ( T ( i ) , G ( i ) , g, t, t (cid:48) , θ ) j | θ (cid:105) is given by

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Conditionally on Θ , the variable Y ( g, t, t (cid:48) )
Revised: Conditionally on Θ , the variable Y ( g, t, t (cid:48) )

Original: ( N ) j is an average of iid random variables with expected value 1 .
Revised: ( N ) j is an average of iid random variables with expected value 1 .

Original: By the strong law of large numbers, Y ( g, t, t (cid:48) ) ( N ) j a.s. −−→ 1 .
Revised: By the strong law of large numbers, Y ( g, t, t (cid:48) ) ( N ) j a.s. −−→ 1 .

Original: Because both X ( g, t, t (cid:48) )
Revised: Because both X ( g, t, t (cid:48) )

Original: ( N ) j and Y ( g, t, t (cid:48) )
Revised: ( N ) j and Y ( g, t, t (cid:48) )

Original: ( N ) j converge almost surely to real numbers (Thomas, 2015, Ch. 3, Property 2),
Revised: ( N ) j converge almost surely to real numbers (Thomas, 2015, Ch. 3, Property 2),

Original: [Equation]
Revised: [Equation]

Original: By Theorem 3.1 and the fact that W ( N ) j is a linear combination of terms X ( g, t
Revised: By Theorem 3.1 and the fact that W ( N ) j is a linear combination of terms X ( g, t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original:  T HEOREM C. 
Revised:  T HEOREM C.

Original: Theorem C
Revised: Theorem C

Original: [Equation]
Revised: [Equation]

Original: The weighted baseline estimator, given by
Revised: The weighted baseline estimator, given by

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: converges almost surely to zero.
Revised: converges almost surely to zero.

Original: Proof.
Revised: Proof.

Original: Let B ( N ) j denote the j -th element of the estimator, which can be written as
Revised: Let B ( N ) j denote the j -th element of the estimator, which can be written as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Using Eqs.
Revised: Using Eqs.

Original: 44 and 47, the expected value E X i
Revised: 44 and 47, the expected value E X i

Original: = E (cid:104) X ( T ( i ) , G ( i ) , g, t, θ )
Revised: = E (cid:104) X ( T ( i ) , G ( i ) , g, t, θ )

Original: j | θ (cid:105) is given by
Revised: j | θ (cid:105) is given by

Original: [Equation]
Revised: [Equation]

Original: Conditionally on Θ , the variable X ( g, t ) ( N ) j is an average of iid random variables with expected value zero.
Revised: Conditionally on Θ , the variable X ( g, t ) ( N ) j is an average of iid random variables with expected value zero.

Original: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13), X ( g, t ) ( N ) j a.s. −−→ 0 .
Revised: By the strong law of large numbers (Sen & Singer, 1994, Theorem 2.3.13), X ( g, t ) ( N ) j a.s. −−→ 0 .

Original: The fact that Y ( g, t ) ( N ) j a.s. −−→ 1 is already established in the proof of Theorem 5.2.
Revised: The fact that Y ( g, t ) ( N ) j a.s. −−→ 1 is already established in the proof of Theorem 5.2.

Original: Because both
Revised: Because both

Original: X ( g, t ) ( N ) j and Y ( g, t ) ( N ) j converge almost surely to real numbers (Thomas, 2015, Ch. 3, Property
Revised: X ( g, t ) ( N ) j and Y ( g, t ) ( N ) j converge almost surely to real numbers (Thomas, 2015, Ch. 3, Property

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Because B ( N ) j is a linear combination of terms X ( g, t ) ( N )
Revised: Because B ( N ) j is a linear combination of terms X ( g, t ) ( N )

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: Clearly, if E ( N ) is a consistent estimator of a some quantity given θ , then so is E ( N ) − B ( N ) j , which allows using this result in combination with Theorem 5.2.
Revised: Clearly, if E ( N ) is a consistent estimator of a some quantity given θ , then so is E ( N ) − B ( N ) j , which allows using this result in combination with Theorem 5.2.

Original: D F UNDAMENTAL RESULTS
Revised: D F UNDAMENTAL RESULTS

Original: This appendix presents results required by previous sections:
Revised: This appendix presents results required by previous sections:

Original: Lemma D.1 (App. D.1), Lemma D.
Revised: Lemma D.1 (App. D.1), Lemma D.

Original: (App. D.2), Theorem 4.4 (App. D.4), and Lemma D.4 (App. D.5).
Revised: (App. D.2), Theorem 4.4 (App. D.4), and Lemma D.4 (App. D.5).

Original: Appendix D.3 contains an auxiliary result.
Revised: Appendix D.3 contains an auxiliary result.

Original: D. L EMMA D. 
Revised: D. L EMMA D.

Original: Lemma D.1.
Revised: Lemma D.1.

Original: For every τ , g, θ , and 1 ≤ t ≤ T − 1 ,
Revised: For every τ , g, θ , and 1 ≤ t ≤ T − 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In order to employ backward induction, consider the case t
Revised: In order to employ backward induction, consider the case t

Original: =
Revised: =

Original: T − 1 .
Revised: T − 1 .

Original: By marginalization,
Revised: By marginalization,

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of the base case.
Revised: which completes the proof of the base case.

Original: Assuming the inductive hypothesis is true for a given 2 ≤ t ≤ T − 1 and considering the case t − 1 , p
Revised: Assuming the inductive hypothesis is true for a given 2 ≤ t ≤ T − 1 and considering the case t − 1 , p

Original: [Equation]
Revised: [Equation]

Original: D. L EMMA D. 
Revised: D. L EMMA D.

Original: Lemma D.2.
Revised: Lemma D.2.

Original: For every τ , g, θ , and 1 ≤ t ≤ T ,
Revised: For every τ , g, θ , and 1 ≤ t ≤ T ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The case t = 1 can be inspected easily.
Revised: The case t = 1 can be inspected easily.

Original: Consider 2 ≤ t ≤ T .
Revised: Consider 2 ≤ t ≤ T .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using Lemma D
Revised: Using Lemma D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , g, θ ) .
Revised: , g, θ ) .

Original: [Equation]
Revised: [Equation]

Original: D.
Revised: D.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma D.3.
Revised: Lemma D.3.

Original: For every t and θ , the action-value function Q θ t is given by
Revised: For every t and θ , the action-value function Q θ t is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the deﬁnition of action-value function and using the fact that S t +1 ⊥⊥ G, Θ | S t , A t ,
Revised: From the deﬁnition of action-value function and using the fact that S t +1 ⊥⊥ G, Θ | S t , A t ,

Original: [Equation]
Revised: [Equation]

Original: Let z denote the second term in the right-hand side of the previous equation, which can also be written as
Revised: Let z denote the second term in the right-hand side of the previous equation, which can also be written as

Original: [Equation]
Revised: [Equation]

Original: Consider the following three independence properties:
Revised: Consider the following three independence properties:

Original: [Equation]
Revised: [Equation]

Original: Together, these properties can be used to demonstrate that
Revised: Together, these properties can be used to demonstrate that

Original: [Equation]
Revised: [Equation]

Original: s t
Revised: s t

Original: +1 ,g, θ ) (cid:88)
Revised: +1 ,g, θ ) (cid:88)

Original: st +2: T p ( s t +2: T | s t +1 ,a t +1 ,g, θ ) T (cid:88) t (cid:48)
Revised: st +2: T p ( s t +2: T | s t +1 ,a t +1 ,g, θ ) T (cid:88) t (cid:48)

Original: =
Revised: =

Original: t +2 r ( s t (cid:48) ,g ) .
Revised: t +2 r ( s t (cid:48) ,g ) .

Original: From the deﬁnition of value function, z = E (cid:2) V θ t +

Revised: From the deﬁnition of value function, z = E (cid:2) V θ t +

Original: ( S t +1 , g ) | S t
Revised: ( S t +1 , g ) | S t

Original: = s, A t = a (cid:3) .
Revised: = s, A t = a (cid:3) .

Original: D. T HEOREM 4. 
Revised: D. T HEOREM 4.

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: For every t and θ , the advantage function A θ t is given by
Revised: For every t and θ , the advantage function A θ t is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result follows from the deﬁnition of advantage function and Lemma D.3.
Revised: The result follows from the deﬁnition of advantage function and Lemma D.3.

Original: D.
Revised: D.

Original: L EMMA D. 
Revised: L EMMA D.

Original: Consider a discrete random variable X and real-valued functions f and h .
Revised: Consider a discrete random variable X and real-valued functions f and h .

Original: Suppose also that E [ h ( X )] = 0 and Var [ h ( X )] > 0 .
Revised: Suppose also that E [ h ( X )] = 0 and Var [ h ( X )] > 0 .

Original: Clearly, for every b ∈ R , we have E [ f ( X ) − bh ( X )]
Revised: Clearly, for every b ∈ R , we have E [ f ( X ) − bh ( X )]

Original: =
Revised: =

Original: E [ f ( X )] .
Revised: E [ f ( X )] .

Original: Lemma D.4.
Revised: Lemma D.4.

Original: The constant b ∈ R that minimizes Var [ f ( X ) − bh ( X )] is given by
Revised: The constant b ∈ R that minimizes Var [ f ( X ) − bh ( X )] is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let v =
Revised: Let v =

Original: Var [ f ( X ) − bh ( X )] .
Revised: Var [ f ( X ) − bh ( X )] .

Original: Using our assumptions and the deﬁnition of variance, v
Revised: Using our assumptions and the deﬁnition of variance, v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f ( X ) 2 (cid:3) − 2 b E [ f ( X ) h
Revised: f ( X ) 2 (cid:3) − 2 b E [ f ( X ) h

Original: ( X )]
Revised: ( X )]

Original: + b 2 E (cid:2) h
Revised: + b 2 E (cid:2) h

Original: ( X ) 2 (cid:3) − E [ f ( X )] 2 .
Revised: ( X ) 2 (cid:3) − E [ f ( X )] 2 .

Original: The ﬁrst and second derivatives of v with respect to b are given by dv/db = − 2 E [ f ( X ) h
Revised: The ﬁrst and second derivatives of v with respect to b are given by dv/db = − 2 E [ f ( X ) h

Original: ( X )]
Revised: ( X )]

Original: + 2 b E (cid:2) h
Revised: + 2 b E (cid:2) h

Original: ( X ) 2 (cid:3) and d 2 v/db 2 = 2 E (cid:2)
Revised: ( X ) 2 (cid:3) and d 2 v/db 2 = 2 E (cid:2)

Original: h ( X ) 2 (cid:3) .
Revised: h ( X ) 2 (cid:3) .

Original: Our assumptions guarantee that E (cid:2) h ( X ) 2 (cid:3) > 0 .
Revised: Our assumptions guarantee that E (cid:2) h ( X ) 2 (cid:3) > 0 .

Original: Therefore, by Fermat’s theorem, if b is a local minimum, then dv/db = 0 , leading to the desired equality.
Revised: Therefore, by Fermat’s theorem, if b is a local minimum, then dv/db = 0 , leading to the desired equality.

Original: By the second derivative test, b must be a local minimum.
Revised: By the second derivative test, b must be a local minimum.

Original: E E XPERIMENTS
Revised: E E XPERIMENTS

Original: This appendix contains additional information about the experiments introduced in Section 6.
Revised: This appendix contains additional information about the experiments introduced in Section 6.

Original: Appendix E.1 details policy and baseline representations.
Revised: Appendix E.1 details policy and baseline representations.

Original: Appendix E.2 documents experimental settings.
Revised: Appendix E.2 documents experimental settings.

Original: Appendix E.3 presents unabridged results.
Revised: Appendix E.3 presents unabridged results.

Original: E. P OLICY AND BASELINE REPRESENTATIONS
Revised: E. P OLICY AND BASELINE REPRESENTATIONS

Original: In every experiment, a policy is represented by a feedforward neural network with a softmax output layer.
Revised: In every experiment, a policy is represented by a feedforward neural network with a softmax output layer.

Original: The input to such a policy is a pair composed of state and goal.
Revised: The input to such a policy is a pair composed of state and goal.

Original: A baseline function is represented by a feedforward neural network with a single (linear) output neuron.
Revised: A baseline function is represented by a feedforward neural network with a single (linear) output neuron.

Original: The input to such a baseline function is a triple composed of state, goal, and time step.
Revised: The input to such a baseline function is a triple composed of state, goal, and time step.

Original: The baseline function is trained to approximate the value function using the mean squared (one-step) temporal difference error (Sutton & Barto, 1998).
Revised: The baseline function is trained to approximate the value function using the mean squared (one-step) temporal difference error (Sutton & Barto, 1998).

Original: Parameters are updated using Adam (Kingma & Ba, 2014).
Revised: Parameters are updated using Adam (Kingma & Ba, 2014).

Original: The networks are given by the following.
Revised: The networks are given by the following.

Original: Bit ﬂipping environments and grid world environments.
Revised: Bit ﬂipping environments and grid world environments.

Original: Both policy and baseline networks have two hidden layers, each with 256 hyperbolic tangent units.
Revised: Both policy and baseline networks have two hidden layers, each with 256 hyperbolic tangent units.

Original: Every weight is initially drawn from a Gaussian distribution with mean 0 and standard deviation 0 .
Revised: Every weight is initially drawn from a Gaussian distribution with mean 0 and standard deviation 0 .

Original: 01 (and redrawn if far from the mean by two standard deviations), and every bias is initially zero.
Revised: 01 (and redrawn if far from the mean by two standard deviations), and every bias is initially zero.

Original: Ms. Pac-man environment.
Revised: Ms. Pac-man environment.

Original: The policy network is represented by a convolutional neural network.
Revised: The policy network is represented by a convolutional neural network.

Original: The network architecture is given by a convolutional layer with 32 ﬁlters ( 8 × 8 , stride 4 ); convolutional layer with 64 ﬁlters ( 4 × 4 , stride 2 ); convolutional layer with 64 ﬁlters ( 3 × 3 , stride 1 ); and three fully-connected layers, each with 256 units.
Revised: The network architecture is given by a convolutional layer with 32 ﬁlters ( 8 × 8 , stride 4 ); convolutional layer with 64 ﬁlters ( 4 × 4 , stride 2 ); convolutional layer with 64 ﬁlters ( 3 × 3 , stride 1 ); and three fully-connected layers, each with 256 units.

Original: Every unit uses a hyperbolic tangent activation function.
Revised: Every unit uses a hyperbolic tangent activation function.

Original: Every weight is initially set using variance scaling (Glorot & Bengio, 2010), and every bias is initially zero.
Revised: Every weight is initially set using variance scaling (Glorot & Bengio, 2010), and every bias is initially zero.

Original: These design decisions are similar to the ones made by Mnih et al.
Revised: These design decisions are similar to the ones made by Mnih et al.

Original: A sequence of images obtained from the Arcade Learning Environment (Bellemare et al., 2013) is preprocessed as follows.
Revised: A sequence of images obtained from the Arcade Learning Environment (Bellemare et al., 2013) is preprocessed as follows.

Original: Individually for each color channel, an elementwise maximum operation is employed between two consecutive images to reduce rendering artifacts.
Revised: Individually for each color channel, an elementwise maximum operation is employed between two consecutive images to reduce rendering artifacts.

Original: Such 210 × 160 ×preprocessed image is converted to grayscale, cropped, and rescaled into an 84 × 84 image x t .
Revised: Such 210 × 160 ×preprocessed image is converted to grayscale, cropped, and rescaled into an 84 × 84 image x t .

Original: A sequence of images x t − 12 , x t − 8 , x t − 4 , x t obtained in this way is stacked into an 84 × 84 × 4 image, which is an input to the policy network (recall that each action is repeated for 13 game ticks).
Revised: A sequence of images x t − 12 , x t − 8 , x t − 4 , x t obtained in this way is stacked into an 84 × 84 × 4 image, which is an input to the policy network (recall that each action is repeated for 13 game ticks).

Original: The goal information is concatenated with the ﬂattened output of the last convolutional layer.
Revised: The goal information is concatenated with the ﬂattened output of the last convolutional layer.

Original: FetchPush environment.
Revised: FetchPush environment.

Original: The policy network has three hidden layers, each with 256 hyperbolic tangent units.
Revised: The policy network has three hidden layers, each with 256 hyperbolic tangent units.

Original: Every weight is initially set using variance scaling (Glorot & Bengio, 2010), and every bias is initially zero.
Revised: Every weight is initially set using variance scaling (Glorot & Bengio, 2010), and every bias is initially zero.

Original: E. E XPERIMENTAL SETTINGS
Revised: E. E XPERIMENTAL SETTINGS

Original: Tables 1 and 2 document the experimental settings.
Revised: Tables 1 and 2 document the experimental settings.

Original: The number of runs, training batches, and batches between evaluations are reported separately for hyperparameter search and deﬁnitive runs.
Revised: The number of runs, training batches, and batches between evaluations are reported separately for hyperparameter search and deﬁnitive runs.

Original: The number of training batches is adapted according to how soon each estimator leads to apparent convergence.
Revised: The number of training batches is adapted according to how soon each estimator leads to apparent convergence.

Original: Note that it is very difﬁcult to establish this setting before hyperparameter search.
Revised: Note that it is very difﬁcult to establish this setting before hyperparameter search.

Original: The number of batches between evaluations is adapted so that there are 100 evaluation steps in total.
Revised: The number of batches between evaluations is adapted so that there are 100 evaluation steps in total.

Original: Other settings include the sets of policy and baseline learning rates under consideration for hyperparameter search, and the number of active goals subsampled per episode.
Revised: Other settings include the sets of policy and baseline learning rates under consideration for hyperparameter search, and the number of active goals subsampled per episode.

Original: In Tables 1 and 2,
Revised: In Tables 1 and 2,

Original: R 1 = { α × 10 − k | α ∈ { 1 , 5 } and k ∈ { 2 , 3 , 4 , 5 }} and R 2 = { β × 10 − 5 | β ∈ { 1 , 2 . 5 , 5 , 7 . 5 , 10 }} .
Revised: R 1 = { α × 10 − k | α ∈ { 1 , 5 } and k ∈ { 2 , 3 , 4 , 5 }} and R 2 = { β × 10 − 5 | β ∈ { 1 , 2 . 5 , 5 , 7 . 5 , 10 }} .

Original: As already mentioned in Section 6, the deﬁnitive runs use the best combination of hyperparameters (learning rates) found for each estimator.
Revised: As already mentioned in Section 6, the deﬁnitive runs use the best combination of hyperparameters (learning rates) found for each estimator.

Original: Every setting was carefully chosen during preliminary experiments to ensure that the best result for each estimator is representative.
Revised: Every setting was carefully chosen during preliminary experiments to ensure that the best result for each estimator is representative.

Original: In particular, the best performing learning rates rarely lie on the extrema of the corresponding search range.
Revised: In particular, the best performing learning rates rarely lie on the extrema of the corresponding search range.

Original: In the single case where the best performing learning rate found by hyperparameter search for a goal-conditional policy gradient estimator was such an extreme value (FetchPush, for a small batch size), evaluating one additional learning rate lead to decreased average performance.
Revised: In the single case where the best performing learning rate found by hyperparameter search for a goal-conditional policy gradient estimator was such an extreme value (FetchPush, for a small batch size), evaluating one additional learning rate lead to decreased average performance.

Original: T a b l e 1 : E xp e r
Revised: T a b l e 1 : E xp e r

Original: i m e n t a l s e tti ng s f o r t
Revised: i m e n t a l s e tti ng s f o r t

Original: h e b it ﬂ i
Revised: h e b it ﬂ i

Original: pp i ng a nd
Revised: pp i ng a nd

Original: g r i d w o r l d e nv
Revised: g r i d w o r l d e nv

Original: i r on m e n t s
Revised: i r on m e n t s

Original: B it ﬂ i pp
Revised: B it ﬂ i pp

Original: i
Revised: i

Original: ng ( 8 b it s )
Revised: ng ( 8 b it s )

Original: B it ﬂ i pp
Revised: B it ﬂ i pp

Original: i
Revised: i

Original: ng (b it s )
Revised: ng (b it s )

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i
Revised: c h s i

Original: z e
Revised: z e

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: R un s ( d e ﬁ n iti v e )
Revised: R un s ( d e ﬁ n iti v e )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( d e ﬁ n iti v e )
Revised: c h e s ( d e ﬁ n iti v e )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )

Original: R un s ( s ea r c h )
Revised: R un s ( s ea r c h )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( s ea r c h )  

Revised: c h e s ( s ea r c h )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )

Original: P o li cy l e a r n i ng r a t
Revised: P o li cy l e a r n i ng r a t

Original: e s
Revised: e s

Original: R
Revised: R

Original: R R
Revised: R R

Original: R
Revised: R

Original: B a s e li n e l e a r n i ng r a t
Revised: B a s e li n e l e a r n i ng r a t

Original: e s
Revised: e s

Original: R R
Revised: R R

Original: R R
Revised: R R

Original: E p i s od e s p e r e v a l u a ti on
Revised: E p i s od e s p e r e v a l u a ti on

Original: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞ ∞ ∞
Revised: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞ ∞ ∞

Original: E m p t y r oo m
Revised: E m p t y r oo m

Original: F ou r r oo m s
Revised: F ou r r oo m s

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i
Revised: c h s i

Original: z e
Revised: z e

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: R un s ( d e ﬁ n iti v e )
Revised: R un s ( d e ﬁ n iti v e )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( d e ﬁ n iti v e )
Revised: c h e s ( d e ﬁ n iti v e )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )

Original: R un s ( s ea r c h )
Revised: R un s ( s ea r c h )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( s ea r c h )
Revised: c h e s ( s ea r c h )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )

Original: P o li cy l e a r n i ng r a t
Revised: P o li cy l e a r n i ng r a t

Original: e s
Revised: e s

Original: R
Revised: R

Original: R R
Revised: R R

Original: R
Revised: R

Original: B a s e li n e l e a r n i ng r a t
Revised: B a s e li n e l e a r n i ng r a t

Original: e s
Revised: e s

Original: R R
Revised: R R

Original: R R
Revised: R R

Original: E p i s od e s p e r e v a l u a ti on
Revised: E p i s od e s p e r e v a l u a ti on

Original: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞ ∞ ∞
Revised: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞ ∞ ∞

Original: T a b l e 2 : E xp e r
Revised: T a b l e 2 : E xp e r

Original: i m e n t a l s e tti ng s f o r t
Revised: i m e n t a l s e tti ng s f o r t

Original: h e
Revised: h e

Original: M s .
Revised: M s .

Original: P acm a n a nd
Revised: P acm a n a nd

Original: F e t c h P u s h e nv
Revised: F e t c h P u s h e nv

Original: i r on m e n t s
Revised: i r on m e n t s

Original: M s .
Revised: M s .

Original: P acm a n
Revised: P acm a n

Original: F e t c h P u s h
Revised: F e t c h P u s h

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: B a t
Revised: B a t

Original: c h s i
Revised: c h s i

Original: z e
Revised: z e

Original: B a t
Revised: B a t

Original: c h s i ze
Revised: c h s i ze

Original: R un s ( d e ﬁ n iti v e )
Revised: R un s ( d e ﬁ n iti v e )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( d e ﬁ n iti v e )
Revised: c h e s ( d e ﬁ n iti v e )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( d e ﬁ n iti v e )

Original: R un s ( s ea r c h )
Revised: R un s ( s ea r c h )

Original: T r a i n i ng b a t
Revised: T r a i n i ng b a t

Original: c h e s ( s ea r c h )
Revised: c h e s ( s ea r c h )

Original: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )
Revised: B a t c h e s b e t w ee n e v a l u a ti on s ( s ea r c h )

Original: P o li cy l e a r n i ng r a t
Revised: P o li cy l e a r n i ng r a t

Original: e s
Revised: e s

Original: R R
Revised: R R

Original: R R
Revised: R R

Original: E p i s od e s p e r e v a l u a ti on
Revised: E p i s od e s p e r e v a l u a ti on

Original: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞
Revised: M a x i m u m ac ti v e go a l s p e r e p i s od e ∞ ∞

Original: E. R ESULTS
Revised: E. R ESULTS

Original: This appendix contains unabridged experimental results.
Revised: This appendix contains unabridged experimental results.

Original: Appendices E.3.1 and E.3.2 present hyperparameter sensitivity plots for every combination of environment and batch size.
Revised: Appendices E.3.1 and E.3.2 present hyperparameter sensitivity plots for every combination of environment and batch size.

Original: A hyperparameter sensitivity plot displays the average performance achieved by each hyperparameter setting (sorted from best to worst along the horizontal axis).
Revised: A hyperparameter sensitivity plot displays the average performance achieved by each hyperparameter setting (sorted from best to worst along the horizontal axis).

Original: Appendices E.3.3 and E.3.4 present learning curves for every combination of environment and batch size.
Revised: Appendices E.3.3 and E.3.4 present learning curves for every combination of environment and batch size.

Original: Appendix E.3.5 presents average performance results.
Revised: Appendix E.3.5 presents average performance results.

Original: Appendix E.3.6 presents an empirical study of likelihood ratios.
Revised: Appendix E.3.6 presents an empirical study of likelihood ratios.

Revised: Appendix E.3.7 presents an empirical comparison with hindsight experience replay (Andrychowicz et al., 2017).

Original: E.3. H YPERPARAMETERSENSITIVITY PLOTS
Revised: E.3. H YPERPARAMETER

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  age pe 
Revised: a v e r age pe r f o r m an c e

Original: [Figure]
Revised: [Figure]

Original: o r m an c e
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E.3. A VERAGE PERFORMANCE RESULTS
Revised: E.3. A VERAGE PERFORMANCE RESULTS

Original: Table 3 presents average performance results for every combination of environment and batch size.
Revised: Table 3 presents average performance results for every combination of environment and batch size.

Original: [Table]
Revised: [Table]

Original: E.3. L IKELIHOOD RATIO PLOTS
Revised: E.3. L IKELIHOOD RATIO PLOTS

Original: This appendix presents a study of the active (normalized) likelihood ratios computed by agents during training.
Revised: This appendix presents a study of the active (normalized) likelihood ratios computed by agents during training.

Original: A likelihood ratio is considered active if and only if it multiplies a non-zero reward (see
Revised: A likelihood ratio is considered active if and only if it multiplies a non-zero reward (see

Original: Expression 12).
Revised: Expression 12).

Original: Note that only these likelihood ratios affect gradient estimates based on HPG.
Revised: Note that only these likelihood ratios affect gradient estimates based on HPG.

Original: This study is conveyed through plots that encode the distribution of active likelihood ratios computed during training, individually for each time step within an episode.
Revised: This study is conveyed through plots that encode the distribution of active likelihood ratios computed during training, individually for each time step within an episode.

Original: Each plot corresponds to an agent that employs HPG and obtains the highest deﬁnitive average performance for a given environment (Figs. 34-39).
Revised: Each plot corresponds to an agent that employs HPG and obtains the highest deﬁnitive average performance for a given environment (Figs. 34-39).

Original: Note that the length of the largest bar for a given time step is ﬁxed to aid visualization.
Revised: Note that the length of the largest bar for a given time step is ﬁxed to aid visualization.

Original: The most important insight provided by these plots is that likelihood ratios behave very differently across environments, even for equivalent time steps (for instance, compare bit ﬂipping environments to grid world environments).
Revised: The most important insight provided by these plots is that likelihood ratios behave very differently across environments, even for equivalent time steps (for instance, compare bit ﬂipping environments to grid world environments).

Original: In contrast, after the ﬁrst time step, the behavior of likelihood ratios changes slowly across time steps within the same environment.
Revised: In contrast, after the ﬁrst time step, the behavior of likelihood ratios changes slowly across time steps within the same environment.

Original: In any case, alternative goals have a signiﬁcant effect on gradient estimates, which agrees with the results presented in Section
Revised: In any case, alternative goals have a signiﬁcant effect on gradient estimates, which agrees with the results presented in Section

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: li k e li hood r a t
Revised: li k e li hood r a t

Original: i
Revised: i

Original: o h
Revised: o h

Original: i s t og r a m
Revised: i s t og r a m

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ( no r m a li z ed ) li k e li hood r a t
Revised: ( no r m a li z ed ) li k e li hood r a t

Original: i
Revised: i

Original: o h
Revised: o h

Original: i s t og r a m
Revised: i s t og r a m

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ( no r m a li z ed ) li k e li hood r a t
Revised: ( no r m a li z ed ) li k e li hood r a t

Original: i
Revised: i

Original: o h
Revised: o h

Original: i s t og r a m
Revised: i s t og r a m

Original: [Figure]
Revised: [Figure]

Revised: E.3. H INDSIGHT EXPERIENCE REPLAY

Revised: This appendix documents an empirical comparison between goal-conditional policy gradients (GCPG), hindsight policy gradients (HPG), deep Q-networks (Mnih et al., 2015, DQN), and a combination of DQN and hindsight experience replay (Andrychowicz et al., 2017, DQN+HER).

Revised: Experience replay.

Revised: Our implementations of both DQN and DQN+HER are based on OpenAI

Revised: Baselines (Dhariwal et al., 2017), and use mostly the same hyperparameters that Andrychowicz et al.

Revised: (2017) used in their experiments on environments with discrete action spaces, all of which resemble our bit ﬂipping environments.

Revised: The only notable differences in our implementations are the lack of both Polyak-averaging and temporal difference target clipping.

Revised: Concretely, a cycle begins when an agent collects a number of episodes ( 16 ) by following an (cid:15) -greedy policy derived from its deep Q-network ( (cid:15)

Revised: = 0 . 2 ).

Revised: The corresponding transitions are included in a replay buffer, which contains at most 10 6 transitions.

Revised: In the case of DQN+HER, hindsight transitions derived from a ﬁnal strategy are also included in this replay buffer.

Revised: Completing the cycle, for a total of 40 different batches, a batch composed of 128 transitions chosen at random from the replay buffer is used to deﬁne a loss function and allow one step of gradient-based minimization.

Revised: The targets required to deﬁne these loss functions are computed using a copy of the deep Q-network from the start of the corresponding cycle.

Revised: Parameters are updated using Adam (Kingma & Ba, 2014).

Revised: A discount factor of γ = 0 .

Revised: 98 is used, and seems necessary to improve the stability of both DQN and DQN+HER.

Revised: Network architectures.

Revised: In every experiment, the deep Q-network is implemented by a feedforward neural network with a linear output neuron corresponding to each action.

Revised: The input to such a network is a triple composed of state, goal, and time step.

Revised: The network architectures are the same as those described in Appendix E.1, except that every weight is initially set using variance scaling (Glorot &

Revised: Bengio, 2010), and all hidden layers use rectiﬁed linear units (Nair & Hinton, 2010).

Revised: For the Ms.

Revised: Pac-man environment, the time step information is concatenated with the ﬂattened output of the last convolutional layer (together with the goal information).

Revised: In comparison to the architecture employed by Andrychowicz et al.

Revised: (2017) for environments with discrete action spaces, our architectures have one or two additional hidden layers (besides the convolutional architecture used for Ms. Pac-man).

Revised: Experimental protocol.

Revised: The experimental protocol employed in our comparison is very similar to the one described in Section 6.

Revised: Each agent is evaluated periodically, after a number of cycles that depends on the environment.

Revised: During this evaluation, the agent collects a number of episodes by following a greedy policy derived from its deep Q-network.

Revised: For each environment, grid search is used to select the learning rates for both DQN and DQN+HER according to average performance scores (after the corresponding standard deviation across runs is subtracted, as in Section 6).

Revised: The candidate sets of learning rates are the following.

Revised: Bit ﬂipping and grid world environments: { α × 10 − k | α ∈ { 1 , 5 } and k ∈ { 2 , 3 , 4 , 5 }} , FetchPush: { 10 − 2 , 5 × 10 − 3 , 10 − 3 , 5 × 10 − 4 , 10 − 4 } , Ms. Pac-man: { 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 , 10 − 5 } .

Revised: These sets were carefully chosen such that the best performing learning rates do not lie on their extrema.

Revised: Deﬁnitive results for a given environment are obtained by using the best hyperparameters found for each method in additional runs.

Revised: These deﬁnitive results are directly comparable to our previous results for GCPG and HPG (batch size 16 ), since every method will have interacted with the environment for the same number of episodes before each evaluation step.

Revised: For each environment, the number of runs, the number of training batches (cycles), the number of batches (cycles) between evaluations, and the number of episodes per evaluation step are the same as those listed in Tables 1 and 2.

Revised: Results.

Revised: The deﬁnitive results for the different environments are represented by learning curves (Figs.

Revised: 40-45, Pg. 38).

Revised: In the bit ﬂipping environment for k = 8 (Figure 40), HPG and DQN+HER lead to equivalent sample efﬁciency, while GCPG lags far behind and DQN is completely unable to learn.

Revised: In the bit ﬂipping environment for k = 16 (Figure 41), HPG surpasses DQN+HER in sample efﬁciency by a small margin, while both GCPG and DQN are completely unable to learn.

Revised: In the empty room environment (Figure 42), HPG is arguably the most sample efﬁcient method, although DQN+HER is more stable upon obtaining a good policy.

Revised: GCPG eventually obtains a good policy, whereas DQN exhibits instability.

Revised: In the four rooms environment (Figure 43), DQN+HER outperforms all other methods by a large margin.

Revised: Although DQN takes much longer to obtain good policies, it would likely surpass both HPG and GCPG given additional training cycles.

Revised: In the Ms.

Revised: Pac-man environment (Figure 44), DQN+HER once again outperforms all other methods, which achieve equivalent sample efﬁciency (although DQN appears unstable by the end of training).

Revised: In the FetchPush environment (Figure 45), HPG dramatically outperforms all other methods.

Revised: Both

Revised: DQN+HER and DQN are completely unable to learn, while GCPG appears to start learning by the end of the training process.

Revised: Note that active goals are harshly subsampled to increase the computational efﬁciency of HPG for both Ms. Pac-man and FetchPush (see Sec.

Revised: 6.3 and Sec. 6.4).

Revised: Discussion.

Revised: Our results suggest that the decision between applying HPG or DQN+HER in a particular sparse-reward environment requires experimentation.

Revised: In contrast, the decision to apply hindsight was always successful.

Revised: Note that we have not employed heuristics that are known to sometimes increase the performance of policy gradient methods (such as entropy bonuses, reward scaling, learning rate annealing, and simple statistical baselines) to avoid introducing confounding factors.

Revised: We believe that such heuristics would allow both GCPG and HPG to achieve good results in both the four rooms environment and Ms.

Revised: Pac-man.

Revised: Furthermore, whereas hindsight experience replay is directly applicable to state-of-the-art techniques, our work can probably beneﬁt from being extended to state-of-the-art policy gradient approaches, which we intend to explore in future work.

Revised: Similarly, we believe that additional heuristics and careful hyperparameter settings would allow DQN+HER to achieve good results in the FetchPush environment.

Revised: This is evidenced by the fact that Andrychowicz et al.

Revised: (2017) achieve good results using the deep deterministic policy gradient (Lillicrap et al., 2016, DDPG) in a similar environment (with a continuous action space and a different reward function).

Revised: The empirical comparisons between either

Revised: GCPG and HPG or DQN and DQN+HER are comparatively more conclusive, since the similarities between the methods minimizes confounding factors.

Revised: Regardless of these empirical results, policy gradient approaches constitute one of the most important classes of model-free reinforcement learning methods, which by itself warrants studying how they can beneﬁt from hindsight.

Revised: Our approach is also complementary to previous work, since it is entirely possible to combine a critic trained by hindsight experience replay with an actor that employs hindsight policy gradients.

Revised: Although hindsight experience replay does not require a correction analogous to importance sampling, indiscriminately adding hindsight transitions to the replay buffer is problematic, which has mostly been tackled by heuristics (Andrychowicz et al., 2017, Sec. 4.5).

Revised: In contrast, our approach seems to beneﬁt from incorporating all available information about goals at every update, which also avoids the need for a replay buffer.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]