Original: U NDERSTANDING H INDSIGHT G OAL R ELABELING R EQUIRES R ETHINKING D IVERGENCE M INIMIZATION
Revised: U NDERSTANDING H INDSIGHT G OAL R ELABELING R EQUIRES R ETHINKING D IVERGENCE M INIMIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hindsight goal relabeling has become a foundational technique for multi-goal reinforcement learning (RL).
Revised: Hindsight goal relabeling has become a foundational technique for multi-goal reinforcement learning (RL).

Original: The idea is quite simple: any arbitrary trajectory can be seen as an expert demonstration for reaching the trajectory’s end state.
Revised: The idea is quite simple: any arbitrary trajectory can be seen as an expert demonstration for reaching the trajectory’s end state.

Original: Intuitively, this procedure trains a goal-conditioned policy to imitate a sub-optimal expert.
Revised: Intuitively, this procedure trains a goal-conditioned policy to imitate a sub-optimal expert.

Original: However, this connection between imitation and hindsight relabeling is not well understood.
Revised: However, this connection between imitation and hindsight relabeling is not well understood.

Original: Modern imitation learning algorithms are described in the language of divergence minimization, and yet it remains an open problem how to recast hindsight goal relabeling into that framework.
Revised: Modern imitation learning algorithms are described in the language of divergence minimization, and yet it remains an open problem how to recast hindsight goal relabeling into that framework.

Original: In this work, we develop a unified objective for goal-reaching that explains such a connection, from which we can derive goal-conditioned supervised learning (GCSL) and the reward function in hindsight experience replay (HER) from first principles.
Revised: In this work, we develop a unified objective for goal-reaching that explains such a connection, from which we can derive goal-conditioned supervised learning (GCSL) and the reward function in hindsight experience replay (HER) from first principles.

Original: Experimentally, we find that despite recent advances in goal-conditioned behaviour cloning (BC), multi-goal
Revised: Experimentally, we find that despite recent advances in goal-conditioned behaviour cloning (BC), multi-goal

Original: Q -learning can still outperform BC-like methods; moreover, a vanilla combination of both actually hurts model performance.
Revised: Q -learning can still outperform BC-like methods; moreover, a vanilla combination of both actually hurts model performance.

Original: Under our framework, we study when
Revised: Under our framework, we study when

Original: BC is expected to help, and empirically validate our findings.
Revised: BC is expected to help, and empirically validate our findings.

Original: Our work further bridges goal-reaching and generative modeling, illustrating the nuances and new pathways of extending the success of generative models to RL.
Revised: Our work further bridges goal-reaching and generative modeling, illustrating the nuances and new pathways of extending the success of generative models to RL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Goal reaching is an essential aspect of intelligence in sequential decision making.
Revised: Goal reaching is an essential aspect of intelligence in sequential decision making.

Original: Unlike the conventional formulation of reinforcement learning (RL), which aims to encode all desired behaviors into a single scalar reward function that is amenable to learning (Silver et al., 2021), goal reaching formulates the problem of RL as applying a sequence of actions to rearrange the environment into a desired state (Batra et al., 2020).
Revised: Unlike the conventional formulation of reinforcement learning (RL), which aims to encode all desired behaviors into a single scalar reward function that is amenable to learning (Silver et al., 2021), goal reaching formulates the problem of RL as applying a sequence of actions to rearrange the environment into a desired state (Batra et al., 2020).

Original: Goal-reaching is a highly flexible formulation.
Revised: Goal-reaching is a highly flexible formulation.

Original: For instance, we can design the goal-space to capture salient information about specific factors of variations that we care about (Plappert et al., 2018); we can use natural language instructions to define more abstract goals (Lynch & Sermanet, 2020; Ahn et al., 2022); we can encourage exploration by prioritizing previously unseen goals (Pong et al., 2019; Warde-Farley et al., 2018; Pitis et al., 2020); and we can even use self-supervised procedures to naturally learn goal-reaching policies without reward engineering (Pong et al., 2018; Nair et al., 2018b; Zhang et al., 2021; OpenAI et al., 2021).
Revised: For instance, we can design the goal-space to capture salient information about specific factors of variations that we care about (Plappert et al., 2018); we can use natural language instructions to define more abstract goals (Lynch & Sermanet, 2020; Ahn et al., 2022); we can encourage exploration by prioritizing previously unseen goals (Pong et al., 2019; Warde-Farley et al., 2018; Pitis et al., 2020); and we can even use self-supervised procedures to naturally learn goal-reaching policies without reward engineering (Pong et al., 2018; Nair et al., 2018b; Zhang et al., 2021; OpenAI et al., 2021).

Original: Reward is not enough.
Revised: Reward is not enough.

Original: Usually, rewards are manually constructed, either through laborious reward engineering, or from task-specific optimal demonstrations, neither of which is a scalable solution.
Revised: Usually, rewards are manually constructed, either through laborious reward engineering, or from task-specific optimal demonstrations, neither of which is a scalable solution.

Original: How can RL agents learn useful behaviors from unlabeled reward-free trajectories, similar to how
Revised: How can RL agents learn useful behaviors from unlabeled reward-free trajectories, similar to how

Original: NLP models such as BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020) are able to learn language from unlabeled text corpus?
Revised: NLP models such as BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020) are able to learn language from unlabeled text corpus?

Original: Goal reaching is a promising paradigm for unsupervised behavior acquisition, but it is unclear how to write down a well-defined objective for goal-conditioned policies, unlike language models that just predict the next token.
Revised: Goal reaching is a promising paradigm for unsupervised behavior acquisition, but it is unclear how to write down a well-defined objective for goal-conditioned policies, unlike language models that just predict the next token.

Original: This paper aims to define such an objective that unifies many prior approaches and strengthens the foundation of goal-conditioned RL.
Revised: This paper aims to define such an objective that unifies many prior approaches and strengthens the foundation of goal-conditioned RL.

Original: We start from the following observation: hindsight goal relabeling (Andrychowicz et al., 2017) can turn an arbitrary trajectory into a sub-optimal expert demonstration.
Revised: We start from the following observation: hindsight goal relabeling (Andrychowicz et al., 2017) can turn an arbitrary trajectory into a sub-optimal expert demonstration.

Original: Thus, goal-conditioned RL might be doing a special kind of imitation learning.
Revised: Thus, goal-conditioned RL might be doing a special kind of imitation learning.

Original: Currently, divergence minimization is the de facto way to describe imitation learning methods (Ghasemipour et al., 2020), so we should be able to recast hindsight goal relabeling into the divergence minimization framework.
Revised: Currently, divergence minimization is the de facto way to describe imitation learning methods (Ghasemipour et al., 2020), so we should be able to recast hindsight goal relabeling into the divergence minimization framework.

Original: On top of that, to tackle the sub-optimality of hindsight-relabeled trajectories, we should explicitly maximize the probability of reaching the desired goal.
Revised: On top of that, to tackle the sub-optimality of hindsight-relabeled trajectories, we should explicitly maximize the probability of reaching the desired goal.

Original: Following these intuitions, we derive the reward function used in HER from first principles, as well as other behaviour cloning (BC) like methods such as GCSL (Ghosh et al., 2019) and Hindsight BC (HBC) (Ding et al., 2019).
Revised: Following these intuitions, we derive the reward function used in HER from first principles, as well as other behaviour cloning (BC) like methods such as GCSL (Ghosh et al., 2019) and Hindsight BC (HBC) (Ding et al., 2019).

Original: Experimentally, we show that multi-goal Q -learning based on HER-like rewards, when carefully tuned, can still outperform goal-conditioned BC (such as GCSL / HBC).
Revised: Experimentally, we show that multi-goal Q -learning based on HER-like rewards, when carefully tuned, can still outperform goal-conditioned BC (such as GCSL / HBC).

Original: Moreover, a vanilla combination of multi-goal Q -learning and BC (HER + HBC), supposedly combining the best of both worlds, in fact hurts performance.
Revised: Moreover, a vanilla combination of multi-goal Q -learning and BC (HER + HBC), supposedly combining the best of both worlds, in fact hurts performance.

Original: We utilize our unified framework to analyze when a BC loss could help, and propose a modified algorithm named Hindsight Divergence Minimization (HDM) that uses
Revised: We utilize our unified framework to analyze when a BC loss could help, and propose a modified algorithm named Hindsight Divergence Minimization (HDM) that uses

Original: Q -learning to account for the worst while imitating the best.
Revised: Q -learning to account for the worst while imitating the best.

Original: HDM avoids the pitfalls of HER +
Revised: HDM avoids the pitfalls of HER +

Original: HBC and improves policy success rates on a variety of self-supervised goal-reaching environments.
Revised: HBC and improves policy success rates on a variety of self-supervised goal-reaching environments.

Original: Additionally, our framework reveals a largely unexplored design space for goal-reaching algorithms and potential paths of importing generative modeling techniques into multi-goal RL.
Revised: Additionally, our framework reveals a largely unexplored design space for goal-reaching algorithms and potential paths of importing generative modeling techniques into multi-goal RL.

Original:  R EINFORCEMENT L EARNING AND G ENERATIVE M ODELING 2. T HE R EINFORCEMENT L EARNING (RL) P ROBLEM
Revised:  R EINFORCEMENT L EARNING AND G ENERATIVE M ODELING 2. T HE R EINFORCEMENT L EARNING (RL) P ROBLEM

Original: We first review the basics of RL and generative modeling.
Revised: We first review the basics of RL and generative modeling.

Original: A Markov Decision Process (MDP) is typically parameterized by ( S , A , ρ 0 , p , r ): a state space S , an action space A , an initial state distribution ρ 0 ( s ) , a dynamics function p ( s ′ | s, a ) which defines the transition probability, and a reward function r ( s, a ) .
Revised: A Markov Decision Process (MDP) is typically parameterized by ( S , A , ρ 0 , p , r ): a state space S , an action space A , an initial state distribution ρ 0 ( s ) , a dynamics function p ( s ′ | s, a ) which defines the transition probability, and a reward function r ( s, a ) .

Original: A policy function µ defines a probability distribution µ : S ×
Revised: A policy function µ defines a probability distribution µ : S ×

Original: A → R + .
Revised: A → R + .

Original: For an infinite-horizon MDP, given the policy µ , and the state distribution at step t (starting from ρ 0 at t = 0 ), the state distribution at step t + 1 is given by:
Revised: For an infinite-horizon MDP, given the policy µ , and the state distribution at step t (starting from ρ 0 at t = 0 ), the state distribution at step t + 1 is given by:

Original: [Equation]
Revised: [Equation]

Original: The state visitation distribution sums over all timesteps via a geometric distribution Geom ( γ ) :
Revised: The state visitation distribution sums over all timesteps via a geometric distribution Geom ( γ ) :

Original: [Equation]
Revised: [Equation]

Original: However, the trajectory sampling process does not happen in this discounted manner, so the discount factor γ ∈ (0 , 1) is often absorbed into the cumulative return instead (Silver et al., 2014):
Revised: However, the trajectory sampling process does not happen in this discounted manner, so the discount factor γ ∈ (0 , 1) is often absorbed into the cumulative return instead (Silver et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: From 1 and 2, we also see that the future state distribution p + of policy µ , defined as a geometrically discounted sum of state distribution at all future timesteps given current state and action, is given by the following recursive relationship (Eysenbach et al., 2020b; Janner et al., 2020):
Revised: From 1 and 2, we also see that the future state distribution p + of policy µ , defined as a geometrically discounted sum of state distribution at all future timesteps given current state and action, is given by the following recursive relationship (Eysenbach et al., 2020b; Janner et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: In multi-goal RL, an MDP is augmented with a goal space G , and we learn a goal-conditioned policy π : S × G ×
Revised: In multi-goal RL, an MDP is augmented with a goal space G , and we learn a goal-conditioned policy π : S × G ×

Original: A → R + .
Revised: A → R + .

Original: Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) gives the agent a reward of 0 when the goal is reached and − 1 otherwise, and uses hindsight goal relabeling to increase learning efficiency of goal-conditioned Q -learning by replacing the initial behavioral goals with achieved goals (future states within the same trajectory).
Revised: Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) gives the agent a reward of 0 when the goal is reached and − 1 otherwise, and uses hindsight goal relabeling to increase learning efficiency of goal-conditioned Q -learning by replacing the initial behavioral goals with achieved goals (future states within the same trajectory).

Original: 2. I MITATION L EARNING (IL) AS D IVERGENCE M INIMIZATION
Revised: 2. I MITATION L EARNING (IL) AS D IVERGENCE M INIMIZATION

Original: We first review f -divergence between two probability distributions P and Q and its variational bound:
Revised: We first review f -divergence between two probability distributions P and Q and its variational bound:

Original: [Equation]
Revised: [Equation]

Original:  where f is a convex function such that f (1) = 0 , f ∗ is the convex conjugate of f , and T is an arbitrary class of functions T : X → R .
Revised: (5) where f is a convex function such that f (1) = 0 , f ∗ is the convex conjugate of f , and T is an arbitrary class of functions T : X → R .

Original: This variational bound was originally derived in (Nguyen et al., 2010) and was popularized by GAN (Goodfellow et al., 2014;
Revised: This variational bound was originally derived in (Nguyen et al., 2010) and was popularized by GAN (Goodfellow et al., 2014;

Original: Nowozin et al., 2016) and subsequently by imitation learning (Ho & Ermon, 2016; Fu et al., 2017; Finn et al., 2016; Ghasemipour et al., 2020).
Revised: Nowozin et al., 2016) and subsequently by imitation learning (Ho & Ermon, 2016; Fu et al., 2017; Finn et al., 2016; Ghasemipour et al., 2020).

Original: The equality holds true under mild conditions (Nguyen et al., 2010), and the optimal T is given by 
Revised: The equality holds true under mild conditions (Nguyen et al., 2010), and the optimal T is given by T

Original: [Equation]
Revised: [Equation]

Original:  ( p ( x ) /q
Revised: ′ ( p ( x ) /q

Original: ( x )) .
Revised: ( x )) .

Original: The canonical formulation of imitation learning follows (Ho & Ermon, 2016;
Revised: The canonical formulation of imitation learning follows (Ho & Ermon, 2016;

Original: Ghasemipour et al., 2020), where ρ exp ( s, a ) is from the expert:
Revised: Ghasemipour et al., 2020), where ρ exp ( s, a ) is from the expert:

Original: [Equation]
Revised: [Equation]

Original: Because of the policy gradient theorem (Sutton et al., 1999), the policy µ needs to optimize the cumulative return under its own trajectory distribution ρ µ ( s, a ) with the reward being r ( s, a ) =
Revised: Because of the policy gradient theorem (Sutton et al., 1999), the policy µ needs to optimize the cumulative return under its own trajectory distribution ρ µ ( s, a ) with the reward being r ( s, a ) =

Original: f ∗ ( T ( s, a )) .
Revised: f ∗ ( T ( s, a )) .

Original: Under this formulation, Jensen-Shannon divergence leads to GAIL (Ho & Ermon, 2016), reverse KL leads to AIRL (Fu et al., 2017).
Revised: Under this formulation, Jensen-Shannon divergence leads to GAIL (Ho & Ermon, 2016), reverse KL leads to AIRL (Fu et al., 2017).

Original: Note that f can in principle be any convex function (we can satisfy f (1) = 0 by simply adding a constant).
Revised: Note that f can in principle be any convex function (we can satisfy f (1) = 0 by simply adding a constant).

Original: 2. E NERGY -B ASED M ODELS (EBM)
Revised: 2. E NERGY -B ASED M ODELS (EBM)

Original: An energy-based model (EBM) is defined by p θ ( x ) = [Equation]
Revised: An energy-based model (EBM) is defined by p θ ( x ) = exp( − E θ ( x ))

Original:  , where E θ : R D → R is the energy function and Z ( θ ) = (cid:82) X exp( − E θ ( x ))
Revised: Z ( θ ) , where E θ : R D → R is the energy function and Z ( θ ) = (cid:82) X exp( − E θ ( x ))

Original: d x is the partition function.
Revised: d x is the partition function.

Original: The gradient of the log-likelihood log p θ ( x ) w.r.t.
Revised: The gradient of the log-likelihood log p θ ( x ) w.r.t.

Original: θ (known as contrastive divergence (Hinton, 2002)) can be expressed as :
Revised: θ (known as contrastive divergence (Hinton, 2002)) can be expressed as :

Original: [Equation]
Revised: [Equation]

Original: Sampling from p θ ( x ) can be difficult, as Langevin dynamics is often required (Welling & Teh, 2011;
Revised: Sampling from p θ ( x ) can be difficult, as Langevin dynamics is often required (Welling & Teh, 2011;

Original: Du & Mordatch, 2019; Grathwohl et al., 2019).
Revised: Du & Mordatch, 2019; Grathwohl et al., 2019).

Original: Alternatively, EBMs can be trained via Noise
Revised: Alternatively, EBMs can be trained via Noise

Original: Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2012; Mnih & Kavukcuoglu, 2013; Gao et al., 2020; Rhodes et al., 2020), which assumes access to a "noise" distribution p n ( x ) and learns energy functions through density ratio estimation.
Revised: Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2012; Mnih & Kavukcuoglu, 2013; Gao et al., 2020; Rhodes et al., 2020), which assumes access to a "noise" distribution p n ( x ) and learns energy functions through density ratio estimation.

Original: Let s θ ( x ) =
Revised: Let s θ ( x ) =

Original: − E θ ( x ) − log Z θ (Mnih & Teh, 2012).
Revised: − E θ ( x ) − log Z θ (Mnih & Teh, 2012).

Original: NCE learns a classifier that distinguishes the data distribution p ( x ) from noise p n ( x ) , where noise samples are k times more frequent (Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013):
Revised: NCE learns a classifier that distinguishes the data distribution p ( x ) from noise p n ( x ) , where noise samples are k times more frequent (Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013):

Original: [Equation]
Revised: [Equation]

Original: To be more concise, we denote ∆ θ ( x, k ) =
Revised: To be more concise, we denote ∆ θ ( x, k ) =

Original: s θ ( x ) − log p n ( x ) − log k .
Revised: s θ ( x ) − log p n ( x ) − log k .

Original: Gradients of the weighted binary cross entropy loss asymptotically approximate contrastive divergence (7) (Mnih & Teh, 2012):
Revised: Gradients of the weighted binary cross entropy loss asymptotically approximate contrastive divergence (7) (Mnih & Teh, 2012):

Original: [Equation]
Revised: [Equation]

Original: Closely related to NCE is the more recent InfoNCE loss (Van den Oord et al., 2018) which has gained popularity in the contrastive learning setting (Chen et al., 2020; He et al., 2020); we can interpret k in NCE as the batch size of negative samples in InfoNCE.
Revised: Closely related to NCE is the more recent InfoNCE loss (Van den Oord et al., 2018) which has gained popularity in the contrastive learning setting (Chen et al., 2020; He et al., 2020); we can interpret k in NCE as the batch size of negative samples in InfoNCE.

Original: In summary, NCE gives EBM a more tractable way to maximize data likelihood.
Revised: In summary, NCE gives EBM a more tractable way to maximize data likelihood.

Original:  G RAPHICAL M ODELS FOR H INDSIGHT G OAL R ELABELING
Revised:  G RAPHICAL M ODELS FOR H INDSIGHT G OAL R ELABELING

Original: Consider the setting: given an environment ξ
Revised: Consider the setting: given an environment ξ

Original: = ( ρ 0 ( s ) , p ( s ′ | s, a ) , ρ + ( g )) where we generate a dataset of trajectories D
Revised: = ( ρ 0 ( s ) , p ( s ′ | s, a ) , ρ + ( g )) where we generate a dataset of trajectories D

Original: = { ( s 0 , a 0 , s 1 , a 1 , · · · ) } by sampling from the initial state distribution ρ 0 ( s ) , an unobserved actor policy µ ( a | s ) , and the dynamics p ( s ′ | s, a ) .
Revised: = { ( s 0 , a 0 , s 1 , a 1 , · · · ) } by sampling from the initial state distribution ρ 0 ( s ) , an unobserved actor policy µ ( a | s ) , and the dynamics p ( s ′ | s, a ) .

Original: We aim to train a goal-conditioned policy π ( a | s, g ) from this arbitrary dataset with relabeled future states as the goals.
Revised: We aim to train a goal-conditioned policy π ( a | s, g ) from this arbitrary dataset with relabeled future states as the goals.

Original: ρ + ( g ) is the behavioral goal distribution assumed to be given apriori by the environment.
Revised: ρ + ( g ) is the behavioral goal distribution assumed to be given apriori by the environment.

Original: To recast the problem of goal-reaching as imitation learning equation 6, we need to set up an f -divergence minimization where we define the target (expert) distribution and the policy distribution we want to match.
Revised: To recast the problem of goal-reaching as imitation learning equation 6, we need to set up an f -divergence minimization where we define the target (expert) distribution and the policy distribution we want to match.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The training signal comes from factorizing the joint distribution of state-action-goal differently.
Revised: The training signal comes from factorizing the joint distribution of state-action-goal differently.

Original: For the relabeled target distribution, we assume an unconditioned actor µ generating a state-action distribution at first, with the goals coming from the future state distribution equation 4 conditioned on the given state and action.
Revised: For the relabeled target distribution, we assume an unconditioned actor µ generating a state-action distribution at first, with the goals coming from the future state distribution equation 4 conditioned on the given state and action.

Original: For the goal-conditioned policy distribution, behavior goals are given apriori , and the state-action distribution is generated conditioned on the behavioral goals.
Revised: For the goal-conditioned policy distribution, behavior goals are given apriori , and the state-action distribution is generated conditioned on the behavioral goals.

Original: Thus, the target distribution (see Figure 2) for states, actions, and hindsight goals is:
Revised: Thus, the target distribution (see Figure 2) for states, actions, and hindsight goals is:

Original: [Equation]
Revised: [Equation]

Original: Note that p + µ is given by equation equation 4, and ρ µ ( s, a ) is similar to ρ exp ( s, a ) in equation equation 6.
Revised: Note that p + µ is given by equation equation 4, and ρ µ ( s, a ) is similar to ρ exp ( s, a ) in equation equation 6.

Original: In the fashion of behavioral cloning (BC), if we do not care about matching the states, we can write the joint distribution we are trying to match as (see Figure 3):
Revised: In the fashion of behavioral cloning (BC), if we do not care about matching the states, we can write the joint distribution we are trying to match as (see Figure 3):

Original: [Equation]
Revised: [Equation]

Original: We can recover the objective of Hindsight Behavior Cloning (HBC) (Ding et al., 2019; Eysenbach et al., 2020a) and Goal-Conditioned Supervised Learning (GCSL) Ghosh et al.
Revised: We can recover the objective of Hindsight Behavior Cloning (HBC) (Ding et al., 2019; Eysenbach et al., 2020a) and Goal-Conditioned Supervised Learning (GCSL) Ghosh et al.

Original: via minimizing a KL-divergence:
Revised: via minimizing a KL-divergence:

Original: [Equation]
Revised: [Equation]

Original: In many cases, matching the states is more important than matching state-conditioned actions (Ross et al., 2011; Ghasemipour et al., 2020).
Revised: In many cases, matching the states is more important than matching state-conditioned actions (Ross et al., 2011; Ghasemipour et al., 2020).

Original: The joint distribution for states, actions, behavioral goals for π (see Figure 4) is:
Revised: The joint distribution for states, actions, behavioral goals for π (see Figure 4) is:

Original: [Equation]
Revised: [Equation]

Original: However, it is important to recognize that even after adding this state-matching part, there is still a missing component of the objective.
Revised: However, it is important to recognize that even after adding this state-matching part, there is still a missing component of the objective.

Original: Divergence minimization encourages the agent to stay in the "right" state-action distribution given the benefit of hindsight, but we still need the policy to actually hit the goal (see Figure 1).
Revised: Divergence minimization encourages the agent to stay in the "right" state-action distribution given the benefit of hindsight, but we still need the policy to actually hit the goal (see Figure 1).

Original: When we condition the policy on a goal, we should maximize the likelihood of seeing that goal in the future state distribution (see Figure 5).
Revised: When we condition the policy on a goal, we should maximize the likelihood of seeing that goal in the future state distribution (see Figure 5).

Original: Together with a maximum entropy regularization on the policy H ( π ) (Ho & Ermon, 2016; Schulman et al., 2017), we propose the following goal-reaching objective:
Revised: Together with a maximum entropy regularization on the policy H ( π ) (Ho & Ermon, 2016; Schulman et al., 2017), we propose the following goal-reaching objective:

Original: [Equation]
Revised: [Equation]

Original: Compared to equation 12 and equation 6, the f -divergence term equation 14(a) swaps the order between the expert and the policy.
Revised: Compared to equation 12 and equation 6, the f -divergence term equation 14(a) swaps the order between the expert and the policy.

Original: The coefficient β controls the importance of "hitting the goal".
Revised: The coefficient β controls the importance of "hitting the goal".

Original: This incentive is already implicit in hindsight-relabeled data, as in many cases a BC objective alone can learn to hit the goal as well (Ding et al., 2019; Ghosh et al., 2019; Lynch et al., 2019; Jang et al., 2021).
Revised: This incentive is already implicit in hindsight-relabeled data, as in many cases a BC objective alone can learn to hit the goal as well (Ding et al., 2019; Ghosh et al., 2019; Lynch et al., 2019; Jang et al., 2021).

Original: However, hindsight-relabeled trajectories are often sub-optimal demonstrations : not every action is acting towards achieving the goal.
Revised: However, hindsight-relabeled trajectories are often sub-optimal demonstrations : not every action is acting towards achieving the goal.

Original: Thus, maximizing the likelihood of achieving the goal is crucial, and it plays a key role in deriving the reward function of HER (Andrychowicz et al., 2017).
Revised: Thus, maximizing the likelihood of achieving the goal is crucial, and it plays a key role in deriving the reward function of HER (Andrychowicz et al., 2017).

Original:  B RIDGING G OAL -R EACHING AND G ENERATIVE M ODELING
Revised:  B RIDGING G OAL -R EACHING AND G ENERATIVE M ODELING

Original: In this section, we study how to optimize the proposed unifying objective 14.
Revised: In this section, we study how to optimize the proposed unifying objective 14.

Original: We show that the
Revised: We show that the

Original: Q -function trained for goal-reaching Q θ ( s, a, g ) is implicitly doing generative modeling.
Revised: Q -function trained for goal-reaching Q θ ( s, a, g ) is implicitly doing generative modeling.

Original: Its temporal difference is modeling the density ratio between the hindsight-relabeled distribution and the goalconditioned behavioral distribution.
Revised: Its temporal difference is modeling the density ratio between the hindsight-relabeled distribution and the goalconditioned behavioral distribution.

Original: It also approximates an energy-based model for future-state distribution when marginalized over the goal distribution.
Revised: It also approximates an energy-based model for future-state distribution when marginalized over the goal distribution.

Original: In section 4.1, we demonstrate how goalconditioned Q -learning does divergence minimization (equation 14(a)).
Revised: In section 4.1, we demonstrate how goalconditioned Q -learning does divergence minimization (equation 14(a)).

Original: In section 4.2, we show how Q -functions can define an EBM for cumulative future states, which in turn allows the policy to maximize goal likelihood (equation 14 b+c).
Revised: In section 4.2, we show how Q -functions can define an EBM for cumulative future states, which in turn allows the policy to maximize goal likelihood (equation 14 b+c).

Original: In section 4.3, we show that combining the derived results yields the HER reward.
Revised: In section 4.3, we show that combining the derived results yields the HER reward.

Original: In section 4.4, we study when a BC loss can help by analyzing the loss terms being left out by the HER reward.
Revised: In section 4.4, we study when a BC loss can help by analyzing the loss terms being left out by the HER reward.

Original: 4. D IVERGENCE M INIMIZATION WITH G OAL -C ONDITIONED Q- LEARNING
Revised: 4. D IVERGENCE M INIMIZATION WITH G OAL -C ONDITIONED Q- LEARNING

Original: This section decomposes equation 14(a).
Revised: This section decomposes equation 14(a).

Original: We start with the f -divergence bound from equations equation 5 and equation 6.
Revised: We start with the f -divergence bound from equations equation 5 and equation 6.

Original: [Equation]
Revised: [Equation]

Original: Now we negate T to get r ( s, a, g )
Revised: Now we negate T to get r ( s, a, g )

Original: =
Revised: =

Original: − T ( s, a, g ) , and the divergence minimization problem becomes:
Revised: − T ( s, a, g ) , and the divergence minimization problem becomes:

Original: [Equation]
Revised: [Equation]

Original: We can interpret r as a GAIL-style (Ho & Ermon, 2016) discriminator or reward.
Revised: We can interpret r as a GAIL-style (Ho & Ermon, 2016) discriminator or reward.

Original: However, we aim to derive a discriminator-free learning process that directly trains the Q -function corresponding to this reward Q θ ( s, a, g )
Revised: However, we aim to derive a discriminator-free learning process that directly trains the Q -function corresponding to this reward Q θ ( s, a, g )

Original: =
Revised: =

Original: r ( s, a, g ) + γ · P π Q ( s, a, g ) where P π is the transition operator :
Revised: r ( s, a, g ) + γ · P π Q ( s, a, g ) where P π is the transition operator :

Original: P π Q ( s, a, g
Revised: P π Q ( s, a, g

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]

Original: s ′ | s
Revised: ′ | s

Original: [Equation]
Revised: [Equation]

Original: a ) 
Revised: a ) π

Original: [Equation]
Revised: [Equation]

Original: a
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | s

Original: [Equation]
Revised: [Equation]

Original: [ Q

Original: [Equation]
Revised: [Equation]

Original: Re-writing the equation equation 16 w.r.t Q :
Revised: Re-writing the equation equation 16 w.r.t Q :

Original: [Equation]
Revised: [Equation]

Original: A similar change-of-variable has been explored in the context of offline RL (Nachum et al., 2019a;b) and imitation learning (Kostrikov et al., 2019; Zhu et al., 2020); we may call those methods the
Revised: A similar change-of-variable has been explored in the context of offline RL (Nachum et al., 2019a;b) and imitation learning (Kostrikov et al., 2019; Zhu et al., 2020); we may call those methods the

Original: DICE (Nachum & Dai, 2020) family.
Revised: DICE (Nachum & Dai, 2020) family.

Original: The major pain point of DICE-like methods is that they require samples from the initial state distribution ρ 0 ( s ) (Garg et al., 2021).
Revised: The major pain point of DICE-like methods is that they require samples from the initial state distribution ρ 0 ( s ) (Garg et al., 2021).

Original: Here, the following lemma shows that in the goal-conditioned case, we can use arbitrary offline trajectories to evaluate the expected rewards under goal-conditioned online rollouts:
Revised: Here, the following lemma shows that in the goal-conditioned case, we can use arbitrary offline trajectories to evaluate the expected rewards under goal-conditioned online rollouts:

Original: Lemma 4.1 (Online-to-offline transformation for goal reaching) .
Revised: Lemma 4.1 (Online-to-offline transformation for goal reaching) .

Original: Given a goal-conditioned policy π ( a | s, g ) , its corresponding Q -function Q π ( s, a, g ) , and arbitrary state-action visitation distribution ρ µ ( s, a ) of another policy µ ( a | s ) , the expected temporal difference for online rollouts under π is:
Revised: Given a goal-conditioned policy π ( a | s, g ) , its corresponding Q -function Q π ( s, a, g ) , and arbitrary state-action visitation distribution ρ µ ( s, a ) of another policy µ ( a | s ) , the expected temporal difference for online rollouts under π is:

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 4.1, the objective in equation 16 now becomes:
Revised: Using Lemma 4.1, the objective in equation 16 now becomes:

Original: [Equation]
Revised: [Equation]

Original: The function f ∗ is the convex conjugate of f in f -divergence.
Revised: The function f ∗ is the convex conjugate of f in f -divergence.

Original: We can pick almost any convex function as f ∗ as long as (( f ∗ ) ∗ )(1) = 0 .
Revised: We can pick almost any convex function as f ∗ as long as (( f ∗ ) ∗ )(1) = 0 .

Original: For instance, in the tradition of Q -learning, we can use:
Revised: For instance, in the tradition of Q -learning, we can use:

Original: [Equation]
Revised: [Equation]

Original: where r and c are constants.
Revised: where r and c are constants.

Original: Its convex conjugate is f ( x ) =
Revised: Its convex conjugate is f ( x ) =

Original: ( f ∗ ) ∗ ( x ) =
Revised: ( f ∗ ) ∗ ( x ) =

Original: x 2 / 2 − rx − c .
Revised: x 2 / 2 − rx − c .

Original: For any r , we can always pick a c that will ensure f (1) =
Revised: For any r , we can always pick a c that will ensure f (1) =

Original: ( f ∗ ) ∗ (1) = 0 , and c does not affect the learning process.
Revised: ( f ∗ ) ∗ (1) = 0 , and c does not affect the learning process.

Original: In summary, we have derived a way to minimize the first term D f ( p µ ( s, a, s + ) ∥ p π ( s, a, g )) in equation 14 directly using goal-conditioned Q -learning.
Revised: In summary, we have derived a way to minimize the first term D f ( p µ ( s, a, s + ) ∥ p π ( s, a, g )) in equation 14 directly using goal-conditioned Q -learning.

Original: 4. EBM FOR P REDICTING ( AND C ONTROLLING ) THE F UTURE
Revised: 4. EBM FOR P REDICTING ( AND C ONTROLLING ) THE F UTURE

Original: In this section, we study how to maximize the goal likelihood term equation 14(b) by defining an energy-based model (EBM) to predict the future-state distribution for a policy, which the policy can in turn optimize to control the future-state distribution.
Revised: In this section, we study how to maximize the goal likelihood term equation 14(b) by defining an energy-based model (EBM) to predict the future-state distribution for a policy, which the policy can in turn optimize to control the future-state distribution.

Original: We start with the Bayes rule:
Revised: We start with the Bayes rule:

Original: [Equation]
Revised: [Equation]

Original: This relationship reflects that, if we define an EBM for ρ π ( s, a | s + ) =
Revised: This relationship reflects that, if we define an EBM for ρ π ( s, a | s + ) =

Original: exp q
Revised: exp q

Original: ( s, a, s + ) /Z
Revised: ( s, a, s + ) /Z

Original: q ( s + ) where Z q ( s + ) =
Revised: q ( s + ) where Z q ( s + ) =

Original: (cid:82) S×A exp q ( s, a, s + )
Revised: (cid:82) S×A exp q ( s, a, s + )

Original: d s d a , then we can contrast the dynamics defined in equationwith the marginal goal distribution defined in equation 13.
Revised: d s d a , then we can contrast the dynamics defined in equationwith the marginal goal distribution defined in equation 13.

Original: In a slightly overloaded notation, we will now define a Q -function as Q θ ( s, a, g )
Revised: In a slightly overloaded notation, we will now define a Q -function as Q θ ( s, a, g )

Original: =
Revised: =

Original: q ( s, a, g ) − log Z q ( g ) .
Revised: q ( s, a, g ) − log Z q ( g ) .

Original: For now, we shall assume that this
Revised: For now, we shall assume that this

Original: Q -function is separate from the one defined in equation 18.
Revised: Q -function is separate from the one defined in equation 18.

Original: Rearranging equation 20 , and setting ρ + ( g ) to be ρ + ( s + ) , we see the density ratio can be expressed as:
Revised: Rearranging equation 20 , and setting ρ + ( g ) to be ρ + ( s + ) , we see the density ratio can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.2 (Gradient of the noise-contrastive term in energy-based goal-reaching) .
Revised: Lemma 4.2 (Gradient of the noise-contrastive term in energy-based goal-reaching) .

Original: Given the following definition for the logit of a NCE-like binary classifier, with ρ + ( g ) =
Revised: Given the following definition for the logit of a NCE-like binary classifier, with ρ + ( g ) =

Original: ρ + ( g ) :
Revised: ρ + ( g ) :

Original: [Equation]
Revised: [Equation]

Original: The gradient of the negative NCE term in the density ratio estimation approaches zero as k → ∞ :
Revised: The gradient of the negative NCE term in the density ratio estimation approaches zero as k → ∞ :

Original: [Equation]
Revised: [Equation]

Original: As for the positive classification term in NCE equation 9, we can make a similar argument that ∇ 
Revised: As for the positive classification term in NCE equation 9, we can make a similar argument that ∇ [Equation] + ,

Original: [Equation]
Revised: [Equation]

Original: 0 (see appendix B.2).
Revised: 0 (see appendix B.2).

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: To optimize the above objective on arbitrary behaviour data, we have to consider the fact that we do not have complete access to the distribution of "positive samples" p + π ( s + | s, a ) , as sampling directly from this distribution requires on-policy rollouts.
Revised: To optimize the above objective on arbitrary behaviour data, we have to consider the fact that we do not have complete access to the distribution of "positive samples" p + π ( s + | s, a ) , as sampling directly from this distribution requires on-policy rollouts.

Original: Utilizing importance weights to learn from off-policy data while accounting for hindsight bias yields the following loss minimization instead:
Revised: Utilizing importance weights to learn from off-policy data while accounting for hindsight bias yields the following loss minimization instead:

Original: [Equation]
Revised: [Equation]

Original: where w ( s, a, s ′ , g ) contrasts the goals that will likely be met by the agent beyond a single step of dynamics p ( s ′ | s, a ) in the future with other random goals that the agent has seen in the past:
Revised: where w ( s, a, s ′ , g ) contrasts the goals that will likely be met by the agent beyond a single step of dynamics p ( s ′ | s, a ) in the future with other random goals that the agent has seen in the past:

Original: [Equation]
Revised: [Equation]

Original: See Appendix B.3 for a full derivation
Revised: See Appendix B.3 for a full derivation.

Original: [Equation]

Original: 4. D ERIVING HER R EWARDS
Revised: 4. D ERIVING HER R EWARDS

Original: For a goal-conditioned Q -function Q θ ( s, a, g ) , besides optimizing a Bellman residual as a regular
Revised: For a goal-conditioned Q -function Q θ ( s, a, g ) , besides optimizing a Bellman residual as a regular

Original: Q ( s, a ) function would do, it has an additional degree-of-freedom for the goal g which can be used to define an EBM that models the future-state distribution.
Revised: Q ( s, a ) function would do, it has an additional degree-of-freedom for the goal g which can be used to define an EBM that models the future-state distribution.

Original: In such an EBM, the Q -function is used as the negative energy defined in equation 21.
Revised: In such an EBM, the Q -function is used as the negative energy defined in equation 21.

Original: Combining the first term in the f -divergence minimization part equation 18 and the single-step dynamics loss term in the EBM learning 24 gives us:
Revised: Combining the first term in the f -divergence minimization part equation 18 and the single-step dynamics loss term in the EBM learning 24 gives us:

Original: [Equation]
Revised: [Equation]

Original: The crucial observation here is, given that f ∗ is a quadratic 19, and that there is a stop gradient sign on P π Q because it uses a target network, the loss above can be re-packaged into a single quadratic loss of Bellman residuals for a specific reward thanks to the property of the dynamics defined in 4:
Revised: The crucial observation here is, given that f ∗ is a quadratic 19, and that there is a stop gradient sign on P π Q because it uses a target network, the loss above can be re-packaged into a single quadratic loss of Bellman residuals for a specific reward thanks to the property of the dynamics defined in 4:

Original: [Equation]
Revised: [Equation]

Original: See Appendix B.4 for a full derivation.
Revised: See Appendix B.4 for a full derivation.

Original: Setting r = − 1 and β = 1 , we have arrived at the reward function of HER r HER ( s, a, s ′ , s + ) .
Revised: Setting r = − 1 and β = 1 , we have arrived at the reward function of HER r HER ( s, a, s ′ , s + ) .

Original: The remaining losses that this reward leaves out are the second term in f -divergence minimization part equation 18 and the second term in the EBM training equation 53, which we will analyze in the next section.
Revised: The remaining losses that this reward leaves out are the second term in f -divergence minimization part equation 18 and the second term in the EBM training equation 53, which we will analyze in the next section.

Original: 4. W HEN DOES B EHAVIOUR C LONING (BC) H ELP ?
Revised: 4. W HEN DOES B EHAVIOUR C LONING (BC) H ELP ?

Original: We now proceed to answer the following question: when does BC help?
Revised: We now proceed to answer the following question: when does BC help?

Original: Our idea is to identify the conditions for a BC-like loss to emerge in the remaining losses unaccounted by the HER reward: a loss term in 18 about pushing Q -values of the current policy down, and the multi-step dynamics term in EBM 54.
Revised: Our idea is to identify the conditions for a BC-like loss to emerge in the remaining losses unaccounted by the HER reward: a loss term in 18 about pushing Q -values of the current policy down, and the multi-step dynamics term in EBM 54.

Original: We may argue that, by not pushing the Q -values of the current policy down, the
Revised: We may argue that, by not pushing the Q -values of the current policy down, the

Original: HER agent becomes more exploratory; by not pushing up the Q -values of the discounted future-state distribution beyond a single step, the HER agent is encouraged to reach a goal sooner rather than later.
Revised: HER agent becomes more exploratory; by not pushing up the Q -values of the discounted future-state distribution beyond a single step, the HER agent is encouraged to reach a goal sooner rather than later.

Original: Nevertheless, we shall see that combining those two remaining terms produces a BC-like term, which imitates the best actions based on how much an action moves the agent closer to the goal.
Revised: Nevertheless, we shall see that combining those two remaining terms produces a BC-like term, which imitates the best actions based on how much an action moves the agent closer to the goal.

Original: We start with the following property of Boltzmann policies π ( a | s, g ) ∝ exp Q ( s, a, g ) (Haarnoja et al., 2017; Schulman et al., 2017) (which we can apply thanks to entropy regularization in 14):
Revised: We start with the following property of Boltzmann policies π ( a | s, g ) ∝ exp Q ( s, a, g ) (Haarnoja et al., 2017; Schulman et al., 2017) (which we can apply thanks to entropy regularization in 14):

Original: [Equation]
Revised: [Equation]

Original: In equation 25, the Q -value of a particular action a is pushed up under the following condition:
Revised: In equation 25, the Q -value of a particular action a is pushed up under the following condition:

Original: [Equation]
Revised: [Equation]

Original: Both denominators on the left and right sides are averaged over all possible behavioral goals, so their values should be roughly the same.
Revised: Both denominators on the left and right sides are averaged over all possible behavioral goals, so their values should be roughly the same.

Original: Moreover, we approximate the average-exponential operation by using the max operation already computed in the backup operator (either exact (Mnih et al., 2015;
Revised: Moreover, we approximate the average-exponential operation by using the max operation already computed in the backup operator (either exact (Mnih et al., 2015;

Original: Van Hasselt et al., 2016) or approximate (Lillicrap et al., 2015; Haarnoja et al., 2018)) and lower the threshold γ hdm (compared to γ ) accordingly.
Revised: Van Hasselt et al., 2016) or approximate (Lillicrap et al., 2015; Haarnoja et al., 2018)) and lower the threshold γ hdm (compared to γ ) accordingly.

Original: After those changes, the indicator function that decides whether the value of an action a should be pushed up becomes:
Revised: After those changes, the indicator function that decides whether the value of an action a should be pushed up becomes:

Original: [Equation]
Revised: [Equation]

Original: Combining with the first term in 29, we arrive at a BC-like loss 12, but with an indicator weighting:
Revised: Combining with the first term in 29, we arrive at a BC-like loss 12, but with an indicator weighting:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this term imitates a particular action when the value functions believe that this action can move the agent closer to the goal by at least − log γ hdm steps.
Revised: Intuitively, this term imitates a particular action when the value functions believe that this action can move the agent closer to the goal by at least − log γ hdm steps.

Original: The idea of imitating the best actions is similar to self-imitation learning (SIL)
Revised: The idea of imitating the best actions is similar to self-imitation learning (SIL)

Original: (Oh et al., 2018; Vinyals et al., 2019), but our algorithmoperates in a (reward-free) goal-reaching setting, with the advantage function in SIL being replaced by the delta of reachability a particular action can produce in getting closer to the goal.
Revised: (Oh et al., 2018; Vinyals et al., 2019), but our algorithmoperates in a (reward-free) goal-reaching setting, with the advantage function in SIL being replaced by the delta of reachability a particular action can produce in getting closer to the goal.

Original: Given : Batch of data { ( s, a, s ′ , s + ) } , where s + is sampled via hindsight relabeling.
Revised: Given : Batch of data { ( s, a, s ′ , s + ) } , where s + is sampled via hindsight relabeling.

Original: [Equation]
Revised: [Equation]

Original: ▷ r

Original: [Equation]
Revised: [Equation]

Original: constant,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ▷ Push up the values for reaching next states 3:
Revised: ▷ Push up the values for reaching next states 3:

Original: [Equation]
Revised: [Equation]

Original: ▷ Behaviour Cloning like loss 4: 
Revised: ▷ Behaviour Cloning like loss 4: L = L − [Equation] [Equation] max

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  γ
Revised: [Equation] s [Equation] < log γ

Revised: [Equation]

Revised: L

Original: [Equation]
Revised: [Equation]

Original: 5: minimize L , update the target network ˆ Q
Revised: 5: minimize L , update the target network ˆ Q

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: 5. S ELF -S UPERVISED G OAL -R EACHING S ETUP
Revised: 5. S ELF -S UPERVISED G OAL -R EACHING S ETUP

Original: We consider the self-supervised goal-reaching setting for the environments described in GCSL (Ghosh et al., 2019).
Revised: We consider the self-supervised goal-reaching setting for the environments described in GCSL (Ghosh et al., 2019).

Original: While the original HER (Andrychowicz et al., 2017) assumes that the agent has direct access to the ground truth binary reward metric (which is used when relabeling is performed), we do not make such an assumption as it can be unrealistic for real-world robot-learning (Lin et al., 2019).
Revised: While the original HER (Andrychowicz et al., 2017) assumes that the agent has direct access to the ground truth binary reward metric (which is used when relabeling is performed), we do not make such an assumption as it can be unrealistic for real-world robot-learning (Lin et al., 2019).

Original: Instead, we simply use next-state relabeling to provide positive rewards.
Revised: Instead, we simply use next-state relabeling to provide positive rewards.

Original: As seen in Eq equation 28, a positive reward is provided only when the relabeled hindsight goal is the immediate next state.
Revised: As seen in Eq equation 28, a positive reward is provided only when the relabeled hindsight goal is the immediate next state.

Original: The benefit of the next-state relabeling reward is that the training procedure is now completely self-supervised, and therefore provides a fair comparison to GCSL (Ghosh et al., 2019). Success
Revised: The benefit of the next-state relabeling reward is that the training procedure is now completely self-supervised, and therefore provides a fair comparison to GCSL (Ghosh et al., 2019)

Original: [Table]
Revised: [Table]

Original: 5. C OMPARISONS AND A BLATIONS
Revised: 5. C OMPARISONS AND A BLATIONS

Original: As shown in Table 1, we compare HDM to the following baselines in terms of their goal-reaching abilities: GCSL (Ghosh et al., 2019) /
Revised: As shown in Table 1, we compare HDM to the following baselines in terms of their goal-reaching abilities: GCSL (Ghosh et al., 2019) /

Original: HBC (Ding et al., 2019), HER (Andrychowicz et al., 2017) with (0 , 1) rewards, HER with Soft Q-Learning (SQL) (Schulman et al., 2017), HER with ( − 1 , 0) rewards, and HER + HBC.
Revised: HBC (Ding et al., 2019), HER (Andrychowicz et al., 2017) with (0 , 1) rewards, HER with Soft Q-Learning (SQL) (Schulman et al., 2017), HER with ( − 1 , 0) rewards, and HER + HBC.

Original: HER with (0 , 1) reward gives the agent a reward of 1 when the goal is reached and 0 otherwise; HER with ( − 1 , 0) reward gives a reward of 0 when a goal is reached and −otherwise.
Revised: HER with (0 , 1) reward gives the agent a reward of 1 when the goal is reached and 0 otherwise; HER with ( − 1 , 0) reward gives a reward of 0 when a goal is reached and −otherwise.

Original: Those two types of rewards lead to different learning dynamics because the Q -function is initialized to output values around 0 .
Revised: Those two types of rewards lead to different learning dynamics because the Q -function is initialized to output values around 0 .

Original: We also compare against HER
Revised: We also compare against HER

Original: + SQL, because soft Q -learning is known to improve the robustness of learning (Haarnoja et al., 2018).
Revised: + SQL, because soft Q -learning is known to improve the robustness of learning (Haarnoja et al., 2018).

Original: HDM builds on top of HER with ( − 1 , 0) rewards and adds a BC-like loss with a clipping condition equation 32 such that only the actions that move an agent closer to the goal get imitated (see Figure 6).
Revised: HDM builds on top of HER with ( − 1 , 0) rewards and adds a BC-like loss with a clipping condition equation 32 such that only the actions that move an agent closer to the goal get imitated (see Figure 6).

Original: Indeed, combining HER with HBC (which blindly imitates all actions in hindsight) produces worse results than not imitating at all and only resorting to value learning; HDM allows for better control over what to imitate.
Revised: Indeed, combining HER with HBC (which blindly imitates all actions in hindsight) produces worse results than not imitating at all and only resorting to value learning; HDM allows for better control over what to imitate.

Original: The results in Table 1 and Figure 7 show that HDM achieves the strongest performance on all environments, while reducing variances in success rates.
Revised: The results in Table 1 and Figure 7 show that HDM achieves the strongest performance on all environments, while reducing variances in success rates.

Original: Interestingly, there is no consensus best baseline
Revised: Interestingly, there is no consensus best baseline

Original: algorithm
Revised: algorithm

Original: , with all five algorithms achieving good results in some environments and subpar performance in others.
Revised: , with all five algorithms achieving good results in some environments and subpar performance in others.

Original: In Figure 10 of the appendix, we ablate the additional hyper-parameters introduced by HDM: γ hdm in equation 31 and the β term in equation 14.
Revised: In Figure 10 of the appendix, we ablate the additional hyper-parameters introduced by HDM: γ hdm in equation 31 and the β term in equation 14.

Original: Intuitively, γ hdm controls the threshold that determines when an action is considered good enough for imitation, and β controls the trade-off between minimizing f -divergence and maximizing future goal likelihood in equation 14.
Revised: Intuitively, γ hdm controls the threshold that determines when an action is considered good enough for imitation, and β controls the trade-off between minimizing f -divergence and maximizing future goal likelihood in equation 14.

Original: The ablation shows that HDM outperforms HER and GCSL across a variety of γ hdm and β values.
Revised: The ablation shows that HDM outperforms HER and GCSL across a variety of γ hdm and β values.

Original: 5. W HY DOES H INDSIGHT BC SOMETIMES FAIL ?
Revised: 5. W HY DOES H INDSIGHT BC SOMETIMES FAIL ?

Original: In this section, we aim to better understand whether the (underwhelming) performance of hindsight BC had anything to do with our specific setting.
Revised: In this section, we aim to better understand whether the (underwhelming) performance of hindsight BC had anything to do with our specific setting.

Original: We report an interesting metric that correlates with the performance of GCSL / HBC, which can be measured prior to training a policy: initial ag (achieved-goal) change ratio .
Revised: We report an interesting metric that correlates with the performance of GCSL / HBC, which can be measured prior to training a policy: initial ag (achieved-goal) change ratio .

Original: We define the ag change ratio of π to be: the percentage of trajectories where the achieved goals in initial states s 0 are different from the achieved goals in final states s T under π .
Revised: We define the ag change ratio of π to be: the percentage of trajectories where the achieved goals in initial states s 0 are different from the achieved goals in final states s T under π .

Original: Most training signals are only created from ag changes, because they provide examples of how to rearrange an environment.
Revised: Most training signals are only created from ag changes, because they provide examples of how to rearrange an environment.

Original: We then define initial ag change ratio to be the ag change ratio of a random-acting policy π 0 .
Revised: We then define initial ag change ratio to be the ag change ratio of a random-acting policy π 0 .

Original: Using notation from 28, it can be computed as E s 0 ··· s T ∼ π 

Revised: Using notation from 28, it can be computed as E s 0 ··· s T ∼ π

Original: [ − r HER ( · , · , s 0 , s T )] .
Revised: [ − r HER ( · , · , s 0 , s T )] .

Original: In Figure 8, we show the relationship between final success rates versus the initial ag (achieved-goal) change ratio, across all environments.
Revised: In Figure 8, we show the relationship between final success rates versus the initial ag (achieved-goal) change ratio, across all environments.

Original: The performance of GCSL seems to be upper-bounded by a linear relationship between the two.
Revised: The performance of GCSL seems to be upper-bounded by a linear relationship between the two.

Original: This makes intuitive sense: a BC-style objective starts off cloning the initially random trajectories, so if initial ag change ratio is low, the policy would not learn to rearrange ag , compounding to a low final performance.
Revised: This makes intuitive sense: a BC-style objective starts off cloning the initially random trajectories, so if initial ag change ratio is low, the policy would not learn to rearrange ag , compounding to a low final performance.

Original: HER and HDM are able to surpass this upper ceiling likely because of the additional goal-likelihood term equation 14 besides imitation.
Revised: HER and HDM are able to surpass this upper ceiling likely because of the additional goal-likelihood term equation 14 besides imitation.

Original: This finding suggests that in order to make goal-reaching easier , we should either modify the initial state distribution ρ 0 ( s ) such that ag can be easily changed through random exploration (Florensa et al., 2017)
Revised: This finding suggests that in order to make goal-reaching easier , we should either modify the initial state distribution ρ 0 ( s ) such that ag can be easily changed through random exploration (Florensa et al., 2017)

Original: (if the policy is training from scratch), or initialize BC from some high-quality demonstrations where ag does change (Ding et al., 2019; Nair et al., 2018a; Lynch et al., 2019).
Revised: (if the policy is training from scratch), or initialize BC from some high-quality demonstrations where ag does change (Ding et al., 2019; Nair et al., 2018a; Lynch et al., 2019).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work presents a unified goal-reaching objective that encompasses a family of goal-conditioned
Revised: This work presents a unified goal-reaching objective that encompasses a family of goal-conditioned

Original: RL algorithms.
Revised: RL algorithms.

Original: Our derivation illustrates the connection between hindsight goal relabeling, divergence minimization, and energy-based models.
Revised: Our derivation illustrates the connection between hindsight goal relabeling, divergence minimization, and energy-based models.

Original: It reveals that there is a largely unexplored design space: we could potentially use other convex functions in f -divergence minimization (Nowozin et al., 2016;
Revised: It reveals that there is a largely unexplored design space: we could potentially use other convex functions in f -divergence minimization (Nowozin et al., 2016;

Original: Ghasemipour et al., 2020), and improve the optimization of the energy-based model (Du et al., 2020) by going beyond NCE (Grathwohl et al., 2020).
Revised: Ghasemipour et al., 2020), and improve the optimization of the energy-based model (Du et al., 2020) by going beyond NCE (Grathwohl et al., 2020).

Original: The primary limitation of our framework is that it does not account for exploration (Hafner et al., 2020).
Revised: The primary limitation of our framework is that it does not account for exploration.

Original: Eventually, we would like to incorporate empowerment (Klyubin et al., 2005;
Revised: Eventually, we would like to incorporate empowerment (Klyubin et al., 2005;

Original: Sekar et al., 2020) and input density exploration (Bellemare et al., 2016;
Revised: Sekar et al., 2020) and input density exploration (Bellemare et al., 2016;

Original: Pong et al., 2019) into the framework.
Revised: Pong et al., 2019) into the framework.

Revised:  A CKNOWLEDGEMENT

Revised: We would like to thank Harris Chan, Kamyar Ghasemipour and Raquel Urtasun for their valuable feedback on an earlier draft.

Revised: Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the

Revised: Vector Institute for Artificial Intelligence ( https://vectorinstitute.ai/partners/ ).

Original: A N OTATIONS
Revised: A N OTATIONS

Original: ρ
Revised: ρ

Original: Initial state distribution p ( s ′ | s, a )
Revised: Initial state distribution p ( s ′ | s, a )

Original: Environmental dynamics p + µ ( s + | s, a )
Revised: Environmental dynamics p + µ ( s + | s, a )

Original: Discounted future state distribution under policy µ ρ µ ( s )
Revised: Discounted future state distribution under policy µ ρ µ ( s )

Original: State distribution (occupancy measure) visited by policy µ
Revised: State distribution (occupancy measure) visited by policy µ

Original: P π Q ( s, a, g )
Revised: P π Q ( s, a, g )

Original: Transition operator in (goal-conditioned) Bellman backup.
Revised: Transition operator in (goal-conditioned) Bellman backup.

Original: P π Q ( s, a, g ) =
Revised: P π Q ( s, a, g ) =

Original: E p ( s ′ | s,a ) π ( a ′ | s ′ ,g )
Revised: E p ( s ′ | s,a ) π ( a ′ | s ′ ,g )

Original: [ Q θ ( s ′ , a ′ , g )] .
Revised: [ Q θ ( s ′ , a ′ , g )] .

Original: f
Revised: f

Original: Convex function in f -divergence D f .
Revised: Convex function in f -divergence D f .

Original: Usually f (1) = 0 .
Revised: Usually f (1) = 0 .

Original: f ∗
Revised: f ∗

Original: Convex conjugate of function f f ′ ( x )
Revised: Convex conjugate of function f f ′ ( x )

Original: Derivative of function f ρ exp ( s, a )
Revised: Derivative of function f ρ exp ( s, a )

Original: State-action visitation distribution of the expert policy ρ µ ( s, a )
Revised: State-action visitation distribution of the expert policy ρ µ ( s, a )

Original: State-action visitation distribution of the policy µ .
Revised: State-action visitation distribution of the policy µ .

Original: Note that ρ µ ( s, a ) =
Revised: Note that ρ µ ( s, a ) =

Original: ρ µ ( s ) µ ( a | s ) .
Revised: ρ µ ( s ) µ ( a | s ) .

Original: T ( x )
Revised: T ( x )

Original: Function used in the variational bound of f -divergence.
Revised: Function used in the variational bound of f -divergence.

Original: Also appears as T ( s, a ) or T ( s, a, g ) in the main text.
Revised: Also appears as T ( s, a ) or T ( s, a, g ) in the main text.

Original: ρ + ( g )
Revised: ρ + ( g )

Original: The behaviour goal distribution assumed to be given apriori by the environment.
Revised: The behaviour goal distribution assumed to be given apriori by the environment.

Original: ρ + ( s + )
Revised: ρ + ( s + )

Original: The marginal hindsight goal distribution of a given dataset / replay, where ρ + ( s + ) =
Revised: The marginal hindsight goal distribution of a given dataset / replay, where ρ + ( s + ) =

Original: E ρ µ ( s,a ) [ p + µ
Revised: E ρ µ ( s,a ) [ p + µ

Original: ( s + | s, a )] ˆ π ( a | s )
Revised: ( s + | s, a )] ˆ π ( a | s )

Original: The goal-conditioned policy π marginalized over the behavioral goal distribution ˆ π ( a | s ) =
Revised: The goal-conditioned policy π marginalized over the behavioral goal distribution ˆ π ( a | s ) =

Original: E p
Revised: E p

Original: + ( g ) [ π ( a | s, g )] .
Revised: + ( g ) [ π ( a | s, g )] .

Original: ρ π ( s, a | g )
Revised: ρ π ( s, a | g )

Original: The state-action visitation distribution of goal-conditioned policy π when conditioned on the behavioral goal g
Revised: The state-action visitation distribution of goal-conditioned policy π when conditioned on the behavioral goal g

Original: E ( x )
Revised: E ( x )

Original: Energy function of an EBM: p θ ( x ) =
Revised: Energy function of an EBM: p θ ( x ) =

Original: exp( − E θ ( x )) /Z
Revised: exp( − E θ ( x )) /Z

Original: ( θ ) .
Revised: ( θ ) .

Original: Z ( θ )
Revised: Z ( θ )

Original: The partition function in an EBM Z ( θ ) = (cid:82) X exp
Revised: The partition function in an EBM Z ( θ ) = (cid:82) X exp

Original: [Table]
Revised: [Table]

Original: [Equation]

Revised: The number of

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: noise

Original: [Table]
Revised: [Table]

Original: The number of times noise samples are sampled more frequently than truedata samples in NCE.
Revised: are sampled more frequently than true

Original: [Table]
Revised: [Table]

Original: The logit of the positive sample classification loss inNCE.
Revised: in

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: The logit of the positive

Original: [Table]
Revised: [Table]

Original: partition function of a conditional EBM
Revised: classification loss in

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The (learned) reward in traditional RL settings without goalconditioning.
Revised: The (learned) reward in traditional RL settings without goalconditioning.

Original: [Table]
Revised: [Table]

Original: The (learned) reward in goal-conditioned divergence minimization.
Revised: The (learned) reward in goal-conditioned divergence minimization.

Original: [Table]
Revised: [Table]

Original: s, a
Revised: s [Table] a

Original: [Table]
Revised: [Table]

Original: s
Revised: s

Original: [Table]
Revised: [Table]

Original: s + ) The (generalized) reward used in HER-style multi-goal RL. 
Revised: The (generalized) reward used in HER-style multi-goal RL. [Table] HER [Table]

Revised: s

Original: [Table]
Revised: [Table]

Original: s, a, s ′ , s + )
Revised: a, s ′ , s + )

Original: The reward function used in HER.
Revised: The reward function used in HER.

Original: It equals 0 when s ′ =
Revised: It equals 0 when s ′ =

Original: s + and − 1 when s ′ ̸
Revised: s + and − 1 when s ′ ̸

Original: = s + .
Revised: = s + .

Original: Also denoted as r HER ( · , · , s ′ , s + ) .
Revised: Also denoted as r HER ( · , · , s ′ , s + ) .

Original: B P ROOFS B. 
Revised: B P ROOFS B.

Original: [Equation]
Revised: [Equation]

Original: Lemma B.1 (Online-to-offline transformation for goal reaching) .
Revised: Lemma B.1 (Online-to-offline transformation for goal reaching) .

Original: Given a goal-conditioned policy π ( a | s, g ) , its corresponding Q -function Q π ( s, a, g )
Revised: Given a goal-conditioned policy π ( a | s, g ) , its corresponding Q -function Q π ( s, a, g )

Original: [Equation]
Revised: [Equation]

Original: and arbitrary state-action visitation distribution ρ µ ( s, a ) of another policy µ ( a | s )
Revised: and arbitrary state-action visitation distribution ρ µ ( s, a ) of another policy µ ( a | s )

Original: [Equation]
Revised: [Equation]

Original: the expected temporal difference for online rollouts under π is:
Revised: the expected temporal difference for online rollouts under π is:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma B
Revised: Lemma B

Original: [Equation]
Revised: [Equation]

Original: 2 (Gradient of the noise-contrastive term in energy-based goal-reaching) .
Revised: 2 (Gradient of the noise-contrastive term in energy-based goal-reaching) .

Original: Given the following definition for the logit of a NCE-like binary classifier
Revised: Given the following definition for the logit of a NCE-like binary classifier

Original: [Equation]
Revised: [Equation]

Original: with ρ + ( g ) =
Revised: with ρ + ( g ) =

Original: ρ + ( g ) :
Revised: ρ + ( g ) :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The gradient of the negative NCE term in the density ratio estimation approaches zero as k → ∞ :
Revised: The gradient of the negative NCE term in the density ratio estimation approaches zero as k → ∞ :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma B.2.
Revised: Proof of Lemma B.2.

Original: σ is the sigmoid function, and Z θ (s
Revised: σ is the sigmoid function, and Z θ (

Original: [Equation]
Revised: [Equation]

Original: Plugging in the above into the loss and taking the gradient:
Revised: Plugging in the above into the loss and taking the gradient:

Original: [Equation]
Revised: [Equation]

Original: The first term inside the expectation of ρ µ ( s, a ) :
Revised: The first term inside the expectation of ρ µ ( s, a ) :

Original: [Equation]
Revised: [Equation]

Original: The second term:
Revised: The second term:

Original: [Equation]
Revised: [Equation]

Original: The two terms cancel out and yield a gradient of 0 .
Revised: The two terms cancel out and yield a gradient of 0 .

Original: Lemma B.3 (Goal-conditioned Q-functions estimate PMI on given trajectories) .
Revised: Lemma B.3 (Goal-conditioned Q-functions estimate PMI on given trajectories) .

Original: Given that we set apriori the behavioral goal distribution to be ρ + ( g ) = (cid:82) S×A p µ ( s, a, s + )d s d a .
Revised: Given that we set apriori the behavioral goal distribution to be ρ + ( g ) = (cid:82) S×A p µ ( s, a, s + )d s d a .

Original: And assuming that the state-action distribution of π marginalized over behavioral goals E ρ + ( g ) [ ρ π ( s, a | g )] is the same as ρ µ ( s, a ) , which means π stays in the same state-action visitation distribution as µ .
Revised: And assuming that the state-action distribution of π marginalized over behavioral goals E ρ + ( g ) [ ρ π ( s, a | g )] is the same as ρ µ ( s, a ) , which means π stays in the same state-action visitation distribution as µ .

Original: Then: on trajectories generated by µ , the point-wise mutual information (PMI) between a state-action pair ( s, a ) and a future state g is given by the Q -function at convergence Q π :
Revised: Then: on trajectories generated by µ , the point-wise mutual information (PMI) between a state-action pair ( s, a ) and a future state g is given by the Q -function at convergence Q π :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma B.3.
Revised: Proof of Lemma B.3.

Original: We first start with the optimal T ∗ in the f -divergence bound equation 5, applied to the special case of the function f being a quadratic equation 19:
Revised: We first start with the optimal T ∗ in the f -divergence bound equation 5, applied to the special case of the function f being a quadratic equation 19:

Original: [Equation]
Revised: [Equation]

Original: Combining the two, we get:
Revised: Combining the two, we get:

Original: [Equation]
Revised: [Equation]

Original: Now applying this identity to the f -divergence minimization problem in equation 15 (note that we have set r ( s, a, g )
Revised: Now applying this identity to the f -divergence minimization problem in equation 15 (note that we have set r ( s, a, g )

Original: = − T ( s, a, g ) in our derivation):
Revised: = − T ( s, a, g ) in our derivation):

Original: [Equation]
Revised: [Equation]

Original: We now take the relationship in equation 21:
Revised: We now take the relationship in equation 21:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using this substitution
Revised: Using this substitution

Original: [Equation]
Revised: [Equation]

Original: we arrive at:
Revised: we arrive at:

Original: [Equation]
Revised: [Equation]

Original: Swapping the nominator and denominator and applying the assumption that ρ π ( s, a )
Revised: Swapping the nominator and denominator and applying the assumption that ρ π ( s, a )

Original: =
Revised: =

Original: ρ µ ( s
Revised: ρ µ ( s

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]
Revised: [Equation]

Revised: Taking the log on both sides

Original: [Equation]
Revised: [Equation]

Original: Taking the log on both sides, we get the following expression of PMI (( s, a ) , g ) :
Revised: we get the following expression of PMI (( s, a ) , g ) :

Original: [Equation]
Revised: [Equation]

Original: B. NCE L OSSES
Revised: B. NCE L OSSES

Original: We now complete the derivation of the NCE losses in section 4.2, by illustrating that the positive classification loss of NCE reduces to an InfoNCE (Van den Oord et al., 2018) loss in equation equation 23 under our definition of the logit equation 22.
Revised: We now complete the derivation of the NCE losses in section 4.2, by illustrating that the positive classification loss of NCE reduces to an InfoNCE (Van den Oord et al., 2018) loss in equation equation 23 under our definition of the logit equation 22.

Original: More specifically, given that we already have Lemma B.2, we only need to show:
Revised: More specifically, given that we already have Lemma B.2, we only need to show:

Original: [Equation]
Revised: [Equation]

Original: We simply follow the definition of ∆ θ in equation equation 22 and take the gradient of the above:
Revised: We simply follow the definition of ∆ θ in equation equation 22 and take the gradient of the above:

Original: [Equation]
Revised: [Equation]

Original: As k → ∞ , we see that the gradient approaches 0 because the scalar on the left approaches 0 .
Revised: As k → ∞ , we see that the gradient approaches 0 because the scalar on the left approaches 0 .

Original: Combining the above result about a part of the positive classification loss in NCE with the Lemma in
Revised: Combining the above result about a part of the positive classification loss in NCE with the Lemma in

Original: B.2 which deals with the negative classification loss in NCE, we can arrive at the combined NCE loss at its limit k → ∞ , as pointed out in equation 23 of the main text:
Revised: B.2 which deals with the negative classification loss in NCE, we can arrive at the combined NCE loss at its limit k → ∞ , as pointed out in equation 23 of the main text:

Original: [Equation]
Revised: [Equation]

Original: As mentioned, this is roughly equivalent to the InfoNCE loss (Van den Oord et al., 2018), further validating our analysis so far.
Revised: As mentioned, this is roughly equivalent to the InfoNCE loss (Van den Oord et al., 2018), further validating our analysis so far.

Original: B. EBM L OSSES
Revised: B. EBM L OSSES

Original: To optimize E ρ µ ( s,a ) [ E p + π
Revised: To optimize E ρ µ ( s,a ) [ E p + π

Original: ( s + | s,a ) [ Q θ ( s, a, s + )] − log E ρ
Revised: ( s + | s,a ) [ Q θ ( s, a, s + )] − log E ρ

Original: + ( g ) [exp Q θ ( s, a, g )]] from an arbitrary dataset of behaviors ρ µ ( s, a, s + ) , we can easily see that the problem lies in accessing p
Revised: + ( g ) [exp Q θ ( s, a, g )]] from an arbitrary dataset of behaviors ρ µ ( s, a, s + ) , we can easily see that the problem lies in accessing p

Original: + π
Revised: + π

Original: ( s + | s, a ) : we do not have complete access to the distribution of "positive samples", as sampling directly from p
Revised: ( s + | s, a ) : we do not have complete access to the distribution of "positive samples", as sampling directly from p

Original: + π
Revised: + π

Original: ( s + | s, a ) requires on-policy rollouts.
Revised: ( s + | s, a ) requires on-policy rollouts.

Original: But this can be resolved by using importance weights and
Revised: But this can be resolved by using importance weights and

Original: Equation equation 4 to rewrite E p + π ( s + | s,a )
Revised: Equation equation 4 to rewrite E p + π ( s + | s,a )

Original: [ Q θ ( s, a, s + )] :
Revised: [ Q θ ( s, a, s + )] :

Original: [Equation]
Revised: [Equation]

Original: Above, we have introduced a new notation ˆ π ( a | s ) to address the following issue with p + π
Revised: Above, we have introduced a new notation ˆ π ( a | s ) to address the following issue with p + π

Original: ( s + | s, a ) : while π is a goal-conditioned policy, p + π ( s + | s, a ) is not conditioned on a goal apriori .
Revised: ( s + | s, a ) : while π is a goal-conditioned policy, p + π ( s + | s, a ) is not conditioned on a goal apriori .

Original: A similar problem was encountered (but ignored) in C-Learning (Eysenbach et al., 2020b), which simply assumed that the multi-goal policy was aposteriori conditioned on the same future state that got sampled from this policy in the first place (a contradiction).
Revised: A similar problem was encountered (but ignored) in C-Learning (Eysenbach et al., 2020b), which simply assumed that the multi-goal policy was aposteriori conditioned on the same future state that got sampled from this policy in the first place (a contradiction).

Original: To avoid this problem, we define ˆ π ( a | s ) =
Revised: To avoid this problem, we define ˆ π ( a | s ) =

Original: E p
Revised: E p

Original: + ( g ) [ π ( a | s, g )] and assume that p + π
Revised: + ( g ) [ π ( a | s, g )] and assume that p + π

Original: ( s + | s, a ) is sampled under ˆ π ( a | s ) beyond a single step.
Revised: ( s + | s, a ) is sampled under ˆ π ( a | s ) beyond a single step.

Original: We package all the EBM training losses into the following:
Revised: We package all the EBM training losses into the following:

Original: [Equation]
Revised: [Equation]

Original: To decompose the importance weight, note that the policy is trying to maximize the Q -values under the entropy constraint in equation 14, resulting in a Boltzmann policy (Haarnoja et al., 2017; Schulman et al., 2017; Haarnoja et al., 2018): arg 
Revised: To decompose the importance weight, note that the policy is trying to maximize the Q -values under the entropy constraint in equation 14, resulting in a Boltzmann policy (Haarnoja et al., 2017; Schulman et al., 2017; Haarnoja et al., 2018): arg [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( a | s
Revised: | s

Original: [Equation]
Revised: [Equation]

Original:  ) [ Q ( s, a, g )]−
Revised: g ) [ Q [Equation] s, a, g )]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: exp Q ( s, a, g ) / (cid:80) a exp Q ( s, a, g ) .
Revised: exp Q ( s, a, g ) / (cid:80) a exp Q ( s, a, g ) .

Original: Minimizing equation 53 is equivalent to minimizing the following loss, with ⊥ ( · ) being the stop-gradient sign:
Revised: Minimizing equation 53 is equivalent to minimizing the following loss, with ⊥ ( · ) being the stop-gradient sign:

Original: [Equation]
Revised: [Equation]

Original: Proof of equation 54.
Revised: Proof of equation 54.

Original: Let w ebm ( s ′ , a ′ , g ) denote the importance weight:
Revised: Let w ebm ( s ′ , a ′ , g ) denote the importance weight:

Original: [Equation]
Revised: [Equation]

Original: Firstly
Revised: Firstly

Original: [Equation]
Revised: [Equation]

Original: because of the entropy constraint in equation
Revised: because of the entropy constraint in equation

Original: [Equation]
Revised: [Equation]

Original: we have a Boltzmann policy defined on the
Revised: we have a Boltzmann policy defined on the

Original: Q -function: arg max 
Revised: Q -function: arg max π

Original: [Equation]
Revised: [Equation]

Original: Secondly
Revised: Secondly

Original: [Equation]
Revised: [Equation]

Original: we note that the first term in importance ratio can be estimated from equation 21.
Revised: we note that the first term in importance ratio can be estimated from equation 21.

Original: Thus we have:
Revised: Thus we have:

Original: [Equation]
Revised: [Equation]

Original: Taking the gradient of equation equation 53 inside the expectation of E ρ µ ( s,a ) p ( s ′ | s,a ) [ · ] : 
Revised: Taking the gradient of equation equation 53 inside the expectation of E ρ µ ( s,a ) p ( s ′ | s,a ) [ · ] : E

Original: [Equation]
Revised: [Equation]

Original: Dividing both the nominator and denominator of the right-hand side by 1 / |A|
Revised: Dividing both the nominator and denominator of the right-hand side by 1 / |A|

Original: [Equation]
Revised: [Equation]

Original: putting the expectation E ρ µ ( s,a ) p ( s ′ | s,a ) [ · ] back in
Revised: putting the expectation E ρ µ ( s,a ) p ( s ′ | s,a ) [ · ] back in

Original: [Equation]
Revised: [Equation]

Original: and utilizing the stop-gradient sign ⊥ ( · )
Revised: and utilizing the stop-gradient sign ⊥ ( · )

Original: [Equation]
Revised: [Equation]

Original: we arrive at the loss:
Revised: we arrive at the loss:

Revised: [Equation]

Revised: s,

Original: [Equation]
Revised: [Equation]

Original: B. D ERIVING HER R EWARDS
Revised: B. D ERIVING HER R EWARDS

Original: The purpose of this section is to derive the reward function used in HER from the equation 26:
Revised: The purpose of this section is to derive the reward function used in HER from the equation 26:

Original: [Equation]
Revised: [Equation]

Original: Recall that we have defined p + µ ( s + | s, a ) in equation 4:
Revised: Recall that we have defined p + µ ( s + | s, a ) in equation 4:

Original: [Equation]
Revised: [Equation]

Original: And that we have defined a quadratic form of f ∗ in equation 19 (with r and c being constants):
Revised: And that we have defined a quadratic form of f ∗ in equation 19 (with r and c being constants):

Original: [Equation]
Revised: [Equation]

Original: Using the dynamics to expand the expectation and applying the choice of f ∗ being a quadratic
Revised: Using the dynamics to expand the expectation and applying the choice of f ∗ being a quadratic

Original: [Equation]
Revised: [Equation]

Original: the loss becomes:
Revised: the loss becomes:

Original: [Equation]
Revised: [Equation]

Original: Assuming that there is a stop gradient sign on P π Q because of the use of a target network , we can rewrite the above as one single quadratic and see that the gradient of the above loss w.r.t Q is equivalent to the gradient of the following squared Bellman residual:
Revised: Assuming that there is a stop gradient sign on P π Q because of the use of a target network , we can rewrite the above as one single quadratic and see that the gradient of the above loss w.r.t Q is equivalent to the gradient of the following squared Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where the reward function r ( s, a, s ′ , s + ) is:
Revised: where the reward function r ( s, a, s ′ , s + ) is:

Original: [Equation]
Revised: [Equation]

Original: C A BLATIONS AND H YPER - PARAMETERS
Revised: C A BLATIONS AND H YPER - PARAMETERS

Original: In this section, we include additional ablations and hyper-parameters.
Revised: In this section, we include additional ablations and hyper-parameters.

Original: We first describe the goalreaching environments in more details in Figure 9.
Revised: We first describe the goalreaching environments in more details in Figure 9.

Original: We use the following thresholds (for Euclidean norms) for determining success: [0 . 08 , 0 .
Revised: We use the following thresholds (for Euclidean norms) for determining success: [0 . 08 , 0 .

Original: 1] , which are tight thresholds based on our visualizations of the environments.
Revised: 1] , which are tight thresholds based on our visualizations of the environments.

Original: We use the same network architecture, sampling and optimization schedules for all the methods, as described in Table 2.
Revised: We use the same network architecture, sampling and optimization schedules for all the methods, as described in Table 2.

Original: As for γ HDM , we set it to be 0 . 85 in Four
Revised: As for γ HDM , we set it to be 0 . 85 in Four

Original: Rooms and Lunar Lander , 0 . 5 in Sawyer Push and Claw Manipulate , and 0 .
Revised: Rooms and Lunar Lander , 0 . 5 in Sawyer Push and Claw Manipulate , and 0 .

Original: 4 for Door Opening .
Revised: 4 for Door Opening .

Original: Ablation on this hyper-parameter can be found in Figure 10.
Revised: Ablation on this hyper-parameter can be found in Figure 10.

Original: For next state relabeling ratio, we set the default to be 0 . 2 , increase it to 0 .
Revised: For next state relabeling ratio, we set the default to be 0 . 2 , increase it to 0 .

Original: 5 in Lunar Lander , and 0 .
Revised: 5 in Lunar Lander , and 0 .

Original: 6 in Sawyer Push and Door Opening .
Revised: 6 in Sawyer Push and Door Opening .

Original: For the soft- Q -learning (Schulman et al., 2017)
Revised: For the soft- Q -learning (Schulman et al., 2017)

Original: + HER baseline, we set the temperature parameter to be 0 . 2 , which we have found to empirically perform well.
Revised: + HER baseline, we set the temperature parameter to be 0 . 2 , which we have found to empirically perform well.

Original:  a goal location in Lunar Lander , pushing a puck to a goal location in Sawyer Push , opening the door to a goal angle in Door Open (Nair et al., 2018b), turning a valve to a goal orientation in Claw Manipulate (Ahn et al., 2020).
Revised: at a goal location in Lunar Lander , pushing a puck to a goal location in Sawyer Push , opening the door to a goal angle in Door Open (Nair et al., 2018b), turning a valve to a goal orientation in Claw Manipulate (Ahn et al., 2020).

Original: [Table]
Revised: [Table]