Original: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression
Revised: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression

Original: Abstract
Revised: Abstract

Original: Offline goal-conditioned reinforcement learning (GCRL) promises general-purposeskill learning in the form of reaching diverse goals from purely offline datasets.
Revised: Offline goal-conditioned reinforcement learning (GCRL) promises general-purposeskill learning in the form of reaching diverse goals from purely offline datasets.

Original: Wepropose Go al-conditioned f - A dvantage R egression (GoFAR), a novel regressionbased offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a stateoccupancy matching problem between a dynamics-abiding imitator agent and anexpert agent that directly teleports to the goal.
Revised: Wepropose Go al-conditioned f - A dvantage R egression (GoFAR), a novel regressionbased offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a stateoccupancy matching problem between a dynamics-abiding imitator agent and anexpert agent that directly teleports to the goal.

Original: In contrast to prior approaches,
Revised: In contrast to prior approaches,

Original: GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks.
Revised: GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks.

Original: These distinct features confer GoFARwith much better offline performance and stability as well as statistical performanceguarantee that is unattainable for prior methods.
Revised: These distinct features confer GoFARwith much better offline performance and stability as well as statistical performanceguarantee that is unattainable for prior methods.

Original: Furthermore, we demonstratethat GoFAR’s training objectives can be re-purposed to learn an agent-independentgoal-conditioned planner from purely offline source-domain data, which enableszero-shot transfer to new target domains.
Revised: Furthermore, we demonstratethat GoFAR’s training objectives can be re-purposed to learn an agent-independentgoal-conditioned planner from purely offline source-domain data, which enableszero-shot transfer to new target domains.

Original: Through extensive experiments, wevalidate GoFAR’s effectiveness in various problem settings and tasks, significantlyoutperforming prior state-of-art.
Revised: Through extensive experiments, wevalidate GoFAR’s effectiveness in various problem settings and tasks, significantlyoutperforming prior state-of-art.

Original: Notably, on a real robotic dexterous manipulationtask, while no other method makes meaningful progress, GoFAR acquires complexmanipulation behavior that successfully accomplishes diverse goals.
Revised: Notably, on a real robotic dexterous manipulationtask, while no other method makes meaningful progress, GoFAR acquires complexmanipulation behavior that successfully accomplishes diverse goals.

Original:  Introduction
Revised:  Introduction

Original: Goal-conditioned reinforcement learning [18, 44, 40] (GCRL) aims to learn a repertoire of skillsin the form of reaching distinct goals.
Revised: Goal-conditioned reinforcement learning [18, 43, 39] (GCRL) aims to learn a repertoire of skillsin the form of reaching distinct goals.

Original: Offline GCRL [6, 48] is particularly promising because itenables learning general goal-reaching policies from purely offline interaction datasets without anyenvironment interaction [28, 25], which can be expensive in the real-world.
Revised: Offline GCRL [6, 47] is particularly promising because itenables learning general goal-reaching policies from purely offline interaction datasets without anyenvironment interaction [28, 25], which can be expensive in the real-world.

Original: As offline datasetscontain diverse goals and become increasingly prevalent [9, 19, 6], policies learned this way canacquire a large set of useful primitives for downstream tasks [30].
Revised: As offline datasetscontain diverse goals and become increasingly prevalent [9, 19, 6], policies learned this way canacquire a large set of useful primitives for downstream tasks [30].

Original: A central challenge in GCRL is thesparsity of reward signal [3]; without any additional knowledge about the environment, an agent at astate typically only accrues positive binary reward when the state lies within the goal neighborhood.
Revised: A central challenge in GCRL is thesparsity of reward signal [3]; without any additional knowledge about the environment, an agent at astate typically only accrues positive binary reward when the state lies within the goal neighborhood.

Original: This sparse reward problem is exacerbated in the offline setting, in which the agent cannot explorethe environment to discover more informative states about desired goals.
Revised: This sparse reward problem is exacerbated in the offline setting, in which the agent cannot explorethe environment to discover more informative states about desired goals.

Original: Therefore, designing aneffective offline GCRL algorithm is a concrete yet challenging path towards general-purpose andscalable policy learning.
Revised: Therefore, designing aneffective offline GCRL algorithm is a concrete yet challenging path towards general-purpose andscalable policy learning.

Original: In this paper, we present a novel offline GCRL algorithm,
Revised: In this paper, we present a novel offline GCRL algorithm,

Original: Go al-conditioned f - A dvantage R egression(GoFAR), first casting GCRL as a state-occupancy matching [27, 32] problem and then deriving aregression-based policy objective.
Revised: Go al-conditioned f - A dvantage R egression(GoFAR), first casting GCRL as a state-occupancy matching [27, 32] problem and then deriving aregression-based policy objective.

Original: In particular, GoFAR begins with the following goal-conditionedstate-matching objective:
Revised: In particular, GoFAR begins with the following goal-conditionedstate-matching objective:

Original: [Equation]
Revised: [Equation]

Original: where d π ( s ; g ) is the goal-conditioned state-occupancy distribution of policy π and p ( s ; g ) is thedistribution of states that satisfy a particular goal g .
Revised: where d π ( s ; g ) is the goal-conditioned state-occupancy distribution of policy π and p ( s ; g ) is thedistribution of states that satisfy a particular goal g .

Original: This objective casts goal-conditioned offline RL as an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expert whocan teleport to the goal in one step; see Figure 2.
Revised: This objective casts goal-conditioned offline RL as an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expertwho can teleport to the goal in one step; see Figure 2.

Original: Posing GCRL this way is mathematically principled,as we show that this objective is equivalent to a probabilistic interpretation of GCRL that additionallyencourages maximizing state entropy.
Revised: Posing GCRL this way is mathematicallyprincipled, as we show that this objective is equivalent to a probabilistic interpretation of GCRLthat additionally encourages maximizing state entropy.

Original: More importantly, this objective admits elegant optimizationusing purely offline data by considering an f -divergence regularized objective and exploiting ideasfrom convex duality theory [42, 4].
Revised: More importantly, this objectives enables usto extend a state-of-art offline imitation learning algorithm [32] to the goal-conditioned setting andadmits elegant optimization using purely offline data by considering an f -divergence regularizedobjective and exploiting ideas from convex duality theory [41, 4].

Original: In particular, we obtain the dual optimal value function V ∗ from asingle unconstrained optimization problem, using which we construct the optimal advantage function(we refer to this as f-advantage ) that serves as importance weighting for a regression-based policytraining objective; see Figure 1 for a schematic illustration.
Revised: In particular, we obtain thedual optimal value function V ∗ from a single unconstrained optimization problem, using which weconstruct the optimal advantage function (we refer to this as f-advantage ) that serves as importanceweighting for a regression-based policy training objective; see Figure 1 for a schematic illustration.

Original: There are several distinct features to our approach.
Revised: There are several distinct features to our approach.

Original: First and foremost is GoFAR’s lackof goal relabeling.
Revised: First and foremost is GoFAR’s lackof goal relabeling.

Original: Hindsight goal relabeling [3](known as HER in the literature) relabels trajectory goals to be states that were actuallyachieved instead of the originally commandedgoals.
Revised: Hindsight goal relabeling [3](known as HER in the literature) relabels trajectory goals to be states that were actuallyachieved instead of the originally commandedgoals.

Original: This heuristic is critical for alleviating thesparse-reward problem in prior GCRL methods,but is unnecessary for GoFAR.
Revised: This heuristic is critical for alleviating thesparse-reward problem in prior GCRL methods,but is unnecessary for GoFAR.

Original: GoFAR updates its policy as if all data is already coming from theoptimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling.
Revised: GoFAR updates its policy as if all data is already coming from theoptimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling.

Original: GoFAR’s relabeling-free training is of significant practical benefits.
Revised: GoFAR’s relabeling-free training is of significant practical benefits.

Original: First, it enables more stable andsimpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot beeasily performed offline [49].
Revised: First, it enables more stable andsimpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot beeasily performed offline [48].

Original: Second, hindsight relabeling suffers from hindsight bias in stochasticenvironments [45], as achieved goals may be accomplished due to noise; this is exacerbated in theoffline setting due to inability to collect more data.
Revised: Second, hindsight relabeling suffers from hindsight bias in stochasticenvironments [44], as achieved goals may be accomplished due to noise; this is exacerbated in theoffline setting due to inability to collect more data.

Original: By bypassing hindsight relabeling, GoFARpromises to be more robust and scalable.
Revised: By bypassing hindsight relabeling, GoFARpromises to be more robust and scalable.

Original: Second, as suggested in Figure 1, GoFAR’s value and policy training steps are completely uninterleaved : we do not need to update the policy until the value function has converged.
Revised: Second, as suggested in Figure 1, GoFAR’s value and policy training steps are completely uninterleaved : we do not need to update the policy until the value function has converged.

Original: This confersgreater offline training stability [23, 32] compared to prior works, which mostly involve alternatingupdates to a critic Q-network and policy network.
Revised: This confersgreater offline training stability [23, 32] compared to prior works, which mostly involve alternatingupdates to a critic Q-network and policy network.

Original: Furthermore, it enables an algorithmic reductionof GoFAR to weighted regression [7], which allows us to obtain strong finite-sample statisticalguarantee on GoFAR’s performance; prior regression-based GCRL approaches [14, 48] do not enjoythis reduction and obtain much weaker theoretical guarantees.
Revised: Furthermore, it enables an algorithmic reductionof GoFAR to weighted regression [7], which allows us to obtain strong finite-sample statisticalguarantee on GoFAR’s performance; prior regression-based GCRL approaches [14, 47] do not enjoythis reduction and obtain much weaker theoretical guarantees.

Original: Finally, we show that GoFAR can also be used to learn a goal-conditioned planner [36, 5].
Revised: Finally, we show that GoFAR can also be used to learn a goal-conditioned planner [36, 5].

Original: Thekey insight lies in observing that, under a mild assumption, GoFAR’s dual value function objectivedoes not depend on the action and can learn from state-only offline data.
Revised: Thekey insight lies in observing that, under a mild assumption, GoFAR’s dual value function objectivedoes not depend on the action and can learn from state-only offline data.

Original: This enables learning agoal-centric value function and thereby a near-optimal goal-conditioned planner that is capable ofzero-shot transferring to new domains of the same task that shares the goal space.
Revised: This enables learning agoal-centric value function and thereby a near-optimal goal-conditioned planner that is capable ofzero-shot transferring to new domains of the same task that shares the goal space.

Original: In our experiments,we illustrate that GoFAR planner can indeed plan effective subgoals in a new target domain andenable a low-level controller to reach distant goals that it is not designed for.
Revised: In our experiments,we illustrate that GoFAR planner can indeed plan effective subgoals in a new target domain andenable a low-level controller to reach distant goals that it is not designed for.

Original: We extensively evaluate GoFAR on a variety of offline GCRL environments of varying task complexityand dataset composition, and show that it outperforms all baselines in all settings.
Revised: We extensively evaluate GoFAR on a variety of offline GCRL environments of varying task complexityand dataset composition, and show that it outperforms all baselines in all settings.

Original: Notably, GoFARis more robust to stochastic environments than methods that depend on hindsight relabeling.
Revised: Notably, GoFARis more robust to stochastic environments than methods that depend on hindsight relabeling.

Original: Weadditionally demonstrate that GoFAR learns complex manipulation behavior on a real-world highdimensional dexterous manipulation task [2], for which most baselines fail to make any progress.
Revised: Weadditionally demonstrate that GoFAR learns complex manipulation behavior on a real-world highdimensional dexterous manipulation task [2], for which most baselines fail to make any progress.

Original: Finally, we showcase GoFAR’s planning capability in a cross-robot zero-shot transfer task.
Revised: Finally, we showcase GoFAR’s planning capability in a cross-robot zero-shot transfer task.

Original: Tosummarize, our contributions are:
Revised: Tosummarize, our contributions are:

Original: 1. GoFAR, a novel offline GCRL algorithm derived from goal-conditioned state-occupancymatching
Revised: 1. GoFAR, a novel offline GCRL algorithm derived from goal-conditioned state-matching

Original: Detailed technical derivation and analysis of GoFAR’s distinctfeatures
Revised: Detailed technical derivation and analysis of GoFAR’s distinct

Original: (1) relabeling 
Revised: features: relabeling free,

Original: (2) uninterleaved optimization
Revised: ) uninterleaved optimization,

Revised: and

Original: (3) planning
Revised: (3) planning

Original: capability
Revised: capability

Original: Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities
Revised: Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities

Original:  Related Work
Revised:  Related Work

Original: Offline Goal-Conditioned Reinforcement Learning.
Revised: Offline Goal-Conditioned Reinforcement Learning.

Original: One core challenge of (offline) GCRL [18, 44]is the sparse-reward nature of goal-reaching tasks.
Revised: One core challenge of (offline) GCRL [18, 43]is the sparse-reward nature of goal-reaching tasks.

Original: A popular strategy is hindsight goal relabeling [3](or HER), which relabels off-policy transitions with future goals they have achieved instead of theoriginal commanded goals.
Revised: A popular strategy is hindsight goal relabeling [3](or HER), which relabels off-policy transitions with future goals they have achieved instead of theoriginal commanded goals.

Original: Existing offline GCRL algorithms [6, 48] adapt HER-based online GCRLalgorithms to the offline setting by incorporating additional components conducive to offline training.[6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [24] as wellas goal-chaining, which expands the pool of candidate relabeling goals to the entire offline dataset.
Revised: Existing offline GCRL algorithms [6, 47] adapt HER-based online GCRLalgorithms to the offline setting by incorporating additional components conducive to offline training.[6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [24] as wellas goal-chaining, which expands the pool of candidate relabeling goals to the entire offline dataset.

Original: Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shownto be a simple and effective method [14].
Revised: Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shownto be a simple and effective method [14].

Original: [48] improves upon [14] by incorporating discount-factorand advantage function weighting [38, 39], and shows improved performance in offline GCRL.
Revised: [47] improves upon [14] by incorporating discount-factorand advantage function weighting [37, 38], and shows improved performance in offline GCRL.

Original: Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting an onlinealgorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matchingand derives an elegant algorithm from the dual formulation that is naturally suited for the offlinesetting and carries many distinct properties that make it more stable, scalable, and versatile.
Revised: Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting an onlinealgorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matchingand derives an elegant algorithm from the dual formulation that is naturally suited for the offlinesetting and carries many distinct properties that make it more stable, scalable, and versatile.

Original: State-Occupancy Matching.
Revised: State-Occupancy Matching.

Original: State occupancy matching objectives have been shown to be effectivefor learning from observations [32, 50], exploration [27], as well as matching a hand-designedtarget state distribution [13]; occupancy matching, in general, has been explored in the imitationlearning literature [17, 46, 13, 20].
Revised: State occupancy matching objectives have been shown to be effectivefor learning from observations [32, 49], exploration [27], as well as matching a hand-designed targetstate distribution [13]; occupancy matching, in general, has been explored in the imitation learningliterature [17, 45, 13, 20].

Original: At a technical level, our strategy of estimating the state-occupancyratio is related tothe DICE technique [33, 35], which has recently been extended to the offlinesetting [26, 21, 32].
Revised: Our strategy of estimating the state-occupancy ratio is related to DICE [Figure] techniques for offline policy optimization [33, 35, 26, 21, 32].

Original: The closest work to ours is [32], which proposes a f -divergence based stateoccupancy matchingobjective for offline imitation learning.
Revised: The closest work to ours is [32], whichproposed f -divergence state-occupancy matching for offline imitation learning.

Original: Our work differs in that we consider agoal-conditioned setting and do not require a separate dataset of demonstrations for policy learning.
Revised: Our work showsthat such a state-occupancy matching approach can be applied to GCRL. Algorithmically, GoFARdiffers from [32] in that it does not require a separate dataset of expert demonstrations; it achievesthis by modifying the discriminator, and in some settings can even be used without a discriminator,whereas a discriminator is always required by [32].

Original: Furthermore, we derive several new results (Section 4.2-4.4) that are of particular interest in the GCRL setting.
Revised: Furthermore, we derive several new algorithmicproperties (Section 4.2-4.4) that are novel and particularly advantageous for offline GCRL.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Goal-Conditioned Reinforcement Learning.
Revised: Goal-Conditioned Reinforcement Learning.

Original: We consider an infinite-horizon Markov decisionprocess (MDP)
Revised: We consider an infinite-horizon Markov decisionprocess (MDP)

Original: [41] M = ( S, A, R, T, µ 0 , γ ) with state space S , action space A , deterministicrewards r ( s, a ) , stochastic transitions s ′ ∼ T ( s, a ) , initial state distribution µ 0 ( s ) , and discountfactor γ ∈ (0 , 1] .
Revised: [40] M = ( S, A, R, T, µ 0 , γ ) with state space S , action space A , deterministicrewards r ( s, a ) , stochastic transitions s ′ ∼ T ( s, a ) , initial state distribution µ 0 ( s ) , and discountfactor γ ∈ (0 , 1] .

Original: A policy π : S → ∆( A ) outputs a distribution over actions to use in a given state.
Revised: A policy π : S → ∆( A ) outputs a distribution over actions to use in a given state.

Original: In goal-conditioned RL, the MDP additionally assumes a goal space G :=
Revised: In goal-conditioned RL, the MDP additionally assumes a goal space G :=

Original: { ϕ ( s ) | s ∈ S } , where thestate-to-goal mapping ϕ : S → G is known.
Revised: { ϕ ( s ) | s ∈ S } , where thestate-to-goal mapping ϕ : S → G is known.

Original: Now, the reward function r ( s ; g ) 1 as well as the policyπ ( a | s, g ) depend on the commanded goal g ∈ G .
Revised: Now, the reward function r ( s ; g ) 1 as well as the policyπ ( a | s, g ) depend on the commanded goal g ∈ G .

Original: Given a distribution over desired goals p ( g ) , theobjective of goal-conditioned RL is to find a policy π that maximizes the discounted cumulativereturn:
Revised: Given a distribution over desired goals p ( g ) , theobjective of goal-conditioned RL is to find a policy π that maximizes the discounted return: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The goal-conditioned state-action occupancy distribution d π ( s, a ; g ) : S ×
Revised: The goal-conditioned state-action occupancy distribution d π ( s, a ; g ) : S ×

Original: A × G → [0 , 1] of π isd π ( s
Revised: A × G → [0 , 1] of π is [Equation] g

Original: [Equation]
Revised: [Equation]

Original: which captures the relative frequency of state-action visitations for a policy π conditioned on goal g .
Revised: which captures the relative frequency of state-action visitations for a policy π conditioned on goal g .

Original: The state-occupancy distribution then marginalizes over actions: d π ( s ; g )
Revised: The state-occupancy distribution then marginalizes over actions: d π ( s ; g )

Original: = (cid:80) a d π ( s, a ; g ) .
Revised: = (cid:80) a d π ( s, a ; g ) .

Original: Then, itfollows that π ( a | s, g ) =
Revised: Then, itfollows that π ( a | s, g ) =

Original: d π ( s,a ; g ) d
Revised: d π ( s,a ; g ) d

Original: π ( s ; g ) .
Revised: π ( s ; g ) .

Original: A state-action occupancy distribution must satisfy the Bellmanflow constraint in order for it to be an occupancy distribution for some stationary policy π :
Revised: A state-action occupancy distribution must satisfy the Bellmanflow constraint in order for it to be an occupancy distribution for some stationary policy π :

Original: [Equation]
Revised: [Equation]

Original: We write d π ( s, g )
Revised: We write d π ( s, g )

Original: =
Revised: =

Original: p ( g ) d π ( s ; g ) as the joint goal-state density induced by p ( g ) and the policy π .
Revised: p ( g ) d π ( s ; g ) as the joint goal-state density induced by p ( g ) and the policy π .

Original: Finally, given d π , we can express the objective function (2) as J ( π ) =
Revised: Finally, given d π , we can express the objective function (2) as J ( π ) =

Original: 11 − γ E
Revised: 11 − γ E

Original: ( s,g )
Revised: ( s,g )

Original: ∼ d π ( s,g ) [ r ( s ; g )] .
Revised: ∼ d π ( s,g ) [ r ( s ; g )] .

Original: Offline GCRL.
Revised: Offline GCRL.

Original: In offline GCRL, the agent cannot interact with the environment M ; instead,it is equipped with a static dataset of logged transitions D := { τ i } Ni =1 , where each trajectoryτ ( i ) =
Revised: In offline GCRL, the agent cannot interact with the environment M ; instead,it is equipped with a static dataset of logged transitions D := { τ i } Ni =1 , where each trajectoryτ ( i ) =

Original: ( s ( i ) 0 , a ( i ) 0 , r
Revised: ( s ( i ) 0 , a ( i ) 0 , r

Original: ( i ) 0 , s ( i ) 1 , ... ; g ( i ) ) with s ( i ) 

Revised: ( i ) 0 , s ( i ) 1 , ... ; g ( i ) ) with s ( i )

Original: ∼ µ 0 and g ( i ) is the commanded goal of the trajectory.
Revised: ∼ µ 0 and g ( i ) is the commanded goal of the trajectory.

Original: Note that trajectories need not to be generated from a goal-directed agent, in which case g ( i ) can berandomly drawn from p ( g ) .
Revised: Note that trajectories need not to be generated from a goal-directed agent, in which case g ( i ) can berandomly drawn from p ( g ) .

Original: We denote the empirical goal-conditioned state-action occupancies of
Revised: We denote the empirical goal-conditioned state-action occupancies of

Original: D O as d O ( s, a ; g ) .
Revised: D O as d O ( s, a ; g ) .

Original:  Goal-Conditioned f -Advantage Regression
Revised:  Goal-Conditioned f -Advantage Regression

Original: In this section, we introduce Goal-conditioned f -Advantage Regression (GoFAR).
Revised: In this section, we introduce Goal-conditioned f -Advantage Regression (GoFAR).

Original: We first derive thealgorithm in full (Section 4.1), then delve deep into its several appealing properties (Section 4.2-4.4).
Revised: We first derive thealgorithm in full (Section 4.1), then delve deep into its several appealing properties (Section 4.2-4.4).

Original: 4.1 Algorithm
Revised: 4.1 Algorithm

Original: We first show that goal-conditioned state-occupancy matching is a mathematically principled approachfor solving general GCRL problems, formalizing the teleportation intuition in the Introduction.
Revised: We first show that goal-conditioned state-occupancy matching is a mathematically principled approachfor solving general GCRL problems, formalizing the teleportation intuition in the introduction.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Given anychoice of r ( s ; g ) , for each g in the support of p ( g ) , define p ( s ; g ) =
Revised: Given any r ( s ; g ) , for each g in the support of p ( g ) , define p ( s ; g ) =

Original: e r ( s ; g ) Z ( g ) ,where Z ( g ) :
Revised: e r ( s ; g ) Z ( g ) , where Z ( g ) :

Original: = (cid:82) e r ( s ; g ) ds is the normalizing constant.
Revised: = (cid:82) e r ( s ; g ) ds is the normalizing constant.

Original: Then, the following equality holds:
Revised: Then, the following equality holds:

Original: [Equation]
Revised: [Equation]

Original: where J ( π ) is the GCRL objective (Eq.
Revised: where J ( π ) is the GCRL objective (Eq.

Original: (2) ) with reward r ( s ; g ) and C :
Revised: (2) ) with reward r ( s ; g ) and C :

Original: =
Revised: =

Original: E g ∼ p
Revised: E g ∼ p

Original: ( g )
Revised: ( g )

Original: [log Z ( g )] .
Revised: [log Z ( g )] .

Original: See Appendix E.1 for proof.
Revised: See Appendix E.1 for proof.

Original: This proposition states that, for any choice of reward r ( s ; g ) , solving the
Revised: This proposition states that, for any choice of reward r ( s ; g ) , solving the

Original: GCRL problem with a maximum state-entropy regularization is equivalent to optimizing for the goalconditioned state-occupancy matching objective with target distribution p ( s ; g ) :=
Revised: GCRL problem with a maximum state-entropy regularization is equivalent to optimizing for the goalconditioned state-occupancy matching objective with target distribution p ( s ; g ) :=

Original: e r ( s ; g ) Z ( g ) .
Revised: e r ( s ; g ) Z ( g ) .

Original: Now, thekey challenge with optimizing this objective offline is that we cannot sample from d π ( s ; g ) .
Revised: Now, thekey challenge with optimizing this objective offline is that we cannot sample from d π ( s ; g ) .

Original: To addressthis issue, following [32], we first derive an offline lower bound involving an f -divergence (see
Revised: To addressthis issue, following [32], we first derive an offline lower bound involving an f -divergence (see

Original: Appendix A for definition) regularization term, which subsequently enables solving this optimizationproblem via its dual using purely offline data:
Revised: Appendix A for definition) regularization term, which subsequently enables solving this optimizationproblem via its dual using purely offline data:

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Assume 2 for all g in support of p ( g ) , ∀ s, d O ( s ; g ) > 0 if p ( s ; g ) > 0 .
Revised: Assume 2 for all g in support of p ( g ) , ∀ s, d O ( s ; g ) > 0 if p ( s ; g ) > 0 .

Original: Then, for anyf -divergence that upper bounds the KL-divergence,
Revised: Then, for anyf -divergence that upper bounds the KL-divergence,

Original: [Equation]
Revised: [Equation]

Original: The RHS of (6) can be understood as an f -divergence regularized GCRL objective with rewardfunction r ( s ; g )
Revised: The RHS of (6) can be understood as an f -divergence regularized GCRL objective with rewardfunction r ( s ; g )

Original: = log p ( s ; g )
Revised: = log p ( s ; g )

Original: d O ( s ; g ) .
Revised: d O ( s ; g ) .

Original: Intuitively, this reward encourages visiting states that occur more oftenin the “expert” state distribution p ( s ; g ) than in the offline dataset, and the f -divergence regularizationthen ensures that the learned policy is supported by the offline dataset.
Revised: Intuitively, this reward encourages visiting states that occur more oftenin the “expert” state distribution p ( s ; g ) than in the offline dataset, and the f -divergence regularizationthen ensures that the learned policy is supported by the offline dataset.

Original: This choice of reward functioncan be estimated in practice by training a discriminator [15] c : S × G → (0 , 1) using the offlinedata:
Revised: This choice of reward functioncan be estimated in practice by training a discriminator [15] c : S × G → (0 , 1) using the offlinedata:

Original: [Equation]
Revised: [Equation]

Original: We can in fact obtain a looser lower bound that does not require training a discriminator:
Revised: We can in fact obtain a looser lower bound that does not require training a discriminator:

Original: [Equation]
Revised: [Equation]

Original: Because p ( s ; g ) ∝ e r ( s ; g ) , we may substitute r ( s ; g ) (when the offline dataset contains reward labels)for log p ( s ; g ) and bypass having to train a discriminator for reward;
Revised: Because p ( s ; g ) ∝ e r ( s ; g ) , we may substitute r ( s ; g ) (when the offline dataset contains reward labels)for log p ( s ; g ) and bypass having to train a discriminator for reward;

Original: Now, for either choice of lower bound, we may pose the optimization problem with respect to validchoices of state-action occupancies directly, introducing the Bellman flow constraint (4):
Revised: Now, for either choice of lower bound, we may pose the optimization problem with respect to validchoices of state-action occupancies directly, introducing the Bellman flow constraint (4):

Original: [Equation]
Revised: [Equation]

Original: This reformulation does not solve the fundamental problem that (9) still requires sampling fromd ( s ; g ) ; however, it has now written the problem in a way amenable to simplification using toolsfrom convex analysis.
Revised: This reformulation does not solve the fundamental problem that (9) still requires sampling fromd ( s ; g ) ; however, it has now written the problem in a way amenable to simplification using toolsfrom convex analysis.

Original: Now, we show that its dual problem can be reduced to an unconstrainedminimization problem over the dual variables which serve the role of a value function; importantly,the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution:
Revised: Now, we show that its dual problem can be reduced to an unconstrainedminimization problem over the dual variables which serve the role of a value function; importantly,the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution:

Original: Proposition 4.3.
Revised: Proposition 4.3.

Original: The dual problem to (9) is
Revised: The dual problem to (9) is

Original: [Equation]
Revised: [Equation]

Original: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and
Revised: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and

Original: T V ( s, a ; g ) =
Revised: T V ( s, a ; g ) =

Original: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .
Revised: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .

Original: Given the optimal V ∗ , the primal optimal d ∗ satisfies:
Revised: Given the optimal V ∗ , the primal optimal d ∗ satisfies:

Original: [Equation]
Revised: [Equation]

Original: A proof is given in Appendix B. Crucially, as neither expectation in (10) depends on samples fromd , this objective can be estimated entirely using offline data, making it suitable for offline GCRL.
Revised: A proof is given in Appendix B. Crucially, as neither expectation in (10) depends on samples fromd , this objective can be estimated entirely using offline data, making it suitable for offline GCRL.

Original: For tabular MDPs, we show that for a suitable choice of f -divergence, the optimal V ∗ in fact admitsclosed-form solutions; see Appendix D for details.
Revised: For tabular MDPs, we show that for a suitable choice of f -divergence, the optimal V ∗ in fact admitsclosed-form solutions; see Appendix D for details.

Original: In the continuous control setting, we can optimize(10) using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.
Revised: In the continuous control setting, we can optimize(10) using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.

Original: Then, once we have obtained the optimal (resp.
Revised: Then, once we have obtained the optimal (resp.

Original: converged) V ∗ , we propose learning the policy viathe following supervised regression update:
Revised: converged) V ∗ , we propose learning the policy viathe following supervised regression update:

Original: [Equation]
Revised: [Equation]

Original: We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluatedat the dual optimal advantage , r ( s ; g ) + γ T V ∗ ( s, a ; g ) − V ∗ ( s ; g ) ; we refer to this weighting termas f -advantage .
Revised: We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluatedat the dual optimal advantage , r ( s ; g ) + γ T V ∗ ( s, a ; g ) − V ∗ ( s ; g ) ; we refer to this weighting termas f -advantage .

Original: Hence, we name our overall method Goal-conditioned f -Advantage Regression(GoFAR) ; an abbreviated high-level pseudocode is provided in Algorithm 1 and a detailed versionis provided in Appendix C.
Revised: Hence, we name our overall method Goal-conditioned f -Advantage Regression(GoFAR) ; an abbreviated high-level pseudocode is provided in Algorithm 1 and a detailed versionis provided in Appendix C.

Original: In practice, we implement GoFAR with χ 2 -divergence, a choice off -divergence that is stable for off-policy optimization [50, 32, 26]; see Appendix C for details.
Revised: In practice, we implement GoFAR with χ 2 -divergence, a choice off -divergence that is stable for off-policy optimization [49, 32, 26]; see Appendix C for details.

Original: 1: (Optionally) Train discriminator-based reward (7) 2: Train (optimal) dual value function V ∗ ( s ; g ) (10) 3: Train policy π via f -Advantage Regression (12)
Revised: 1: (Optionally) Train discriminator-based reward (7) 2: Train (optimal) dual value function V ∗ ( s ; g ) (10) 3: Train policy π via f -Advantage Regression (12)

Original: Note that (12) forgos directly minimizing (1) offline, which has been foundto suffer from training instability [21].
Revised: Note that (12) forgos directly minimizing (1) offline, which has been foundto suffer from training instability [21].

Original: Instead, it naturally incorporates theprimal-dual optimal solutions in a regression loss.
Revised: Instead, it naturally incorporates theprimal-dual optimal solutions in a regression loss.

Original: Now, we will show thatthis policy objective has several theoretical and practical benefits for offline GCRL that make it particularly appealing.
Revised: Now, we will show thatthis policy objective has several theoretical and practical benefits for offline GCRL that make it particularly appealing.

Original: 4.2 Optimal Goal-Weighting Property
Revised: 4.2 Optimal Goal-Weighting Property

Original: We show that optimizing (12) automatically obtains the optimal goal-weighting distribution .
Revised: We show that optimizing (12) automatically obtains the optimal goal-weighting distribution .

Original: That is,
Revised: That is,

Original: GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (9).
Revised: GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (9).

Original: In particular, this property is achieved without any explicit hindsight relabeling (see Appendix A fora technical definition), a mechanism that prior works heavily depend on.
Revised: In particular, this property is achieved without any explicit hindsight relabeling (see Appendix A fora technical definition), a mechanism that prior works heavily depend on.

Original: To this end, we first definep ( g | s, a ) to be the conditional distribution of goals in the offline dataset conditioned on state-actionpair ( s, a ) .
Revised: To this end, we first definep ( g | s, a ) to be the conditional distribution of goals in the offline dataset conditioned on state-actionpair ( s, a ) .

Original: Then, according to Bayes rule, we have that
Revised: Then, according to Bayes rule, we have that

Original: [Equation]
Revised: [Equation]

Original: Using this equality, we can rewrite the policy objective (12) as follows:
Revised: Using this equality, we can rewrite the policy objective (12) as follows:

Original: [Equation]
Revised: [Equation]

Original: 15) where
Revised: 15) where

Original: [Equation]
Revised: [Equation]

Original: Thus, we see that GoFAR’s f -advantage weighting scheme is equivalent to performing supervisedpolicy regression where goals are sampled from ˜ p
Revised: Thus, we see that GoFAR’s f -advantage weighting scheme is equivalent to performing supervisedpolicy regression where goals are sampled from ˜ p

Original: ( g | s, a ) .
Revised: ( g | s, a ) .

Original: Now, combining (16) and Bayes rulegives
Revised: Now, combining (16) and Bayes rulegives

Original: [Equation]
Revised: [Equation]

Original: Thus, we can replace the nested expectations in (15) and obtain that GoFAR policy update amountsto supervised regression of the state-action occupancy distribution of the optimal policy to theregularized GCRL problem (9):
Revised: Thus, we can replace the nested expectations in (15) and obtain that GoFAR policy update amountsto supervised regression of the state-action occupancy distribution of the optimal policy to theregularized GCRL problem (9):

Original: [Equation]
Revised: [Equation]

Original: This derivation makes clear GoFAR’s connection with the hindsight goal relabeling mechanism [3]that is ubiquitous in GCRL:
Revised: This derivation makes clear GoFAR’s connection with the hindsight goal relabeling mechanism [3]that is ubiquitous in GCRL:

Original: GoFAR automatically performs the optimal goal-weighting policy updatewithout any explicit goal relabeling.
Revised: GoFAR automatically performs the optimal goal-weighting policy updatewithout any explicit goal relabeling.

Original: Furthermore, our derivation also suggests why hindsight relabeling is sub-optimal without further assumption on the reward function [10]: it heuristically chooses˜ p ( g | s, a ) to be the empirical trajectory-wise future achieved goal distribution (i.e., HER), whichgenerally does not coincide with the goals that the optimal policy would reach; see Appendix E.

Revised: Furthermore, our derivation also suggests why hindsight relabeling is sub-optimal without further assumption on the reward function [10]: it heuristically chooses˜ p ( g | s, a ) to be the empirical trajectory-wise future achieved goal distribution (i.e., HER), whichgenerally does not coincide with the goals that the optimal policy would reach; see Appendix E.

Original: forfurther discussion.
Revised: forfurther discussion.

Original: 4.3 Uninterleaved Optimization and Performance Guarantee
Revised: 4.3 Uninterleaved Optimization and Performance Guarantee

Original: An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the valuenetwork and the policy network.
Revised: An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the valuenetwork and the policy network.

Original: This can be observed by noting that GoFAR’s advantage term (11)is computed using V ∗ , the optimal solution of the dual problem.
Revised: This can be observed by noting that GoFAR’s advantage term (11)is computed using V ∗ , the optimal solution of the dual problem.

Original: This has the practical significanceof disentangling the value-function update (10) from the policy update (12), as we do not need tooptimize the latter until the former has converged.
Revised: This has the practical significanceof disentangling the value-function update (10) from the policy update (12), as we do not need tooptimize the latter until the former has converged.

Original: This disentanglement is in sharp contrast to prior
Revised: This disentanglement is in sharp contrast to prior

Original: GCRL works [48, 6, 11, 3], which typically involve alternating updates to the critic Q-network andthe policy network, a training procedure that has found to be unstable in the offline setting [23].
Revised: GCRL works [47, 6, 11, 3], which typically involve alternating updates to the critic Q-network andthe policy network, a training procedure that has found to be unstable in the offline setting [23].

Original: Thisis unavoidable for prior works because their advantage functions are estimated using the Q-estimateof the current policy, whereas our advantage term naturally falls out from primal-dual optimality.
Revised: Thisis unavoidable for prior works because their advantage functions are estimated using the Q-estimateof the current policy, whereas our advantage term naturally falls out from primal-dual optimality.

Original: The uninterleaved and relabeling-free nature of GoFAR also allows us to derive strong performanceguarantees.
Revised: The uninterleaved and relabeling-free nature of GoFAR also allows us to derive strong performanceguarantees.

Original: Because V ∗ is fixed in (12), this policy objective amounts to a weighted supervisedlearning problem.
Revised: Because V ∗ is fixed in (12), this policy objective amounts to a weighted supervisedlearning problem.

Original: Therefore, we can extend and adapt mature theoretical results for analyzing
Revised: Therefore, we can extend and adapt mature theoretical results for analyzing

Original: Behavior Cloning [1, 43, 47] as well as finite sample error guarantees for weighted regression [7] toobtain statistical guarantees on GoFAR’s performance with respect to the optimal π ∗ for (9):
Revised: Behavior Cloning [1, 42, 46] as well as finite sample error guarantees for weighted regression [7] toobtain statistical guarantees on GoFAR’s performance with respect to the optimal π ∗ for (9):

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Assume sup s,a,g d ∗ ( s,a ; g )
Revised: Assume sup s,a,g d ∗ ( s,a ; g )

Original: d O ( s,a ; g ) ≤ M and sup | r ( s, g )
Revised: d O ( s,a ; g ) ≤ M and sup | r ( s, g )

Original: | ≤ R max .
Revised: | ≤ R max .

Original: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .
Revised: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .

Original: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,

Original: GoFAR (18) will return a policy ˆ π such that:
Revised: GoFAR (18) will return a policy ˆ π such that:

Original: [Equation]
Revised: [Equation]

Original: Notably, the error shrinks as the size of the offline data N increases, requiring no dependency onaccess to data from the “expert” distribution d ∗ .
Revised: Notably, the error shrinks as the size of the offline data N increases, requiring no dependency onaccess to data from the “expert” distribution d ∗ .

Original: This provides a theoretical basis for GoFAR’sempirical scalability as we are guaranteed to obtain good results when the offline data becomes moreexpansive.
Revised: This provides a theoretical basis for GoFAR’sempirical scalability as we are guaranteed to obtain good results when the offline data becomes moreexpansive.

Original: In contrast, prior regression-based GCRL methods cannot be easily reduced to a simpleweighted regression with respect to the desired goal distribution p ( g ) , so they only obtain weakerresults under stronger assumptions on the offline data (e.g, full state-space coverage) as well as thepolicy; see Appendix E.3 for discussion.
Revised: In contrast, prior regression-based GCRL methods cannot be easily reduced to a simpleweighted regression with respect to the desired goal distribution p ( g ) , so they only obtain weakerresults under stronger assumptions on the offline data (e.g, full state-space coverage) as well as thepolicy; see Appendix E.3 for discussion.

Original: 4.4 Goal-Conditioned Planning
Revised: 4.4 Goal-Conditioned Planning

Original: Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zeroshot transfer to other domains of the same task.
Revised: Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zeroshot transfer to other domains of the same task.

Original: The key insight is that GoFAR’s value functionobjective (10) does not depend on actions assuming deterministic transitions.
Revised: The key insight is that GoFAR’s value functionobjective (10) does not depend on actions assuming deterministic transitions.

Original: This is because givena transition ( s, a, s ′ , g ) ∼ d
Revised: This is because givena transition ( s, a, s ′ , g ) ∼ d

Original: O , T V ( s, a ; g ) =
Revised: O , T V ( s, a ; g ) =

Original: V ( s ′ ; g ) , so we can rewrite the second term in (10)as E ( s,a,s ′ ,g )
Revised: V ( s ′ ; g ) , so we can rewrite the second term in (10)as E ( s,a,s ′ ,g )

Original: ∼ d
Revised: ∼ d

Original: O
Revised: O

Original: [ f ⋆ ( r ( s, g ) + γV ( s ′ ; g ) − V
Revised: [ f ⋆ ( r ( s, g ) + γV ( s ′ ; g ) − V

Original: s ; g ))] , which does not depend on the action.
Revised: s ; g ))] , which does not depend on the action.

Original: This property enables us to learn a goal-centric value function that is independent of the agent’s statespace because the agent’s actions are not relevant.
Revised: This property enables us to learn a goal-centric value function that is independent of the agent’s statespace because the agent’s actions are not relevant.

Original: Specifically, we propose the following objective:
Revised: Specifically, we propose the following objective:

Original: [Equation]
Revised: [Equation]

Original: We can think of this objective as learning a value function with the inductive bias that the firstoperation transforms the state input to the goal space via ϕ .
Revised: We can think of this objective as learning a value function with the inductive bias that the firstoperation transforms the state input to the goal space via ϕ .

Original: Since V is now independent of the agent,we can use f -advantage regression to instead learn a goal-conditioned planner :
Revised: Since V is now independent of the agent,we can use f -advantage regression to instead learn a goal-conditioned planner :

Original: [Equation]
Revised: [Equation]

Original: where π now outputs the next subgoal ϕ ( s ′ ) conditioned on the current achieved goal ϕ ( s ) and thedesired goal g .
Revised: where π now outputs the next subgoal ϕ ( s ′ ) conditioned on the current achieved goal ϕ ( s ) and thedesired goal g .

Original: In our experiments, we show how this planner can achieve hierarchical control throughzero-shot transfer subgoal plans to a new target domain.
Revised: In our experiments, we show how this planner can achieve hierarchical control throughzero-shot transfer subgoal plans to a new target domain.

Original:  Experiments
Revised:  Experiments

Original: We pose the following questions and provide affirmative answers in our experiments:
Revised: We pose the following questions and provide affirmative answers in our experiments:

Original: Is GoFAR effective for offline GCRL?
Revised: Is GoFAR effective for offline GCRL?

Original: What components are important for performance?2.
Revised: What components are important for performance?2.

Original: Is GoFAR more robust to stochastic environments than hindsight relabeling methods?3.
Revised: Is GoFAR more robust to stochastic environments than hindsight relabeling methods?3.

Original: Can GoFAR be applied to a real-robotics system?4.
Revised: Can GoFAR be applied to a real-robotics system?4.

Original: Can GoFAR learn a goal-conditioned planner for zero-shot cross-embodiment transfer?5.

Revised: Can GoFAR learn a goal-conditioned planner for zero-shot cross-embodiment transfer?5.

Original: Offline GCRL
Revised: Offline GCRL

Original: Tasks.
Revised: Tasks.

Original: In our simulation experiments, we consider six distinct environments.
Revised: In our simulation experiments, we consider six distinct environments.

Original: They include fourrobot manipulation environments[40]: FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide,and two dexterous manipulation environments: HandReach [40], and D’ClawTurn [2].
Revised: They include fourrobot manipulation environments[39]: FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide,and two dexterous manipulation environments: HandReach [39], and D’ClawTurn [2].

Original: All tasks usesparse reward, and their respective goal distributions are defined over valid configurations in eitherthe robot space or object space, depending on whether the task involves object manipulation.
Revised: All tasks usesparse reward, and their respective goal distributions are defined over valid configurations in eitherthe robot space or object space, depending on whether the task involves object manipulation.

Original: Theoffline dataset for each task is collected by either a random policy or a mixture of 90% random policyand 10% expert policy [32, 21], depending on whether random data provides enough coverage of thedesired goal distribution.
Revised: Theoffline dataset for each task is collected by either a random policy or a mixture of 90% random policyand 10% expert policy [32, 21], depending on whether random data provides enough coverage of thedesired goal distribution.

Original: The first five tasks and their datasets are taken from [48].
Revised: The first five tasks and their datasets are taken from [47].

Original: See Appendix G.3for dataset details and Appendix F for detailed task descriptions and figures.
Revised: See Appendix G.3for dataset details and Appendix F for detailed task descriptions and figures.

Original: [Table]
Revised: [Table]

Original: Algorithms.
Revised: Algorithms.

Original: We compare tostate-of-art offline GCRL algorithms, consisting of bothregression-based and actor-criticmethods.
Revised: We compare tostate-of-art offline GCRL algorithms, consisting of bothregression-based and actor-criticmethods.

Original: The regression-basedmethods are: (1) GCSL [14],which incorporates hindsight relabeling in conjunction with behavior cloning to clone actionsthat lead to a specified goal, and (2) WGCSL [48], which improves upon GCSL by incorporatingdiscount factor and advantage weighting into the supervised policy learning update.
Revised: The regression-basedmethods are: (1) GCSL [14],which incorporates hindsight relabeling in conjunction with behavior cloning to clone actionsthat lead to a specified goal, and (2) WGCSL [47], which improves upon GCSL by incorporatingdiscount factor and advantage weighting into the supervised policy learning update.

Original: The actor-criticmethods are (1) DDPG [3], which adapts DDPG [29] to the goal-conditioned setting by incorporating hindsight relabeling, and
Revised: The actor-criticmethods are (1) DDPG [3], which adapts DDPG [29] to the goal-conditioned setting by incorporating hindsight relabeling, and

Original: (2) ActionableModel (AM) [6], which incorporates conservative
Revised: (2) ActionableModel (AM) [6], which incorporates conservative

Original: Q-Learning [24] as well as goal-chaining on top of an actor-critic method.
Revised: Q-Learning [24] as well as goal-chaining on top of an actor-critic method.

Original: We use tuned hyperparameters for each baseline on all tasks; in particular, we search the best HERratio from { 0 . 2 , 0 . 5 , 1 . 0 } for each baseline on each task separately and report the best-performingone.
Revised: We use tuned hyperparameters for each baseline on all tasks; in particular, we search the best HERratio from { 0 . 2 , 0 . 5 , 1 . 0 } for each baseline on each task separately and report the best-performingone.

Original: For GoFAR, we use identical hyperparameters as WGCSL for the shared network componentsand do not tune further.
Revised: For GoFAR, we use identical hyperparameters as WGCSL for the shared network componentsand do not tune further.

Original: We train each method for 10 seeds, and each training run uses 400k minibatchupdates of size 512.
Revised: We train each method for 10 seeds, and each training run uses 400k minibatchupdates of size 512.

Original: Complete architecture and hyperparameter table as well as additional trainingdetails are provided in Appendix G.
Revised: Complete architecture and hyperparameter table as well as additional trainingdetails are provided in Appendix G.

Original: Evaluations and Results.
Revised: Evaluations and Results.

Original: We report the discounted return using the sparse binary task reward.
Revised: We report the discounted return using the sparse binary task reward.

Original: Thismetric rewards algorithms that reach goals as fast as possible and stay in the goal region thereafter.
Revised: Thismetric rewards algorithms that reach goals as fast as possible and stay in the goal region thereafter.

Original: In
Revised: In

Original: Appendix H, we also report the final success rate using the same sparse binary criterion.
Revised: Appendix H, we also report the final success rate using the same sparse binary criterion.

Original: Because thetask reward is binary, these metrics do not take account into how precisely a goal is being reached.
Revised: Because thetask reward is binary, these metrics do not take account into how precisely a goal is being reached.

Original: Therefore, we additionally
Revised: Therefore, we additionally

Original: The full discounted return results are shown in Table 1; ( ⋆ ) indicates statistically significant improvement over the second best method under a two-sample t -test.
Revised: The full discounted return results are shown in Table 1; ( ⋆ ) indicates statistically significant improvement over the second best method under a two-sample t -test.

Original: The fraction inside () indicates thebest HER rate for each baseline.
Revised: The fraction inside () indicates thebest HER rate for each baseline.

Original: As shown, GoFAR attains the best overall performance across sixtasks, and the results are statistically significant on three tasks, including the two more challengingdexterous manipulation tasks.
Revised: As shown, GoFAR attains the best overall performance across sixtasks, and the results are statistically significant on three tasks, including the two more challengingdexterous manipulation tasks.

Original: In Table 6 in Appendix H, we find GoFAR is also superior on the finaldistance metric.
Revised: In Table 6 in Appendix H, we find GoFAR is also superior on the finaldistance metric.

Original: In other words, GoFAR reaches the goals fastest and most precisely.
Revised: In other words, GoFAR reaches the goals fastest and most precisely.

Original: Ablations.
Revised: Ablations.

Original: Recall that GoFAR achieves optimal goal-weighting, and does not require the HER heuristic that has been key to prior GCRL approaches.
Revised: Recall that GoFAR achieves optimal goal-weighting, and does not require the HER heuristic that has been key to prior GCRL approaches.

Original: We now experimentally confirm that the baselines arenot performant without HER.
Revised: We now experimentally confirm that the baselines arenot performant without HER.

Original: Conversely, we investigate whether HER can help GoFAR.
Revised: Conversely, we investigate whether HER can help GoFAR.

Original: Additionally,we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in (8).
Revised: Additionally,we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in (8).

Original: Finally, we replace χ 2 -divergencewith KL to understand whether thef -divergence lower bound in (6) isnecessary.
Revised: Finally, we replace χ 2 -divergencewith KL to understand whether thef -divergence lower bound in (6) isnecessary.

Original: The results are shown in
Revised: The results are shown in

Original: Figure 3 (task-specific breakdown isincluded in Appendix H).
Revised: Figure 3 (task-specific breakdown isincluded in Appendix H).

Original: Comparedto their original performances (colored dash lines), baselines without
Revised: Comparedto their original performances (colored dash lines), baselines without

Original: HER suffer severely, especially GCSLand WGCSL: both these methods infact require 100% relabeling and relysolely on HER for positive learningsignals.
Revised: HER suffer severely, especially GCSLand WGCSL: both these methods infact require 100% relabeling and relysolely on HER for positive learningsignals.

Original: In sharp contrast, GoFAR isalso a regression-based method, yet itperforms very well without HER and its performance is unaffected by adding HER.
Revised: In sharp contrast, GoFAR isalso a regression-based method, yet itperforms very well without HER and its performance is unaffected by adding HER.

Original: These findingsare consistent with the theoretical results of Section 4.2.
Revised: These findingsare consistent with the theoretical results of Section 4.2.

Original: GoFAR (Binary) performs slightly worsethan GoFAR, which is to be expected due to the looser bound; however, it is still better than allbaselines, which use the same binary reward and are aided by HER.
Revised: GoFAR (Binary) performs slightly worsethan GoFAR, which is to be expected due to the looser bound; however, it is still better than allbaselines, which use the same binary reward and are aided by HER.

Original: Again, this ablation highlightsthe merit of our optimization approach.
Revised: Again, this ablation highlightsthe merit of our optimization approach.

Original: Finally, GoFAR (KL) suffers from numerical instability andfails to learn, showing that our general f -divergence lower bound (6) is not only of theoretical valuebut also practically significant.
Revised: Finally, GoFAR (KL) suffers from numerical instability andfails to learn, showing that our general f -divergence lower bound (6) is not only of theoretical valuebut also practically significant.

Original: 5.2 Robustness in Stochastic Offline GCRL Settings
Revised: 5.2 Robustness in Stochastic Offline GCRL Settings

Original: For this experiment, we create noisy variants ofthe FetchReach environment by adding varyinglevels of white Gaussian noise to policy actionoutputs.
Revised: For this experiment, we create noisy variants ofthe FetchReach environment by adding varyinglevels of white Gaussian noise to policy actionoutputs.

Original: FetchReach is well-suited for this experiment because all baselines attain satisfactoryperformance and even outperform GoFAR in Table 1.
Revised: FetchReach is well-suited for this experiment because all baselines attain satisfactoryperformance and even outperform GoFAR in Table 1.

Original: Furthermore, to isolate the effect of noisyactions, we use GoFAR with binary reward sothat all methods now use the same reward input and perform comparatively without addednoise.
Revised: Furthermore, to isolate the effect of noisyactions, we use GoFAR with binary reward sothat all methods now use the same reward input and perform comparatively without addednoise.

Original: We consider zero-mean Gaussian noiseof σ value from 0 .
Revised: We consider zero-mean Gaussian noiseof σ value from 0 .

Original: 5 , 1 , and 1 . 5 .
Revised: 5 , 1 , and 1 . 5 .

Original: For each noise level, we re-collect offline data by executing randomactions in the respective noisy environment and train on these noisy offline random data.
Revised: For each noise level, we re-collect offline data by executing randomactions in the respective noisy environment and train on these noisy offline random data.

Original: The average discounted returns over differentnoise levels are illustrated in Figure 4.
Revised: The average discounted returns over differentnoise levels are illustrated in Figure 4.

Original: As shown,at noise level 0 .
Revised: As shown,at noise level 0 .

Original: 5 , while GoFAR’s performanceis not effected, all baselines already exhibitdegraded performance.
Revised: 5 , while GoFAR’s performanceis not effected, all baselines already exhibitdegraded performance.

Original: In particular, DDPGsharply degrades, underperforming GoFAR despite better original performance.
Revised: In particular, DDPGsharply degrades, underperforming GoFAR despite better original performance.

Original: At 1 . 0 , thegap continues to widen, and AM notably collapses, highlighting its sensitivity to noise.
Revised: At 1 . 0 , thegap continues to widen, and AM notably collapses, highlighting its sensitivity to noise.

Original: This is expected as AM’s relabeling mechanism alsosamples future goals from other trajectories and labels the selected goals with their current Q-valueestimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity,thus contributing to the instability observed in this experiment.
Revised: This is expected as AM’s relabeling mechanism alsosamples future goals from other trajectories and labels the selected goals with their current Q-valueestimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity,thus contributing to the instability observed in this experiment.

Original: It is only at noise level 1 . 5 that
Revised: It is only at noise level 1 . 5 that

Original: GoFAR’s performance degradation becomes comparable to those of other methods.
Revised: GoFAR’s performance degradation becomes comparable to those of other methods.

Original: Thus, GoFAR’s relabeling-free optimization indeed confers greater robustness to environmental stochasticity; incontrast, all baselines suffer from “false positive” relabeled goals due to such disturbances.
Revised: Thus, GoFAR’s relabeling-free optimization indeed confers greater robustness to environmental stochasticity; incontrast, all baselines suffer from “false positive” relabeled goals due to such disturbances.

Original: 5.3 Real-World Robotic Dexterous Manipulation
Revised: 5.3 Real-World Robotic Dexterous Manipulation

Original: Now, we evaluate on a real D’Claw [2] dexterous manipulation robot (Figure 5 (left)).
Revised: Now, we evaluate on a real D’Claw [2] dexterous manipulation robot (Figure 5 (left)).

Original: The task is tohave the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix Ffor detail.
Revised: The task is tohave the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix Ffor detail.

Original: The offline dataset contains 400K transitions, collected by executing purely random actionsin the environment, which provides sufficient coverage of the goal space.
Revised: The offline dataset contains 400K transitions, collected by executing purely random actionsin the environment, which provides sufficient coverage of the goal space.

Original: Compared to the simulated
Revised: Compared to the simulated

Original: D’Claw, an additional challenge is the inherent stochasticity on a real robot system due to imperfectactuation and hardware wear and tear.
Revised: D’Claw, an additional challenge is the inherent stochasticity on a real robot system due to imperfectactuation and hardware wear and tear.

Original: Given our conclusion in the stochastic environment experimentabove, we should expect GoFAR to perform better.
Revised: Given our conclusion in the stochastic environment experimentabove, we should expect GoFAR to perform better.

Original: The training curve of final distances averaged over 3 seedsis shown in Figure 6.
Revised: The training curve of final distances averaged over 3 seedsis shown in Figure 6.

Original: We see that GoFAR exhibits smoothprogress over training and significantly outperforms allbaselines.
Revised: We see that GoFAR exhibits smoothprogress over training and significantly outperforms allbaselines.

Original: At the end of training, GoFAR is able to rotatethe valve on average within 10 degrees of the specifiedgoal angle.
Revised: At the end of training, GoFAR is able to rotatethe valve on average within 10 degrees of the specifiedgoal angle.

Original: In contrast, most baselines do not make anyprogress on this difficult task and learns policies that do notmanage to turn the valve at all.
Revised: In contrast, most baselines do not make anyprogress on this difficult task and learns policies that do notmanage to turn the valve at all.

Original: AM is the only baseline thatlearns a non-trivial policy; however, its training fluctuatessignificantly, and the learned policy often executes extremeactions during evaluation that fail to reach the desired goal.
Revised: AM is the only baseline thatlearns a non-trivial policy; however, its training fluctuatessignificantly, and the learned policy often executes extremeactions during evaluation that fail to reach the desired goal.

Original: We provide qualitative analysis in Appendix H.3, and videos are included in the supplement.
Revised: We provide qualitative analysis in Appendix H.3, and videos are included in the supplement.

Original: 5.4 Zero-Shot Transfer Across Robots
Revised: 5.4 Zero-Shot Transfer Across Robots

Original: [Table]
Revised: [Table]

Original: For this experiment, we consider zero-shot transfer fromthe FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Pandarobot [12]; the environments are illustrated in Figure 5.
Revised: For this experiment, we consider zero-shot transfer fromthe FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Pandarobot [12]; the environments are illustrated in Figure 5.

Original: In the target domain, we assume a pre-programmed orlearned low-level goal-reaching controller π low ( a | s, g )that can push objects over short distances but fails for longer tasks.
Revised: In the target domain, we assume a pre-programmed orlearned low-level goal-reaching controller π low ( a | s, g )that can push objects over short distances but fails for longer tasks.

Original: For these distant goals, we trainthe goal-conditioned GoFAR value function V ∗ ( ϕ ( s ); g ) and planner π ( ϕ ( s ′ )
Revised: For these distant goals, we trainthe goal-conditioned GoFAR value function V ∗ ( ϕ ( s ); g ) and planner π ( ϕ ( s ′ )

Original: | ϕ ( s ) , g ) accordingto (20) and (21), to set goals for π low .
Revised: | ϕ ( s ) , g ) accordingto (20) and (21), to set goals for π low .

Original: We use the same offline FetchPush data as in Section 5.1.
Revised: We use the same offline FetchPush data as in Section 5.1.

Original: Wecompare two approaches: (1) naively using the Low-Level Controller , and (2) using the GoFARplanner to guide the low-level controller ( GoFAR Hierarchical Controller ).
Revised: Wecompare two approaches: (1) naively using the Low-Level Controller , and (2) using the GoFARplanner to guide the low-level controller ( GoFAR Hierarchical Controller ).

Original: See Appendix G.4for additional details.
Revised: See Appendix G.4for additional details.

Original: We report the success rate over 100 random distant test goals in Table 2.
Revised: We report the success rate over 100 random distant test goals in Table 2.

Original: Wesee that naively commanding the low-level controller with these distant goals indeed results in verylow success rate.
Revised: Wesee that naively commanding the low-level controller with these distant goals indeed results in verylow success rate.

Original: When the controller is augmented with GoFAR planner, the success rate nearlydoubles.
Revised: When the controller is augmented with GoFAR planner, the success rate nearlydoubles.

Original: Note that our planner itself is much superior: if all subgoals are perfectly reached, GoFAR
Revised: Note that our planner itself is much superior: if all subgoals are perfectly reached, GoFAR

Original: Planner (Oracle) achieves 84% success.
Revised: Planner (Oracle) achieves 84% success.

Original: See Appendix G.4 for additional experimental details.
Revised: See Appendix G.4 for additional experimental details.

Original: These experiments validate GoFAR’s ability to zero-shot transferring subgoal plans to enhance thecapabability of low-level controllers.
Revised: These experiments validate GoFAR’s ability to zero-shot transferring subgoal plans to enhance thecapabability of low-level controllers.

Original: Note that we have no access to any data from the target domain,and this planning capability naturally emerges from our training objectives and does not require anychange in the algorithm.
Revised: Note that we have no access to any data from the target domain,and this planning capability naturally emerges from our training objectives and does not require anychange in the algorithm.

Original: We show qualitative results in Appendix H.4 and videos in the supplement.
Revised: We show qualitative results in Appendix H.4 and videos in the supplement.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented GoFAR, a novel regression-based offline GCRL algorithm derived from a stateoccupancy matching perspective.
Revised: We have presented GoFAR, a novel regression-based offline GCRL algorithm derived from a stateoccupancy matching perspective.

Original: GoFAR is relabeling-free and enjoys uninterleaved training, makingit both theoretically and practically advantageous compared to prior state-of-art.
Revised: GoFAR is relabeling-free and enjoys uninterleaved training, makingit both theoretically and practically advantageous compared to prior state-of-art.

Original: Furthermore, GoFARsupports training a goal-conditioned planner with promising zero-shot transfer capability.
Revised: Furthermore, GoFARsupports training a goal-conditioned planner with promising zero-shot transfer capability.

Original: Throughextensive experiments, we have validated the practical utility of GoFAR in various challengingproblem settings, showing significant improvement over prior state-of-art.
Revised: Throughextensive experiments, we have validated the practical utility of GoFAR in various challengingproblem settings, showing significant improvement over prior state-of-art.

Original: We believe that GoFAR’sstrong theoretical foundation and empirical performance make it an ideal candidate for scalable skilllearning in the real world.
Revised: We believe that GoFAR’sstrong theoretical foundation and empirical performance make it an ideal candidate for scalable skilllearning in the real world.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We include the
Revised: [Yes] We include the

Original: code and data for our simulation experiments in the supplementary material.
Revised: code and data for our simulation experiments in the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Appendix G.
Revised: [Yes] See Appendix G.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See Appendix H.
Revised: [Yes] See Appendix H.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix G.
Revised: [Yes] See Appendix G.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: We include videos in our supplementary material.
Revised: We include videos in our supplementary material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Part I
Revised: Part I

Original: Appendix
Revised: Appendix

Original: These appendices contain material supplementary to submission 7863: "Offline Goal-Conditioned 583 Reinforcement Learning via f -Advantage Regression"
Revised: These appendices contain material supplementary to submission 7863: "Offline Goal-Conditioned 579 Reinforcement Learning via f -Advantage Regression"

Original: Table of Contents
Revised: Table of Contents

Original: [Table]

Original: A Additional Technical Background
Revised: A Additional Technical Background

Original: A
Revised: A.1 f

Original: [Table]
Revised: [Table]

Original: A. [Table]
Revised: Divergence and Fenchel Duality . . . . . . . . . . . . . . . . . . . . . . . . A.2 Hindsight Goal-Relabeling . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Proofs 

Revised: B Proofs

Original: B.1 Proof of Proposition 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Proof of Proposition 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Proof of Proposition 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: B.2 Proof of Proposition 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.

Revised: [Table]

Original: B.3 Proof of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C
Revised: [Table]

Original: GoFAR Technical Details
Revised: GoFAR Technical Details

Original: C.
Revised: C

Revised: .1 Discriminator Training

Original: C
Revised: C

Revised: .2 GoFAR with common f -Divergences

Revised: C

Revised: .3 Full Pseudocode

Original: D
Revised: D

Original: GoFAR for Tabular MDPs
Revised: GoFAR for Tabular MDPs

Original: E Additional Technical Discussion[-
-]
Revised: E Additional Technical Discussion[-
-]

Original: E
Revised: [Table]

Original: 1 Connecting Goal-Conditioned State-Matching and Probabilistic 
Revised: .1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL

Original: E.2 Optimality Conditions for Hindsight Relabeling
Revised: E.2 Optimality Conditions for Hindsight Relabeling

Original: E.3 Theoretical Comparison to
Revised: E.3 Theoretical Comparison to

Original: Prior Regression-based GCRL methods . . . . . . .
Revised: Prior Regression-based GCRL methods . . . . . . .

Original: F Task Descriptions 

Revised: F Task Descriptions

Original: F.1 Fetch Tasks . .
Revised: F.1 Fetch Tasks . .

Original: F.2 Hand Reach . .
Revised: F.2 Hand Reach . .

Original: F.3 D’ClawTurn (Simulation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: F.3 D’ClawTurn (Simulation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.4 D’ClawTurn (Real) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: F.4 D’ClawTurn (Real) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Experimental Details {+
+}
Revised: G Experimental Details {+
+}

Original: G.1 Baseline Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 Baseline Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Architectures and Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.2 Architectures and Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.3 Offline GCRL Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.3 Offline GCRL Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.4 Zero-Shot Transfer Experiments . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.4 Zero-Shot Transfer Experiments . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H Additional Results
Revised: H Additional Results

Original: H.1 Offline GCRL Full Results . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.1 Offline GCRL Full Results . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.2 Ablations . .
Revised: H.2 Ablations . .

Original: H.3 Real-World Dexterous Manipulations . .
Revised: H.3 Real-World Dexterous Manipulations . .

Original: [Table]
Revised: [Table]

Original: A Additional Technical Background
Revised: A Additional Technical Background

Original: We provide some technical definitions that are needed in our proofs in Appendix B and discussionsof hindsight relabeling in Section 4.1 of the main text as well as Appendix E.2.
Revised: We provide some technical definitions that are needed in our proofs in Appendix B and discussionsof hindsight relabeling in Section 4.1 of the main text as well as Appendix E.2.

Original: A.1 f -Divergence and Fenchel Duality
Revised: A.1 f -Divergence and Fenchel Duality

Original: These definitions are adapted from [32, 34].
Revised: These definitions are adapted from [32, 34].

Original: Definition A.1 ( f -divergence) .
Revised: Definition A.1 ( f -divergence) .

Original: For any continuous, convex function f and two probability distributions p, q ∈ ∆( X ) over a domain X , the f -divergence of p computed at q is defined as
Revised: For any continuous, convex function f and two probability distributions p, q ∈ ∆( X ) over a domain X , the f -divergence of p computed at q is defined as

Original: [Equation]
Revised: [Equation]

Original: Some common choices of f -divergence includes the KL-divergence and the χ 2 -divergence, whichcorresponds to choosing f ( x ) = x log x and f ( x ) = 12 ( x − 1) 2 , respectively.
Revised: Some common choices of f -divergence includes the KL-divergence and the χ 2 -divergence, whichcorresponds to choosing f ( x ) = x log x and f ( x ) = 12 ( x − 1) 2 , respectively.

Original: Definition A.2 (Fenchel conjugate) .
Revised: Definition A.2 (Fenchel conjugate) .

Original: Given a vector space X with inner-product ⟨· , ·⟩ , the Fenchelconjugate f ⋆ : X ⋆ → R of a convex and differentiable function f : Ω → R is
Revised: Given a vector space X with inner-product ⟨· , ·⟩ , the Fenchelconjugate f ⋆ : X ⋆ → R of a convex and differentiable function f : Ω → R is

Original: [Equation]
Revised: [Equation]

Original: 23) and any maximizer x ∗ of f ⋆ ( y ) satisfies x ∗ = f ′ ⋆ ( y ) .
Revised: 23) and any maximizer x ∗ of f ⋆ ( y ) satisfies x ∗ = f ′ ⋆ ( y ) .

Original: For an f -divergence, under mild realizability assumptions [8] on f , the Fenchel conjugate of D f ( p ∥ q )at y : X → R is
Revised: For an f -divergence, under mild realizability assumptions [8] on f , the Fenchel conjugate of D f ( p ∥ q )at y : X → R is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 25) and any maximizer p ∗ of D
Revised: 25) and any maximizer p ∗ of D

Original: ⋆,f ( y ) satisfies p ∗ ( x ) =
Revised: ⋆,f ( y ) satisfies p ∗ ( x ) =

Original: q ( x ) f ′ ⋆ ( y ( x )) .
Revised: q ( x ) f ′ ⋆ ( y ( x )) .

Original: These optimality conditions canbe seen as extensions of the KKT-condition.
Revised: These optimality conditions canbe seen as extensions of the KKT-condition.

Original: A.2 Hindsight Goal-Relabeling
Revised: A.2 Hindsight Goal-Relabeling

Original: We provide a mathematical formalism of hindsight goal relabeling [3].
Revised: We provide a mathematical formalism of hindsight goal relabeling [3].

Original: Definition A.3.
Revised: Definition A.3.

Original: Given a state s t from a trajectory τ = { s 0 , a 0 , r 0 , ..., s T ; g } , hindsight goalrelabeling is the goal-relabeling distribution
Revised: Given a state s t from a trajectory τ = { s 0 , a 0 , r 0 , ..., s T ; g } , hindsight goalrelabeling is the goal-relabeling distribution

Original: [Equation]
Revised: [Equation]

Original: where q is some categorical distribution taking values in { ϕ ( s t ) , ..., ϕ ( s T ) } .
Revised: where q is some categorical distribution taking values in { ϕ ( s t ) , ..., ϕ ( s T ) } .

Original: That is, the relabeled goal is selected from some distribution goals that are reached in the future inthe same trajectory.
Revised: That is, the relabeled goal is selected from some distribution goals that are reached in the future inthe same trajectory.

Original: The most canonical choice of q , known as hindsight experience replay (HER),selects q to be the uniform distribution.
Revised: The most canonical choice of q , known as hindsight experience replay (HER),selects q to be the uniform distribution.

Original: Once a goal ˜ g is chosen, the reward label is also re-computedusing the reward function assumed by the algorithm: r t := r ( s t , ˜ g ) .
Revised: Once a goal ˜ g is chosen, the reward label is also re-computedusing the reward function assumed by the algorithm: r t := r ( s t , ˜ g ) .

Original: B Proofs
Revised: B Proofs

Original: In this section, we restate propositions and theorems in the paper and present their proofs.
Revised: In this section, we restate propositions and theorems in the paper and present their proofs.

Original: B.1 Proof of Proposition 4.2 
Revised: B.1 Proof of Proposition 4.

Original: Proposition B.1.
Revised: Proposition B.1.

Original: Assume for all g in support of p ( g ) , ∀ s, d O ( s ; g ) >
Revised: Assume for all g in support of p ( g ) , ∀ s, d O ( s ; g ) >

Original: if p ( s ; g ) > 0 .
Revised: if p ( s ; g ) > 0 .

Original: Then, for anyf -divergence that upper bounds the KL-divergence,
Revised: Then, for anyf -divergence that upper bounds the KL-divergence,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first present and prove some technical lemma needed to prove this result.
Revised: We first present and prove some technical lemma needed to prove this result.

Original: The followinglemmas and proofs are adapted from [32]; in particular, we extend these known results to thegoal-conditioned setting.
Revised: The followinglemmas and proofs are adapted from [32]; in particular, we extend these known results to thegoal-conditioned setting.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: For any pair of valid occupancy distributions d 1 and d 2 , we have
Revised: For any pair of valid occupancy distributions d 1 and d 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This lemma hinges on proving the following lemma first
Revised: This lemma hinges on proving the following lemma first

Original: [Equation]
Revised: [Equation]

Original: Lemma B
Revised: Lemma B

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Using this result
Revised: Using this result

Original: [Equation]
Revised: [Equation]

Original: we can prove Lemma B
Revised: we can prove Lemma B

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: we assume that d O ( s ; g ) > 0 whenever p ( s ; g ) > 0 .
Revised: we assume that d O ( s ; g ) > 0 whenever p ( s ; g ) > 0 .

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from Lemma B.1.
Revised: where the last step follows from Lemma B.1.

Original: Then, for any D f ≥ D KL , we have that
Revised: Then, for any D f ≥ D KL , we have that

Original: [Equation]
Revised: [Equation]

Original: Then, since E ( s,g ) ∼ d π ( s,g ) (cid:104) log
Revised: Then, since E ( s,g ) ∼ d π ( s,g ) (cid:104) log

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 , we also obtain the following looser bound:
Revised: ≥ 0 , we also obtain the following looser bound:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: B.2 Proof of Proposition 4. 
Revised: B.2 Proof of Proposition 4.

Original: Proposition B.2.
Revised: Proposition B.2.

Original: The dual problem to (9) is
Revised: The dual problem to (9) is

Original: [Equation]
Revised: [Equation]

Original: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and
Revised: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and

Original: T V ( s, a ; g ) =
Revised: T V ( s, a ; g ) =

Original: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .
Revised: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .

Original: Given the optimal V ∗ , the primal optimal d ∗ satisfies:
Revised: Given the optimal V ∗ , the primal optimal d ∗ satisfies:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We begin by writing the Lagrangian dual of the primal problem:
Revised: We begin by writing the Lagrangian dual of the primal problem:

Original: [Equation]
Revised: [Equation]

Original: where p ( g ) V ( s ; g ) is the Lagrangian vector.
Revised: where p ( g ) V ( s ; g ) is the Lagrangian vector.

Original: Then, we note that
Revised: Then, we note that

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: Using this, we can rewrite (35) as
Revised: Using this, we can rewrite (35) as

Original: [Equation]
Revised: [Equation]

Original: And finally,
Revised: And finally,

Original: [Equation]
Revised: [Equation]

Original: Now, we make the key observation that the inner maximization problem in (38) is in fact the Fenchelconjugate of D f ( d ( s, a, g ) ∥
Revised: Now, we make the key observation that the inner maximization problem in (38) is in fact the Fenchelconjugate of D f ( d ( s, a, g ) ∥

Original: d O ( s, a, g )) at r ( s, g ) + γ T V ( s, a, g ) − V ( s, g ) .
Revised: d O ( s, a, g )) at r ( s, g ) + γ T V ( s, a, g ) − V ( s, g ) .

Original: Therefore, we canreduce (38) to an unconstrained minimization problem over the dual variables
Revised: Therefore, we canreduce (38) to an unconstrained minimization problem over the dual variables

Original: [Equation]
Revised: [Equation]

Original: and consequently, we can relate the dual-optimal V ∗ to the primal-optimal d ∗ using Fenchel duality(see Appendix A:
Revised: and consequently, we can relate the dual-optimal V ∗ to the primal-optimal d ∗ using Fenchel duality(see Appendix A:

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: B.3 Proof of Theorem
Revised: B.3 Proof of Theorem

Original: Theorem B.3.
Revised: Theorem B.3.

Original: Assume sup s,a,g d ∗ ( s,a ; g )
Revised: Assume sup s,a,g d ∗ ( s,a ; g )

Original: d O ( s,a ; g ) ≤ M and sup | r ( s, g )
Revised: d O ( s,a ; g ) ≤ M and sup | r ( s, g )

Original: | ≤ R max .
Revised: | ≤ R max .

Original: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .
Revised: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .

Original: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,

Original: GoFAR (18) will return a policy ˆ π such that:
Revised: GoFAR (18) will return a policy ˆ π such that:

Original: [Equation]
Revised: [Equation]

Original: N (41) where V π := E ( s,g ) ∼ ( µ 0 ,g )
Revised: N (41) where V π := E ( s,g ) ∼ ( µ 0 ,g )

Original: [ V ( s ; g )] .
Revised: [ V ( s ; g )] .

Original: Proof.
Revised: Proof.

Original: We begin by deriving an upper bound using the performance difference lemma [1]:
Revised: We begin by deriving an upper bound using the performance difference lemma [1]:

Original: [Equation]
Revised: [Equation]

Original: Then, using standard algebraic manipulations, we have:
Revised: Then, using standard algebraic manipulations, we have:

Original: [Equation]
Revised: [Equation]

Original: Then, since sup s,a,g d ∗ ( s,a ; g ) d O ( s,a ; g ) ≤ M , we can use Hoeffding’s inequality with weighted empiricalloss [7] to obtain that:
Revised: Then, since sup s,a,g d ∗ ( s,a ; g ) d O ( s,a ; g ) ≤ M , we can use Hoeffding’s inequality with weighted empiricalloss [7] to obtain that:

Original: [Equation]
Revised: [Equation]

Original: C GoFAR Technical Details
Revised: C GoFAR Technical Details

Original: In this section, we provide additional technical details of GoFAR that are omitted in the main text.
Revised: In this section, we provide additional technical details of GoFAR that are omitted in the main text.

Original: These include (1) detail of the GoFAR discriminator training, (2) mathematical expressions of GoFARspecialized to common f -Divergences, and (3) a full pseudocode.
Revised: These include (1) detail of the GoFAR discriminator training, (2) mathematical expressions of GoFARspecialized to common f -Divergences, and (3) a full pseudocode.

Original: C.1 Discriminator Training
Revised: C.1 Discriminator Training

Original: Training the discriminator 7 in practice requires choosing p ( s ; g ) .
Revised: Training the discriminator 7 in practice requires choosing p ( s ; g ) .

Original: For simplicity, we set p ( s ; g ) to bethe Dirac distribution centered at g : I ( ϕ ( s ) = g ) ; this precludes having to choose hyperparametersfor p ( s ; g ) .
Revised: For simplicity, we set p ( s ; g ) to bethe Dirac distribution centered at g : I ( ϕ ( s ) = g ) ; this precludes having to choose hyperparametersfor p ( s ; g ) .

Original: Once the discriminator has converged, we can retrieve the reward function r ( s ; g )
Revised: Once the discriminator has converged, we can retrieve the reward function r ( s ; g )

Original: = log p ( s ; g )
Revised: = log p ( s ; g )

Original: d O ( s ; g )
Revised: d O ( s ; g )

Original: =− log (cid:16) 1 c ∗ ( s ; g ) − 1 (cid:17) , since c ∗ ( s ; g ) =
Revised: =− log (cid:16) 1 c ∗ ( s ; g ) − 1 (cid:17) , since c ∗ ( s ; g ) =

Original: d O ( s ; g ) p ( s ; g )+
Revised: d O ( s ; g ) p ( s ; g )+

Original: d O ( s ; g ) .
Revised: d O ( s ; g ) .

Original: C.2 GoFAR with common f -Divergences
Revised: C.2 GoFAR with common f -Divergences

Original: GoFAR requires choosing a f -divergence.
Revised: GoFAR requires choosing a f -divergence.

Original: Here, we specialize GoFAR to χ 2 -divergence as well as
Revised: Here, we specialize GoFAR to χ 2 -divergence as well as

Original: KL-divergence as examples.
Revised: KL-divergence as examples.

Original: Our practical implementation uses χ 2 -divergence, which we found to besignificantly more stable than KL-divergence (see Section 5.1).
Revised: Our practical implementation uses χ 2 -divergence, which we found to besignificantly more stable than KL-divergence (see Section 5.1).

Original: Example 1 (GoFAR with χ 2 -divergence) .
Revised: Example 1 (GoFAR with χ 2 -divergence) .

Original: f ( x ) = 12 ( x − 1) 2 , and we can show that f ⋆ ( x ) =12 ( x + 1) 2 and f ′
Revised: f ( x ) = 12 ( x − 1) 2 , and we can show that f ⋆ ( x ) =12 ( x + 1) 2 and f ′

Original: ⋆ ( x ) = x
Revised: ⋆ ( x ) = x

Original: + 1 .
Revised: + 1 .

Original: Hence, the GoFAR objective amounts tomin
Revised: Hence, the GoFAR objective amounts tomin

Original: [Equation]
Revised: [Equation]

Original: (45) and
Revised: (45) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Example 2 (GoFAR with KL-divergence) .
Revised: Example 2 (GoFAR with KL-divergence) .

Original: We have f ( x ) = x log x and that D
Revised: We have f ( x ) = x log x and that D

Original: ⋆,f ( y ) =
Revised: ⋆,f ( y ) =

Original: log E
Revised: log E

Original: x ∼ q [exp y ( x )] [4].
Revised: x ∼ q [exp y ( x )] [4].

Original: Hence, the KL-divergence GoFAR objective ismin
Revised: Hence, the KL-divergence GoFAR objective ismin

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 47) and
Revised: 47) and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, we provide the full pseudocode for GoFAR implemented using χ 2 -divergence in Algorithm 2.
Revised: Now, we provide the full pseudocode for GoFAR implemented using χ 2 -divergence in Algorithm 2.

Original: C.3 Full Pseudocode
Revised: C.3 Full Pseudocode

Original: 1: Require : Offline dataset d O , choice of f -divergence f , choice of p ( s ; g ) 2: Randomly initialize discriminator c ψ , value function V θ , and policy π ϕ .
Revised: 1: Require : Offline dataset d O , choice of f -divergence f , choice of p ( s ; g ) 2: Randomly initialize discriminator c ψ , value function V θ , and policy π ϕ .

Original: Train Discriminator (Optional) 4: for number of discriminator iterations do 5:
Revised: Train Discriminator (Optional) 4: for number of discriminator iterations do 5:

Original: Sample minibatch { s id , g i } Ni =1 ∼ d O 6:
Revised: Sample minibatch { s id , g i } Ni =1 ∼ d O 6:

Original: Sample { s jg } Mj =1 ∼ p ( s ; g i ) ∀ i ∈ 1 . . .
Revised: Sample { s jg } Mj =1 ∼ p ( s ; g i ) ∀ i ∈ 1 . . .

Original: N 7:
Revised: N 7:

Original: Discriminator objective: L c ( ψ ) = 1 N (cid:80) Ni
Revised: Discriminator objective: L c ( ψ ) = 1 N (cid:80) Ni

Original: =1 [log(1 − c ψ ( s jd , g i ))
Revised: =1 [log(1 − c ψ ( s jd , g i ))

Original: + 1 M (cid:80) Mj
Revised: + 1 M (cid:80) Mj

Original: =1 [log c ψ ( s jg , g i )]] 8:
Revised: =1 [log c ψ ( s jg , g i )]] 8:

Original: Update c ψ using SGD: c ψ ← c ψ − α c ∇L c ( ψ ) 9: end for 10: //
Revised: Update c ψ using SGD: c ψ ← c ψ − α c ∇L c ( ψ ) 9: end for 10: //

Original: Train Dual Value Function 11: for number of value iterations do 12:
Revised: Train Dual Value Function 11: for number of value iterations do 12:

Original: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d O , { s i 0 }
Revised: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d O , { s i 0 }

Original: Mi =1 ∼ µ 0 , { g i 0 }
Revised: Mi =1 ∼ µ 0 , { g i 0 }

Original: Mi =1 ∼ d O 13:
Revised: Mi =1 ∼ d O 13:

Original: If discriminator, obtain reward: r ( s i t ; g i t )
Revised: If discriminator, obtain reward: r ( s i t ; g i t )

Original: = − log (cid:16) 1 c ψ ( s i t ; g i t ) − 1 (cid:17) ∀ i = 1 . . .
Revised: = − log (cid:16) 1 c ψ ( s i t ; g i t ) − 1 (cid:17) ∀ i = 1 . . .

Original: N 14:
Revised: N 14:

Original: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d O 15:
Revised: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d O 15:

Original: Value objective: L V ( θ ) = 1 − γM (cid:80) Mi
Revised: Value objective: L V ( θ ) = 1 − γM (cid:80) Mi

Original: =1 [ V θ ( s i 0 ; g i 0 )]
Revised: =1 [ V θ ( s i 0 ; g i 0 )]

Original: + 1 N (cid:80) Ni =1 (cid:2)
Revised: + 1 N (cid:80) Ni =1 (cid:2)

Original: f ⋆ ( r it + γV
Revised: f ⋆ ( r it + γV

Original: ( s it +1 ; g it ) − V
Revised: ( s it +1 ; g it ) − V

Original: ( s it ; g it )) (cid:3) 16:
Revised: ( s it ; g it )) (cid:3) 16:

Original: Update V θ using SGD: V θ ← V θ − α V ∇L V ( θ ) 17: end for 18: // Train Policy With f -Advantage Regression 19: for number of policy iterations do 20:
Revised: Update V θ using SGD: V θ ← V θ − α V ∇L V ( θ ) 17: end for 18: // Train Policy With f -Advantage Regression 19: for number of policy iterations do 20:

Original: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d
Revised: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d

Original: O 21:
Revised: O 21:

Original: If discriminator, obtain reward: r ( s it ; g it )
Revised: If discriminator, obtain reward: r ( s it ; g it )

Original: = − log (cid:16) 1 c ψ ( s it ; g it ) − 1 (cid:17) ∀ i = 1 . . .
Revised: = − log (cid:16) 1 c ψ ( s it ; g it ) − 1 (cid:17) ∀ i = 1 . . .

Original: N 22:
Revised: N 22:

Original: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d
Revised: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d

Original: O 23:
Revised: O 23:

Original: Policy objective: L π ( ϕ ) =
Revised: Policy objective: L π ( ϕ ) =

Original: (cid:80) Ni =1 (cid:2)(cid:0)
Revised: (cid:80) Ni =1 (cid:2)(cid:0)

Original: f ′ ⋆ (cid:0) r it + γV
Revised: f ′ ⋆ (cid:0) r it + γV

Original: θ ( s it +1 ; g it ) − V θ
Revised: θ ( s it +1 ; g it ) − V θ

Original: ( s it ; g it (cid:1)(cid:1) log π ( a | s, g ) (cid:3) 24:
Revised: ( s it ; g it (cid:1)(cid:1) log π ( a | s, g ) (cid:3) 24:

Original: Update π ϕ using SGD: π ϕ ← π ϕ − α π ∇L ( ϕ ) 25: end for
Revised: Update π ϕ using SGD: π ϕ ← π ϕ − α π ∇L ( ϕ ) 25: end for

Original: D GoFAR for Tabular MDPs
Revised: D GoFAR for Tabular MDPs

Original: In Section 4.1 of the main text, we have stated that in tabular MDPs, GoFAR’s optimal dual valuefunction (10) admits closed-form solution when we choose χ 2 -divergence.
Revised: In Section 4.1 of the main text, we have stated that in tabular MDPs, GoFAR’s optimal dual valuefunction (10) admits closed-form solution when we choose χ 2 -divergence.

Original: Here, we provide aderivation of this result.
Revised: Here, we provide aderivation of this result.

Original: Recall the dual problem (1)min
Revised: Recall the dual problem (1)min

Original: [Equation]
Revised: [Equation]

Original: To derive a closed-form solution, we rewrite the problem in vectorized notation; we borrow ournotations from [32].
Revised: To derive a closed-form solution, we rewrite the problem in vectorized notation; we borrow ournotations from [32].

Original: We first augment the state-space by concatenating the state dimensions andthe goal dimensions so that the new state space ˜ S has dimension S + G .
Revised: We first augment the state-space by concatenating the state dimensions andthe goal dimensions so that the new state space ˜ S has dimension S + G .

Original: Then, the new transitionfunction, with slight abuse of notation, T (( s ′ , g ′ )
Revised: Then, the new transitionfunction, with slight abuse of notation, T (( s ′ , g ′ )

Original: | ( s, g ) , a ) =
Revised: | ( s, g ) , a ) =

Original: T ( s | s, a ) I ( g ′ =
Revised: T ( s | s, a ) I ( g ′ =

Original: g ) ; the new initialstate distribution is thus µ 0 ( s, g )
Revised: g ) ; the new initialstate distribution is thus µ 0 ( s, g )

Original: = µ 0 ( s ) p ( g ) .
Revised: = µ 0 ( s ) p ( g ) .

Original: Therefore, ˜ T ∈ R | S
Revised: Therefore, ˜ T ∈ R | S

Original: ||
Revised: ||

Original: G ||
Revised: G ||

Original: A |×| S || G | + and µ 0 ∈ R | S || G | + .
Revised: A |×| S || G | + and µ 0 ∈ R | S || G | + .

Original: We assume that the offline dataset D O is collected by a behavior policy π b .
Revised: We assume that the offline dataset D O is collected by a behavior policy π b .

Original: We constructa surrogate MDP ˆ M using maximum likelihood estimation; that is, ˆ T (( s ′ , g ′ )
Revised: We constructa surrogate MDP ˆ M using maximum likelihood estimation; that is, ˆ T (( s ′ , g ′ )

Original: | ( s, g ) , a ) =
Revised: | ( s, g ) , a ) =

Original: n ( s,a,s ′ )
Revised: n ( s,a,s ′ )

Original: n ( s,a ) I
Revised: n ( s,a ) I

Original: ( g ′ = g ) , and we impute ˆ T (( s ′ , g ) | ( s, g ) , a ) = 1 S when n ( s, a ) = 0 .
Revised: ( g ′ = g ) , and we impute ˆ T (( s ′ , g ) | ( s, g ) , a ) = 1 S when n ( s, a ) = 0 .

Original: Then, usingˆ M , we can compute d O ∈ R | S
Revised: Then, usingˆ M , we can compute d O ∈ R | S

Original: ||
Revised: ||

Original: G ||
Revised: G ||

Original: A | + using linear programming and define reward r ∈ R | S || G | +as r ( s ; g )
Revised: A | + using linear programming and define reward r ∈ R | S || G | +as r ( s ; g )

Original: = log p ( s ; g )
Revised: = log p ( s ; g )

Original: d O ( s ; g ) , where p ( s ; g ) ∈ R | S || G | + .
Revised: d O ( s ; g ) , where p ( s ; g ) ∈ R | S || G | + .

Original: Now, define T ∈ R | S
Revised: Now, define T ∈ R | S

Original: ||
Revised: ||

Original: G ||
Revised: G ||

Original: A |×| S || G | such that( T V )( s, a ; g ) =
Revised: A |×| S || G | such that( T V )( s, a ; g ) =

Original: (cid:80) s ′ T (( s ′ , g )
Revised: (cid:80) s ′ T (( s ′ , g )

Original: | ( s, g ) , a ) V ( s ′ ; g ) , where V ∈ R | S || G | + is the dual optimizationvariables.
Revised: | ( s, g ) , a ) V ( s ′ ; g ) , where V ∈ R | S || G | + is the dual optimizationvariables.

Original: We also define B ∈ R | S
Revised: We also define B ∈ R | S

Original: ||
Revised: ||

Original: G ||
Revised: G ||

Original: A |×| S || G | such that ( B V )( s, a ; g ) =
Revised: A |×| S || G | such that ( B V )( s, a ; g ) =

Original: V ( s ; g ) .
Revised: V ( s ; g ) .

Original: Finally, we define
Revised: Finally, we define

Original: D =
Revised: D =

Original: [Equation]

Original: |

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: || A | .
Revised: | .

Original: Now, we can rewrite the dual problem as follow:
Revised: Now, we can rewrite the dual problem as follow:

Original: [Equation]
Revised: [Equation]

Original: Now, we recognize that (50) is equivalent to Equation 49 in [32]
Revised: Now, we recognize that (50) is equivalent to Equation 49 in [32]

Original: [Equation]
Revised: [Equation]

Original: as we have reduced goal-conditioned
Revised: as we have reduced goal-conditioned

Original: RL to regular RL with an augmented state-space.
Revised: RL to regular RL with an augmented state-space.

Original: Now, using the same derivation as in [32]
Revised: Now, using the same derivation as in [32]

Original: [Equation]
Revised: [Equation]

Original: we havethat
Revised: we havethat

Original: [Equation]
Revised: [Equation]

Original: and we can recover d ∗ ( s, a ; g ) :
Revised: and we can recover d ∗ ( s, a ; g ) :

Original: [Equation]
Revised: [Equation]

Original: Given d ∗ , we may extract the optimal policy π ∗ by marginalizing over actions:[Equation]
Revised: Given d ∗ [Equation] we may extract the optimal policy π ∗ by marginalizing over actions:

Original: [Equation]
Revised: [Equation]

Original: E Additional Technical Discussion
Revised: E Additional Technical Discussion

Original: E.1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL
Revised: E.1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL

Original: We begin by proving the following result:

Original: Proposition E.1.

Original: For any choice of p ( s ; g ) , define r ( s ; g )

Original: = log p ( s ; g ) .Then, the following equalityholds: [Equation] [Equation]
Revised: Suppose the GCRL problem comes with a reward function r ( s ; g ) .

Original: Proof.

Original: We have that

Original: [Equation]

Original: Rearranging the inequality gives the desired result.

Original: This proposition states that starting with a choice of p ( s ; g ) , minimizing the state-occupancy matchingobjective with respect to p ( s ; g ) is equivalent to solving a GCRL problem with reward r ( s ; g )

Original: =log p ( s ; g ) in addition to maximizing the policy’s state-entropy.

Original: Therefore, minimizing the goalconditioned state-occupancy matching objective is always solving some GCRL problem, for whichthe reward function can be interpreted as the log probability of the goal-satsifaction state distribution.

Original: Now, suppose the GCRL problem comes with a reward function r ( s ; g ) .

Original: We also show that there isan equivalent goal-conditioned state-occupancy matching problem with a target distribution p ( s ; g )implicitly defined via r ( s ; g ) .
Revised: We also show that there is anequivalent goal-conditioned state-occupancy matching problem with a target distribution p ( s ; g )defined based on r ( s ; g ) .

Original: Proposition E.2.
Revised: Proposition E.1.

Revised: (Proposition 4.1 in the paper)

Original: Given anychoice of r ( s ; g ) , for each g in the support of p ( g ) , define p ( s ; g ) =
Revised: Given any r ( s ; g ) , for each g in the support of p ( g ) ,define p ( s ; g ) =

Original: e r ( s ; g ) Z ( g ) , where Z ( g ) :
Revised: e r ( s ; g ) Z ( g ) , where Z ( g ) :

Original: = (cid:82) e r ( s ; g ) ds is the normalizing constant.
Revised: = (cid:82) e r ( s ; g ) ds is the normalizing constant.

Original: Then, the following equality holds:
Revised: Then, the followingequality holds:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where J ( π ) is the GCRL objective (Eq.
Revised: where J ( π ) is the GCRL objective (Eq.

Original: (2) ) with reward r ( s ; g ) and C :
Revised: (2) ) with reward r ( s ; g ) and C :

Original: =
Revised: =

Original: E g ∼ p
Revised: E g ∼ p

Original: ( g ) [log Z ( g )] is aconstant.
Revised: ( g ) [log Z ( g )] is aconstant.

Original: Proof.
Revised: Proof.

Original: The proof is analogous to that of Proposition E.3.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Rearranging the inequality gives the desired result.
Revised: Rearranging the inequality gives the desired result.

Original: Compared to Proposition E.3, an extra constant term C appear in the equality to account for the needfor normalizing e r ( s ; g ) to make it a proper distribution.
Revised: A constant term C appear in the equality to account for the need for normalizing e r ( s ; g ) to make it aproper distribution.

Original: This, however, does not change the optimalsolution for the goal-conditioned state-occpancy matching objective.
Revised: This, however, does not change the optimal solution for the goal-conditionedstate-occupancy matching objective.

Original: Therefore, we have shownthat for any choice of reward r ( s ; g ) , solving the GCRL problem with a maximum state-entropyregularization is equivalent to optimizing for the goal-conditioned state-occupancy matching objectivewith target distribution p ( s ; g ) : = e r ( s ; g ) Z ( g ) . Proposition E.3. Choose r ( s ; g )
Revised: Therefore, we have shown that for any choice of reward r ( s ; g ) ,solving the GCRL problem with a maximum state-entropy regularization is equivalent to optimizingfor the goal-conditioned state-occupancy matching objective with target distribution p ( s ; g ) :=

Original: =

Original: p ( s ; g ) , the density of states that satisfy goal g  .Then, [Equation] [Equation]
Revised: e r ( s ; g ) Z ( g ) .

Original: Proof.

Original: We have that

Original: [Equation]

Original: Rearranging the inequality gives the desired result.

Original: This result shows that our objective lower bounds the log-transformation of the original GCRLobjective using reward function r ( s ; g )

Original: =

Original: p ( s ; g ) , with the addition of a state entropy maximizationterm, which helps learning an unique and non-degenerate solution.

Original: If we push log( · ) inside theexpectations of J ( π ) (2) through Jensen’s inequality, we in fact obtain equality.

Original: The resultingexpression is

Original: [Equation]

Original: [Equation]

Original: This expression can be understood as a probabilistic interpretation of GCRL.

Original: To see this, for anyr ( s ; g ) , we can choose p ( s ; g ) =

Original: e r ( s ; g ) , corresponding to a softmax distribution with inputs r ( s ; g ) [37].

Original: Therefore, solving the original GCRL objective plus maximizing the state entropyis equivalent to minimizing state-occupancy divergence with respect to the “expert” state-distributione r ( s ; g ) .

Original: E.2 Optimality Conditions for Hindsight Relabeling
Revised: E.2 Optimality Conditions for Hindsight Relabeling

Original: In section 4.2, we have stated that HER is not optimal for most choices of reward functions.
Revised: In section 4.2, we have stated that HER is not optimal for most choices of reward functions.

Original: In thissection, we investigate conditions under which hindsight relabeling methods such as HER would beoptimal.
Revised: In thissection, we investigate conditions under which hindsight relabeling methods such as HER would beoptimal.

Original: Let the goal-relabeling distribution for HER be p HER ( g | s, a ) ; we do not specify the functional formof p HER ( g | s, a ) for generality (see 26).
Revised: Let the goal-relabeling distribution for HER be p HER ( g | s, a ) ; we do not specify the functional formof p HER ( g | s, a ) for generality (see 26).

Original: Then, in order for this distribution to be optimal, then itmust satisfy
Revised: Then, in order for this distribution to be optimal, then itmust satisfy

Original: [Equation]
Revised: [Equation]

Original: Then, the choice of r ( s ; g ) such that this equality holds is the reward function for which HER wouldbe optimal.
Revised: Then, the choice of r ( s ; g ) such that this equality holds is the reward function for which HER wouldbe optimal.

Original: However, solving for r ( s ; g ) is generally challenging and we leave it to future work forinvestigating whether doing so is possible for general f -divergence coupled with neural networks.
Revised: However, solving for r ( s ; g ) is generally challenging and we leave it to future work forinvestigating whether doing so is possible for general f -divergence coupled with neural networks.

Original: This optimality condition is related to a prior work [10], which has found that hindsight relabeling isoptimal in the sense of maximum-entropy inverse RL [51] for a peculiar choice of reward function(see Equation 9 in [10]), which cannot be implemented in practice.
Revised: This optimality condition is related to a prior work [10], which has found that hindsight relabeling isoptimal in the sense of maximum-entropy inverse RL [50] for a peculiar choice of reward function(see Equation 9 in [10]), which cannot be implemented in practice.

Original: Our result is more general as itapplies to any choice of f -divergence, and is not restricted to the form of maximum-entropy inverse
Revised: Our result is more general as itapplies to any choice of f -divergence, and is not restricted to the form of maximum-entropy inverse

Original: RL.
Revised: RL.

Original: E.3 Theoretical Comparison to Prior Regression-based GCRL methods
Revised: E.3 Theoretical Comparison to Prior Regression-based GCRL methods

Original: In section 4.3, we have stated that GoFAR’s theoretical guarantee (Theorem 4.1) is stronger in naturecompared to prior regression-based GCRL methods.
Revised: In section 4.3, we have stated that GoFAR’s theoretical guarantee (Theorem 4.1) is stronger in naturecompared to prior regression-based GCRL methods.

Original: Here, we provide an in-depth discussion.
Revised: Here, we provide an in-depth discussion.

Original: Both GCSL [14] and WGCSL [48] prove that their objectives are lower bounds of the true RLobjective (Theorem 3.1 in [14] and Theorem 1 in [48], respectively); however, in both works, thelower bounds are loose due to constant terms that do not depend on the policy and hence do notvanish to zero.
Revised: Both GCSL [14] and WGCSL [47] prove that their objectives are lower bounds of the true RLobjective (Theorem 3.1 in [14] and Theorem 1 in [47], respectively); however, in both works, thelower bounds are loose due to constant terms that do not depend on the policy and hence do notvanish to zero.

Original: In contrast, GoFAR’s objective (6) is, by construction, a lower bound on the RLobjective, as it simply incorporates a f -divergence regularization.
Revised: In contrast, GoFAR’s objective (6) is, by construction, a lower bound on the RLobjective, as it simply incorporates a f -divergence regularization.

Original: If the offline data d O is on-policy ,then our lower bound is an equality.
Revised: If the offline data d O is on-policy ,then our lower bound is an equality.

Original: In contrast, even with on-policy data, the lower bounds in both
Revised: In contrast, even with on-policy data, the lower bounds in both

Original: GCSL and WGCSL are still loose due to the unavoidable constant terms.
Revised: GCSL and WGCSL are still loose due to the unavoidable constant terms.

Original: GCSL also proves a sub-optimality guarantee (Theorem 3.2 in [14]) under the assumption of fullstate-space coverage.
Revised: GCSL also proves a sub-optimality guarantee (Theorem 3.2 in [14]) under the assumption of fullstate-space coverage.

Original: Though full state-space coverage has been considered in some prior offline
Revised: Though full state-space coverage has been considered in some prior offline

Original: RL works [24, 31], it is much stronger than the concentrability assumption in our Theorem 4.1,which only applies to d ∗ .
Revised: RL works [24, 31], it is much stronger than the concentrability assumption in our Theorem 4.1,which only applies to d ∗ .

Original: Furthermore, this guarantee is not statistical in nature, and instead directlymakes a strong assumption on the maximum total-variance distance between π and optimal π ∗ for the
Revised: Furthermore, this guarantee is not statistical in nature, and instead directlymakes a strong assumption on the maximum total-variance distance between π and optimal π ∗ for the

Original: GCSL objective, which is difficult to verify in practice.
Revised: GCSL objective, which is difficult to verify in practice.

Original: In contrast, our bound suggests asymptoticoptimality: given enough offline data, the solution to GoFAR’s policy objective will converge toπ ∗ .
Revised: In contrast, our bound suggests asymptoticoptimality: given enough offline data, the solution to GoFAR’s policy objective will converge toπ ∗ .

Original: Finally, WGCSL proves a policy improvement guarantee (Proposition 1 in [48]) under theirexponentially weighted advantage; the improvement is not a strict equality, and consequently there isno convergence guarantee to the optimal policy.
Revised: Finally, WGCSL proves a policy improvement guarantee (Proposition 1 in [47]) under theirexponentially weighted advantage; the improvement is not a strict equality, and consequently there isno convergence guarantee to the optimal policy.

Original: Furthermore, this result is not directly dependent ontheir use of an advantage function, so it is not clear the precise role of their advantage function intheir algorithm.
Revised: Furthermore, this result is not directly dependent ontheir use of an advantage function, so it is not clear the precise role of their advantage function intheir algorithm.

Original: F Task Descriptions
Revised: F Task Descriptions

Original: In this section, we describe the tasks in our experiments in Section 5.
Revised: In this section, we describe the tasks in our experiments in Section 5.

Original: F.1 Fetch Tasks
Revised: F.1 Fetch Tasks

Original: [Figure]
Revised: [Figure]

Original: The Fetch environments involve the Fetch robot with the following specifications, developed by
Revised: The Fetch environments involve the Fetch robot with the following specifications, developed by

Original: Plappert et al.
Revised: Plappert et al.

Original: • Seven degrees of freedom
Revised: • Seven degrees of freedom

Revised: [Figure]

Original: • Two pronged parallel gripper
Revised: • Two pronged parallel gripper

Original:  • Three dimensional goal representing Cartesian coordinates of target• Sparse, binary reward signal: 0 when at goal with tolerance 5cm and -1 otherwisejk• 25 Hertz simulation frequency• Four dimensional action space– Three Cartesian dimensions– One dimension to control gripper
Revised: [Figure] • Three dimensional goal representing Cartesian coordinates of target [Figure] • Sparse, binary reward signal: 0 when at goal with tolerance 5cm and -1 otherwisejk

Revised: [Figure]

Revised: • 25 Hertz simulation frequency

Revised: [Figure]

Revised: • Four dimensional action space

Original: F.1.1 Fetch Reach
Revised: [Figure] – Three Cartesian dimensions [Figure] – One dimension to control gripper

Revised: [Figure]

Revised: F.1.1 Fetch Reach

Original: The task is to place the end effector at the target goal position.
Revised: The task is to place the end effector at the target goal position.

Original: Observations consist of the endeffector’s positional state, whether the gripper is closed, and the end effector’s velocity.
Revised: Observations consist of the endeffector’s positional state, whether the gripper is closed, and the end effector’s velocity.

Original: The reward isgiven by:
Revised: The reward isgiven by:

Original: [Equation]
Revised: [Equation]

Original: Push
Revised: Push

Original: The task is to push an object to the target goal position.
Revised: The task is to push an object to the target goal position.

Original: Observations consist of the end effector’sposition, velocity, and gripper state as well as the object’s position, rotational orientation, linearvelocity, and angular velocity.
Revised: Observations consist of the end effector’sposition, velocity, and gripper state as well as the object’s position, rotational orientation, linearvelocity, and angular velocity.

Original: The reward is given by:
Revised: The reward is given by:

Original: [Equation]
Revised: [Equation]

Original: F.1.3 Fetch Slide
Revised: F.1.3 Fetch Slide

Original: In this task, the goal position lies outside of the robot’s reach and the robot must slide the puck-likeobject across the table to the goal.
Revised: In this task, the goal position lies outside of the robot’s reach and the robot must slide the puck-likeobject across the table to the goal.

Original: Observations consist of the end effector’s position, velocity,and gripper state as well as the object’s position, rotational orientation, linear velocity, and angularvelocity.
Revised: Observations consist of the end effector’s position, velocity,and gripper state as well as the object’s position, rotational orientation, linear velocity, and angularvelocity.

Original: The reward is given by:
Revised: The reward is given by:

Original: [Equation]
Revised: [Equation]

Original: The task is to grasp the object and hold it at the goal, which could be on or above the table.
Revised: The task is to grasp the object and hold it at the goal, which could be on or above the table.

Original: Observations consist of the end effector’s position, velocity, and gripper state as well as the object’sposition, rotational orientation, linear velocity, and angular velocity.
Revised: Observations consist of the end effector’s position, velocity, and gripper state as well as the object’sposition, rotational orientation, linear velocity, and angular velocity.

Original: The reward is given by:
Revised: The reward is given by:

Original: [Equation]
Revised: [Equation]

Original: F.2 Hand Reach
Revised: F.2 Hand Reach

Original: Uses a 24 DoF robot hand with a 20 dimensional action space.
Revised: Uses a 24 DoF robot hand with a 20 dimensional action space.

Original: Observations consist of each of the 24joints’ positions and velocities.
Revised: Observations consist of each of the 24joints’ positions and velocities.

Original: The goal space is 15 dimensional corresponding to the positions ofeach of its five fingers.
Revised: The goal space is 15 dimensional corresponding to the positions ofeach of its five fingers.

Original: The goal is achieved when the mean distance of the fingers to their goals isless than 1cm.
Revised: The goal is achieved when the mean distance of the fingers to their goals isless than 1cm.

Original: The reward is binary and sparse: 0 if the goal is reached and -1 otherwise, i.e.r ( s, a, g) = 1 −(cid:32)
Revised: The reward is binary and sparse: 0 if the goal is reached and -1 otherwise, i.e.r ( s, a, g

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: 0 .(cid:33)
Revised: (cid:33)

Original: F.3 D’ClawTurn (Simulation)
Revised: F.3 D’ClawTurn (Simulation)

Original: First introduced by Ahn et al.
Revised: First introduced by Ahn et al.

Original: [2], the D’Claw environment has a 9 DoF three-fingered robotic hand.
Revised: [2], the D’Claw environment has a 9 DoF three-fingered robotic hand.

Original: The turn task consists of turning the valve to a desired angle.
Revised: The turn task consists of turning the valve to a desired angle.

Original: The initial angle is randomly chosenfrom [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − 2 ∗ π 3 , 2 ∗ π 3 ] .
Revised: The initial angle is randomly chosenfrom [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − 2 ∗ π 3 , 2 ∗ π 3 ] .

Original: The observation space is 21D,consisting of the current joint angles θ t , their velocities ˙ θ , angle between current and goal angle, andthe previous action.
Revised: The observation space is 21D,consisting of the current joint angles θ t , their velocities ˙ θ , angle between current and goal angle, andthe previous action.

Original: The environment terminates after 80 steps.
Revised: The environment terminates after 80 steps.

Original: The reward function is defined as:
Revised: The reward function is defined as:

Original: [Equation]
Revised: [Equation]

Original: F.4 D’ClawTurn (Real)
Revised: F.4 D’ClawTurn (Real)

Original: To make real-world data collection easier, we slightly modify the initial and target angle distributions.
Revised: To make real-world data collection easier, we slightly modify the initial and target angle distributions.

Original: The initial angle is randomly chosen from [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − π 2 , π 2 ] .
Revised: The initial angle is randomly chosen from [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − π 2 , π 2 ] .

Original: Using this task distribution, collecting 400K transitions with random actions takes about 15 hours.
Revised: Using this task distribution, collecting 400K transitions with random actions takes about 15 hours.

Original: In
Revised: In

Original: Figure 8, we also include a larger picture of the robot platform.
Revised: Figure 8, we also include a larger picture of the robot platform.

Original: G Experimental Details
Revised: G Experimental Details

Original: In this section, we provide experimental details omitted in Section 5 of the main text.
Revised: In this section, we provide experimental details omitted in Section 5 of the main text.

Original: These include(1) technical details of the baseline methods, (2) hyperparameter and architecture details for allmethods, (3) offline GCRL dataset details, and finally, (4) experimental details of the zero-shottransfer experiment.
Revised: These include(1) technical details of the baseline methods, (2) hyperparameter and architecture details for allmethods, (3) offline GCRL dataset details, and finally, (4) experimental details of the zero-shottransfer experiment.

Original: G.1 Baseline Implementation Details
Revised: G.1 Baseline Implementation Details

Original: DDPG.
Revised: DDPG.

Original: We use an open-source implementation of DDPG, which has already tuned DDPG on theset of Fetch tasks.
Revised: We use an open-source implementation of DDPG, which has already tuned DDPG on theset of Fetch tasks.

Original: We implement all other methods on top of this implementation, keeping identicalarchitectures and hyperparameters when appropriate.
Revised: We implement all other methods on top of this implementation, keeping identicalarchitectures and hyperparameters when appropriate.

Original: The critic objective is
Revised: The critic objective is

Original: [Equation]
Revised: [Equation]

Original: (62) where ¯ Q denotes the stop-gradient operation.
Revised: (57) where ¯ Q denotes the stop-gradient operation.

Original: The policy objective is
Revised: The policy objective is

Original: [Equation]
Revised: [Equation]

Original: DDPG updates the critic and the policy in an alternating fashion.
Revised: DDPG updates the critic and the policy in an alternating fashion.

Original: ActionableModel.
Revised: ActionableModel.

Original: We implement AM on top of DDPG.
Revised: We implement AM on top of DDPG.

Original: Specifically, we add a CQL loss in the criticupdate:
Revised: Specifically, we add a CQL loss in the criticupdate:

Original: [Equation]
Revised: [Equation]

Original: where d ˜ O is the distribution of the relabelled dataset.
Revised: where d ˜ O is the distribution of the relabelled dataset.

Original: In practice, we sample 10 random actionsfrom the action-space to approximate this expectation.
Revised: In practice, we sample 10 random actionsfrom the action-space to approximate this expectation.

Original: Furthermore, we implement goal-chaining,where for half of the relabeled transitions in each minibatch update, the relabelled goals are randomlysampled from the offline dataset.
Revised: Furthermore, we implement goal-chaining,where for half of the relabeled transitions in each minibatch update, the relabelled goals are randomlysampled from the offline dataset.

Original: We found goal-chaining to not be stable in some environments, inparticular, FetchPush, FetchPickAndPlace, and FetchSlide.
Revised: We found goal-chaining to not be stable in some environments, inparticular, FetchPush, FetchPickAndPlace, and FetchSlide.

Original: Therefore, to obtain better results, weremove goal-chaining for these environments in our experiments.
Revised: Therefore, to obtain better results, weremove goal-chaining for these environments in our experiments.

Original: GCSL.
Revised: GCSL.

Original: We implement GCSL by removing the DDPG critic component and changing the policy lossto maximum likelihood:
Revised: We implement GCSL by removing the DDPG critic component and changing the policy lossto maximum likelihood:

Original: [Equation]
Revised: [Equation]

Original: WGCSL.
Revised: WGCSL.

Original: We implement WGCSL on top of GCSL by including a Q-function.
Revised: We implement WGCSL on top of GCSL by including a Q-function.

Original: The Q-functionis trained using TD error as in DDPG.
Revised: The Q-functionis trained using TD error as in DDPG.

Original: To compute the advantage term, we follow the original
Revised: To compute the advantage term, we follow the original

Original: WGCSL algorithm by setting V ( s t ; g ) =
Revised: WGCSL algorithm by setting V ( s t ; g ) =

Original: Q ( s t , π ( s t +1 , g ); g ) and then computing A ( s t , a t , g )
Revised: Q ( s t , π ( s t +1 , g ); g ) and then computing A ( s t , a t , g )

Original: =
Revised: =

Original: r ( s t ; g ) + γV ( s t +1 ; g ) − V ( s t ; g ) .
Revised: r ( s t ; g ) + γV ( s t +1 ; g ) − V ( s t ; g ) .

Original: Using this, the WGCSL policy objective is
Revised: Using this, the WGCSL policy objective is

Original: [Equation]
Revised: [Equation]

Original: (66) where we clip exp( · ) for numerical stability.
Revised: (61) where we clip exp( · ) for numerical stability.

Original: The original WGCSL uses different HER rates forthe critic and the actor training.
Revised: The original WGCSL uses different HER rates forthe critic and the actor training.

Original: To make the implementation simple and consistent with all otherapproaches, we use the same HER rate for both components.
Revised: To make the implementation simple and consistent with all otherapproaches, we use the same HER rate for both components.

Original: With the exception of AM, all baselines set the goal-relabeling distribution d ˜ O to be the uniformdistribution over future states in the same trajectory (See Equation (26)).
Revised: With the exception of AM, all baselines set the goal-relabeling distribution d ˜ O to be the uniformdistribution over future states in the same trajectory (See Equation (26)).

Original: G.2 Architectures and Hyperparameters
Revised: G.2 Architectures and Hyperparameters

Original: Each algorithm uses their own set of fixed hyperparameters for all tasks.
Revised: Each algorithm uses their own set of fixed hyperparameters for all tasks.

Original: WGCSL, GCSL, and DDPGare already tuned on our set of tasks [40, 48], so we use the reported values from prior works; AM, inour implementation, shares same networks as DDPG, so we use DDPG’s values.
Revised: WGCSL, GCSL, and DDPGare already tuned on our set of tasks [39, 47], so we use the reported values from prior works; AM, inour implementation, shares same networks as DDPG, so we use DDPG’s values.

Original: For GoFAR, weuse identical hyperparameters as WGCSL because they share similar network components; GoFARadditionally trains a discriminator, for which we use the same architecture and learning rate as thevalue network.
Revised: For GoFAR, weuse identical hyperparameters as WGCSL because they share similar network components; GoFARadditionally trains a discriminator, for which we use the same architecture and learning rate as thevalue network.

Original: We impose a small discriminator gradient penalty [16] to prevent overfitting.
Revised: We impose a small discriminator gradient penalty [16] to prevent overfitting.

Original: For allexperiments, We train each method for 3 seeds, and each training run uses 400k minibatch updates ofsize 512.
Revised: For allexperiments, We train each method for 3 seeds, and each training run uses 400k minibatch updates ofsize 512.

Original: The architectures and hyperparameters for all methods are reported in Table 3.
Revised: The architectures and hyperparameters for all methods are reported in Table 3.

Original: G.3 Offline GCRL Experiments
Revised: G.3 Offline GCRL Experiments

Original: Datasets.
Revised: Datasets.

Original: For each environment, the offline dataset composition is determined by whether datacollected by random actions provides sufficient coverage of the desired goal distribution.
Revised: For each environment, the offline dataset composition is determined by whether datacollected by random actions provides sufficient coverage of the desired goal distribution.

Original: For
Revised: For

Original: FetchReach and D’ClawTurn, we find this to be the case and choose the offline dataset to be 1 millionrandom transitions.
Revised: FetchReach and D’ClawTurn, we find this to be the case and choose the offline dataset to be 1 millionrandom transitions.

Original: For the other four tasks, random data does not capture meaningful goals, so wecreate a mixture dataset with 100K transitions from a trained DDPG-HER agent and 900K randomtransitions; the transitions are not labeled with their sources.
Revised: For the other four tasks, random data does not capture meaningful goals, so wecreate a mixture dataset with 100K transitions from a trained DDPG-HER agent and 900K randomtransitions; the transitions are not labeled with their sources.

Original: This mixture setup has been consideredin prior works [21, 32] and is reminiscent of real-world datasets, where only a small portion of thedataset is task-relevant but all transitions provide useful information about the environment.
Revised: This mixture setup has been consideredin prior works [21, 32] and is reminiscent of real-world datasets, where only a small portion of thedataset is task-relevant but all transitions provide useful information about the environment.

Original: G.4 Zero-Shot Transfer Experiments
Revised: G.4 Zero-Shot Transfer Experiments

Original: We use GoFAR (Binary) variant for trainning the GoFAR planner.
Revised: We use GoFAR (Binary) variant for trainning the GoFAR planner.

Original: The low-level controller is trainedusing an online DDPG algorithm on a narrow goal distribution, set to be closed to the object’s initialpositions.
Revised: The low-level controller is trainedusing an online DDPG algorithm on a narrow goal distribution, set to be closed to the object’s initialpositions.

Original: GoFAR Hierarchical Controller operates by first generating a sequence of subgoals ( g 1 , ..., g T ) usingπ high by recursively feeding the newest generated goal and conditioning on the final goal g .
Revised: GoFAR Hierarchical Controller operates by first generating a sequence of subgoals ( g 1 , ..., g T ) usingπ high by recursively feeding the newest generated goal and conditioning on the final goal g .

Original: Then, at
Revised: Then, at

Original: [Table]
Revised: [Table]

Original: each time step t , the low-level controller executes action π low ( a t | s t , g t ) .
Revised: each time step t , the low-level controller executes action π low ( a t | s t , g t ) .

Original: The high-level subgoalsare not re-planned during low-level controller execution.
Revised: The high-level subgoalsare not re-planned during low-level controller execution.

Original: We note that this is a simple planningalgorithm
Revised: We note that this is a simple planningalgorithm

Original: [Table]
Revised: [Table]

Original: and improvement in performance can be expected by considering more sophisticatedplanning approaches.
Revised: and improvement in performance can be expected by considering more sophisticatedplanning approaches.

Original: H Additional ResultsH.1 Offline GCRL Full Results
Revised: H Additional ResultsH.1 Offline GCRL Full Results

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide the full results table for discounted return, final distance [Table] and success ratemetrics, including error bars over 10 random seeds.
Revised: we provide the full results table for discounted return, final distance, and success ratemetrics, including error bars over 10 random seeds.

Original: The number inside the parenthesis indicatesthe best HER rate for the baseline methods on the task.
Revised: The number inside the parenthesis indicatesthe best HER rate for the baseline methods on the task.

Original: Star ( ⋆ ) indicate statistically significantimprovement over the second best-performing method under a 2-sample t -test.
Revised: Star ( ⋆ ) indicate statistically significantimprovement over the second best-performing method under a 2-sample t -test.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H.2 Ablations
Revised: H.2 Ablations

Original: [Table]
Revised: [Table]

Original: We also include the full task-breakdown table of GoFAR ablations presented in Figure 3 for completeness.
Revised: We also include the full task-breakdown table of GoFAR ablations presented in Figure 3 for completeness.

Original: As shown in 7, GoFAR and GoFAR (HER) perform comparatively on all tasks.
Revised: As shown in 7, GoFAR and GoFAR (HER) perform comparatively on all tasks.

Original: GoFAR (binary)is slightly worse across tasks, and GoFAR (KL) collapses due to the use of an unstable f -divergence.
Revised: GoFAR (binary)is slightly worse across tasks, and GoFAR (KL) collapses due to the use of an unstable f -divergence.

Original: [Table]
Revised: [Table]

Original: H.3 Real-World Dexterous Manipulations
Revised: H.3 Real-World Dexterous Manipulations

Original: In our qualitative analysis, we visualize all methods on a specific task instance of turning the valveprong (marked by the red strip) clockwise for 90 degree; the goal location is marked by the greenstrip.
Revised: In our qualitative analysis, we visualize all methods on a specific task instance of turning the valveprong (marked by the red strip) clockwise for 90 degree; the goal location is marked by the greenstrip.

Original: The robot initial pose is randomized.
Revised: The robot initial pose is randomized.

Original: As shown in Figure 9, GoFAR reaches the goal with threerandom initial poses, whereas all baselines fail.
Revised: As shown in Figure 9, GoFAR reaches the goal with threerandom initial poses, whereas all baselines fail.

Original: See the figure captions for detail.
Revised: See the figure captions for detail.

Original: Policy videos areincluded in the supplementary material.
Revised: Policy videos areincluded in the supplementary material.

Original: [Figure]
Revised: [Figure]

Original: H.4 Zero-Shot Plan Transfer
Revised: H.4 Zero-Shot Plan Transfer

Original: We visualize GoFAR hierarchical controller and the plain low-level controller on three distinct goalsin Figure 10.
Revised: We visualize GoFAR hierarchical controller and the plain low-level controller on three distinct goalsin Figure 10.

Original: See the figure caption for detail.
Revised: See the figure caption for detail.

Original: Policy videos are included in the supplementarymaterial.
Revised: Policy videos are included in the supplementarymaterial.

Original: [Figure]
Revised: [Figure]