Original: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression
Revised: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression

Revised: Author s)

Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Offline goal-conditioned reinforcement learning (GCRL) promises general-purposeskill learning in the form of reaching diverse goals from purely offline datasets.
Revised: Offline goal-conditioned reinforcement learning (GCRL) promises general-purposeskill learning in the form of reaching diverse goals from purely offline datasets.

Original: Wepropose Go al-conditioned f - A dvantage R egression (GoFAR), a novel regressionbased offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a stateoccupancy matching problem between a dynamics-abiding imitator agent and anexpert agent that directly teleports to the goal.
Revised: Wepropose Go al-conditioned f - A dvantage R egression (GoFAR), a novel regressionbased offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a stateoccupancy matching problem between a dynamics-abiding imitator agent and anexpert agent that directly teleports to the goal.

Original: In contrast to prior approaches,
Revised: In contrast to prior approaches,

Original: GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks.
Revised: GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks.

Original: These distinct features confer GoFARwith much better offline performance and stability as well as statistical performanceguarantee that is unattainable for prior methods.
Revised: These distinct features confer GoFARwith much better offline performance and stability as well as statistical performanceguarantee that is unattainable for prior methods.

Original: Furthermore, we demonstratethat GoFAR’s training objectives can be re-purposed to learn an agent-independentgoal-conditioned planner from purely offline source-domain data, which enableszero-shot transfer to new target domains.
Revised: Furthermore, we demonstratethat GoFAR’s training objectives can be re-purposed to learn an agent-independentgoal-conditioned planner from purely offline source-domain data, which enableszero-shot transfer to new target domains.

Original: Through extensive experiments, wevalidate GoFAR’s effectiveness in various problem settings and tasks, significantlyoutperforming prior state-of-art.
Revised: Through extensive experiments, wevalidate GoFAR’s effectiveness in various problem settings and tasks, significantlyoutperforming prior state-of-art.

Original: Notably, on a real robotic dexterous manipulationtask, while no other method makes meaningful progress, GoFAR acquires complexmanipulation behavior that successfully accomplishes diverse goals.
Revised: Notably, on a real robotic dexterous manipulationtask, while no other method makes meaningful progress, GoFAR acquires complexmanipulation behavior that successfully accomplishes diverse goals.

Original:  Introduction
Revised:  Introduction

Original: Goal-conditioned reinforcement learning [18, 43, 39] (GCRL) aims to learn a repertoire of skillsin the form of reaching distinct goals.
Revised: Goal-conditioned reinforcement learning [18, 44, 40] (GCRL) aims to learn a repertoire of skillsin the form of reaching distinct goals.

Original: Offline GCRL [6, 47] is particularly promising because itenables learning general goal-reaching policies from purely offline interaction datasets without anyenvironment interaction [28, 25], which can be expensive in the real-world.
Revised: Offline GCRL [6, 48] is particularly promising because itenables learning general goal-reaching policies from purely offline interaction datasets without anyenvironment interaction [28, 25], which can be expensive in the real-world.

Original: As offline datasetscontain diverse goals and become increasingly prevalent [9, 19, 6], policies learned this way canacquire a large set of useful primitives for downstream tasks [30].
Revised: As offline datasetscontain diverse goals and become increasingly prevalent [9, 19, 6], policies learned this way canacquire a large set of useful primitives for downstream tasks [30].

Original: A central challenge in GCRL is thesparsity of reward signal [3]; without any additional knowledge about the environment, an agent at astate typically only accrues positive binary reward when the state lies within the goal neighborhood.
Revised: A central challenge in GCRL is thesparsity of reward signal [3]; without any additional knowledge about the environment, an agent at astate typically only accrues positive binary reward when the state lies within the goal neighborhood.

Original: This sparse reward problem is exacerbated in the offline setting, in which the agent cannot explorethe environment to discover more informative states about desired goals.
Revised: This sparse reward problem is exacerbated in the offline setting, in which the agent cannot explorethe environment to discover more informative states about desired goals.

Original: Therefore, designing aneffective offline GCRL algorithm is a concrete yet challenging path towards general-purpose andscalable policy learning.
Revised: Therefore, designing aneffective offline GCRL algorithm is a concrete yet challenging path towards general-purpose andscalable policy learning.

Original: In this paper, we present a novel offline GCRL algorithm,
Revised: In this paper, we present a novel offline GCRL algorithm,

Original: Go al-conditioned f - A dvantage R egression(GoFAR), first casting GCRL as a state-occupancy matching [27, 32] problem and then deriving aregression-based policy objective.
Revised: Go al-conditioned f - A dvantage R egression(GoFAR), first casting GCRL as a state-occupancy matching [27, 32] problem and then deriving aregression-based policy objective.

Original: In particular, GoFAR begins with the following goal-conditionedstate-matching objective:
Revised: In particular, GoFAR begins with the following goal-conditionedstate-matching objective:

Original: [Equation]
Revised: [Equation]

Original: where d π ( s ; g ) is the goal-conditioned state-occupancy distribution of policy π and p ( s ; g ) is thedistribution of states that satisfy a particular goal g .
Revised: where d π ( s ; g ) is the goal-conditioned state-occupancy distribution of policy π and p ( s ; g ) is thedistribution of states that satisfy a particular goal g .

Original: This objective casts goal-conditioned offline RL as an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expertwho can teleport to the goal in one step; see Figure 2.
Revised: This objective casts goal-conditioned offline RLas an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expert whocan teleport to the goal in one step; see Figure 2.

Original: Posing GCRL this way is mathematicallyprincipled, as we show that this objective is equivalent to a probabilistic interpretation of GCRLthat additionally encourages maximizing state entropy.
Revised: Posing GCRL this way is mathematically principled,as we show that this objective is equivalent to a probabilistic interpretation of GCRL that additionallyencourages maximizing state entropy.

Original: More importantly, this objectives enables usto extend a state-of-art offline imitation learning algorithm [32] to the goal-conditioned setting andadmits elegant optimization using purely offline data by considering an f -divergence regularizedobjective and exploiting ideas from convex duality theory [41, 4].
Revised: More importantly, this objective admits elegant optimizationusing purely offline data by considering an f -divergence regularized objective and exploiting ideasfrom convex duality theory [42, 4].

Original: In particular, we obtain thedual optimal value function V ∗ from a single unconstrained optimization problem, using which weconstruct the optimal advantage function (we refer to this as f-advantage ) that serves as importanceweighting for a regression-based policy training objective; see Figure 1 for a schematic illustration.
Revised: In particular, we obtain the dual optimal value function V ∗ from asingle unconstrained optimization problem, using which we construct the optimal advantage function(we refer to this as f-advantage ) that serves as importance weighting for a regression-based policytraining objective; see Figure 1 for a schematic illustration.

Original: There are several distinct features to our approach.
Revised: There are several distinct features to our approach.

Original: First and foremost is GoFAR’s lackof goal relabeling.
Revised: First and foremost is GoFAR’s lackof goal relabeling.

Original: Hindsight goal relabeling [3](known as HER in the literature) relabels trajectory goals to be states that were actuallyachieved instead of the originally commandedgoals.
Revised: Hindsight goal relabeling [3](known as HER in the literature) relabels trajectory goals to be states that were actuallyachieved instead of the originally commandedgoals.

Original: This heuristic is critical for alleviating thesparse-reward problem in prior GCRL methods,but is unnecessary for GoFAR.
Revised: This heuristic is critical for alleviating thesparse-reward problem in prior GCRL methods,but is unnecessary for GoFAR.

Original: GoFAR updates its policy as if all data is already coming from theoptimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling.
Revised: GoFAR updates its policy as if all data is already coming from theoptimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling.

Original: GoFAR’s relabeling-free training is of significant practical benefits.
Revised: GoFAR’s relabeling-free training is of significant practical benefits.

Original: First, it enables more stable andsimpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot beeasily performed offline [48].
Revised: First, it enables more stable andsimpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot beeasily performed offline [49].

Original: Second, hindsight relabeling suffers from hindsight bias in stochasticenvironments [44], as achieved goals may be accomplished due to noise; this is exacerbated in theoffline setting due to inability to collect more data.
Revised: Second, hindsight relabeling suffers from hindsight bias in stochasticenvironments [45], as achieved goals may be accomplished due to noise; this is exacerbated in theoffline setting due to inability to collect more data.

Original: By bypassing hindsight relabeling, GoFARpromises to be more robust and scalable.
Revised: By bypassing hindsight relabeling, GoFARpromises to be more robust and scalable.

Original: Second, as suggested in Figure 1, GoFAR’s value and policy training steps are completely uninterleaved : we do not need to update the policy until the value function has converged.
Revised: Second, as suggested in Figure 1, GoFAR’s value and policy training steps are completely uninterleaved : we do not need to update the policy until the value function has converged.

Original: This confersgreater offline training stability [23, 32] compared to prior works, which mostly involve alternatingupdates to a critic Q-network and policy network.
Revised: This confersgreater offline training stability [23, 32] compared to prior works, which mostly involve alternatingupdates to a critic Q-network and policy network.

Original: Furthermore, it enables an algorithmic reductionof GoFAR to weighted regression [7], which allows us to obtain strong finite-sample statisticalguarantee on GoFAR’s performance; prior regression-based GCRL approaches [14, 47] do not enjoythis reduction and obtain much weaker theoretical guarantees.
Revised: Furthermore, it enables an algorithmic reductionof GoFAR to weighted regression [7], which allows us to obtain strong finite-sample statisticalguarantee on GoFAR’s performance; prior regression-based GCRL approaches [14, 48] do not enjoythis reduction and obtain much weaker theoretical guarantees.

Original: Finally, we show that GoFAR can also be used to learn a goal-conditioned planner [36, 5].
Revised: Finally, we show that GoFAR can also be used to learn a goal-conditioned planner [36, 5].

Original: Thekey insight lies in observing that, under a mild assumption, GoFAR’s dual value function objectivedoes not depend on the action and can learn from state-only offline data.
Revised: Thekey insight lies in observing that, under a mild assumption, GoFAR’s dual value function objectivedoes not depend on the action and can learn from state-only offline data.

Original: This enables learning agoal-centric value function and thereby a near-optimal goal-conditioned planner that is capable ofzero-shot transferring to new domains of the same task that shares the goal space.
Revised: This enables learning agoal-centric value function and thereby a near-optimal goal-conditioned planner that is capable ofzero-shot transferring to new domains of the same task that shares the goal space.

Original: In our experiments,we illustrate that GoFAR planner can indeed plan effective subgoals in a new target domain andenable a low-level controller to reach distant goals that it is not designed for.
Revised: In our experiments,we illustrate that GoFAR planner can indeed plan effective subgoals in a new target domain andenable a low-level controller to reach distant goals that it is not designed for.

Original: We extensively evaluate GoFAR on a variety of offline GCRL environments of varying task complexityand dataset composition, and show that it outperforms all baselines in all settings.
Revised: We extensively evaluate GoFAR on a variety of offline GCRL environments of varying task complexityand dataset composition, and show that it outperforms all baselines in all settings.

Original: Notably, GoFARis more robust to stochastic environments than methods that depend on hindsight relabeling.
Revised: Notably, GoFARis more robust to stochastic environments than methods that depend on hindsight relabeling.

Original: Weadditionally demonstrate that GoFAR learns complex manipulation behavior on a real-world highdimensional dexterous manipulation task [2], for which most baselines fail to make any progress.
Revised: Weadditionally demonstrate that GoFAR learns complex manipulation behavior on a real-world highdimensional dexterous manipulation task [2], for which most baselines fail to make any progress.

Original: Finally, we showcase GoFAR’s planning capability in a cross-robot zero-shot transfer task.
Revised: Finally, we showcase GoFAR’s planning capability in a cross-robot zero-shot transfer task.

Original: Tosummarize, our contributions are:
Revised: Tosummarize, our contributions are

Original: 1. GoFAR, a novel offline GCRL algorithm derived from goal-conditioned state-matching Detailed technical derivation and analysis of GoFAR’s distinct
Revised: 1. GoFAR, a novel offline GCRL algorithm derived from goal-conditioned state-occupancymatching2.

Original:  features: relabeling free, ) uninterleaved optimization, and (3) planning 
Revised: Detailed technical derivation and analysis of GoFAR’s distinct features: (1) relabeling-free,(2) uninterleaved optimization, and (3) planning capability3.

Original: capability Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities
Revised: Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities

Original:  Related Work
Revised:  Related Work

Original: Offline Goal-Conditioned Reinforcement Learning.
Revised: Offline Goal-Conditioned Reinforcement Learning.

Original: One core challenge of (offline) GCRL [18, 43]is the sparse-reward nature of goal-reaching tasks.
Revised: One core challenge of (offline) GCRL [18, 44]is the sparse-reward nature of goal-reaching tasks.

Original: A popular strategy is hindsight goal relabeling [3](or HER), which relabels off-policy transitions with future goals they have achieved instead of theoriginal commanded goals.
Revised: A popular strategy is hindsight goal relabeling [3](or HER), which relabels off-policy transitions with future goals they have achieved instead of theoriginal commanded goals.

Original: Existing offline GCRL algorithms [6, 47] adapt HER-based online GCRLalgorithms to the offline setting by incorporating additional components conducive to offline training.[6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [24] as wellas goal-chaining, which expands the pool of candidate relabeling goals to the entire offline dataset.
Revised: Existing offline GCRL algorithms [6, 48] adapt HER-based online GCRLalgorithms to the offline setting by incorporating additional components conducive to offline training.[6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [24] as wellas goal-chaining, which expands the pool of candidate relabeling goals to the entire offline dataset.

Original: Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shownto be a simple and effective method [14].
Revised: Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shownto be a simple and effective method [14].

Original: [47] improves upon [14] by incorporating discount-factorand advantage function weighting [37, 38], and shows improved performance in offline GCRL.
Revised: [48] improves upon [14] by incorporating discount-factorand advantage function weighting [38, 39], and shows improved performance in offline GCRL.

Original: Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting an onlinealgorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matchingand derives an elegant algorithm from the dual formulation that is naturally suited for the offlinesetting and carries many distinct properties that make it more stable, scalable, and versatile.
Revised: Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting an onlinealgorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matchingand derives an elegant algorithm from the dual formulation that is naturally suited for the offlinesetting and carries many distinct properties that make it more stable, scalable, and versatile.

Original: State-Occupancy Matching.
Revised: State-Occupancy Matching.

Original: State occupancy matching objectives have been shown to be effectivefor learning from observations [32, 49], exploration [27], as well as matching a hand-designed targetstate distribution [13]; occupancy matching, in general, has been explored in the imitation learningliterature [17, 45, 13, 20].
Revised: State occupancy matching objectives have been shown to be effectivefor learning from observations [32, 50] as well as exploration [27]; occupancy matching, in general,has been explored in the imitation learning literature [17, 46, 13, 20].

Original: Our strategy of estimating the state-occupancy ratio is related to  DICE [Figure] techniques for offline policy optimization [33, 35, 26, 21, 32].
Revised: At a technical level, ourstrategy of estimating the state-occupancy ratio is related to the DICE technique [33, 35], which hasrecently been extended to the offline setting [26, 21, 32].

Original: The closest work to ours is [32], whichproposed f -divergence  state-occupancy matching  for offline imitation learning.Our work showsthat such a state-occupancy matching approach can be applied to GCRL.
Revised: The closest work to ours is [32], whichproposes a f -divergence based state-occupancy matching objective for offline imitation learning.

Original: Algorithmically, GoFARdiffers from [32] in that it does not require a separate dataset of expert demonstrations; it achievesthis by modifying the discriminator, and in some settings can even be used without a discriminator,whereas a discriminator is always required by [32].
Revised: Ourwork differs in that we consider a goal-conditioned setting and do not require a separate dataset ofdemonstrations for policy learning.

Original: Furthermore, we derive several new algorithmicproperties (Section 4.2-4.4) that are novel and particularly advantageous for offline GCRL.
Revised: Furthermore, we derive several new results (Section 4.2-4.4) thatare of particular interest in the GCRL setting.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Goal-Conditioned Reinforcement Learning.
Revised: Goal-Conditioned Reinforcement Learning.

Original: We consider an infinite-horizon Markov decisionprocess (MDP)
Revised: We consider an infinite-horizon Markov decisionprocess (MDP)

Original: [40] M = ( S, A, R, T, µ 0 , γ ) with state space S , action space A , deterministicrewards r ( s, a ) , stochastic transitions s ′ ∼ T ( s, a ) , initial state distribution µ 0 ( s ) , and discountfactor γ ∈ (0 , 1] .
Revised: [41] M = ( S, A, R, T, µ 0 , γ ) with state space S , action space A , deterministicrewards r ( s, a ) , stochastic transitions s ′ ∼ T ( s, a ) , initial state distribution µ 0 ( s ) , and discountfactor γ ∈ (0 , 1] .

Original: A policy π : S → ∆( A ) outputs a distribution over actions to use in a given state.
Revised: A policy π : S → ∆( A ) outputs a distribution over actions to use in a given state.

Original: In goal-conditioned RL, the MDP additionally assumes a goal space G :=
Revised: In goal-conditioned RL, the MDP additionally assumes a goal space G :=

Original: { ϕ ( s ) | s ∈ S } , where thestate-to-goal mapping ϕ : S → G is known.
Revised: { ϕ ( s ) | s ∈ S } , where thestate-to-goal mapping ϕ : S → G is known.

Original: Now, the reward function r ( s ; g ) 1 as well as the policyπ ( a | s, g ) depend on the commanded goal g ∈ G .
Revised: Now, the reward function r ( s ; g ) 1 as well as the policyπ ( a | s, g ) depend on the commanded goal g ∈ G .

Original: Given a distribution over desired goals p ( g ) , theobjective of goal-conditioned RL is to find a policy π that maximizes the discounted return:
Revised: Given a distribution over desired goals p ( g ) , theobjective of goal-conditioned RL is to find a policy π that maximizes the discounted cumulativereturn:

Original: [Equation]
Revised: J ( π ) : =

Original: [Equation]
Revised: E g ∼ p ( g ) ,s 0 ∼ µ 0 ,a t ∼ π ( ·| s t ,g ) ,s t +1 ∼ T ( ·| s t ,a t ) (cid:34) ∞ ( cid:88) t = γ t r ( s t ; g ) (cid:35) (2)

Original: The goal-conditioned state-action occupancy distribution d π ( s, a ; g ) : S ×
Revised: The goal-conditioned state-action occupancy distribution d π ( s, a ; g ) : S ×

Original: A × G → [0 , 1] of π is[Equation] g
Revised: A × G → [0 , 1] of π is

Original: [Equation]
Revised: [Equation]

Original: which captures the relative frequency of state-action visitations for a policy π conditioned on goal g .
Revised: which captures the relative frequency of state-action visitations for a policy π conditioned on goal g .

Original: The state-occupancy distribution then marginalizes over actions: d π ( s ; g )
Revised: The state-occupancy distribution then marginalizes over actions: d π ( s ; g )

Original: = (cid:80) a d π ( s, a ; g ) .
Revised: = (cid:80) a d π ( s, a ; g ) .

Original: Then, itfollows that π ( a | s, g ) =
Revised: Then, itfollows that π ( a | s, g ) =

Original: d π ( s,a ; g ) d
Revised: d π ( s,a ; g ) d

Original: π ( s ; g ) .
Revised: π ( s ; g ) .

Original: A state-action occupancy distribution must satisfy the Bellmanflow constraint in order for it to be an occupancy distribution for some stationary policy π :
Revised: A state-action occupancy distribution must satisfy the Bellmanflow constraint in order for it to be an occupancy distribution for some stationary policy π :

Original: [Equation]
Revised: [Equation]

Original: We write d π ( s, g )
Revised: We write d π ( s, g )

Original: =
Revised: =

Original: p ( g ) d π ( s ; g ) as the joint goal-state density induced by p ( g ) and the policy π .
Revised: p ( g ) d π ( s ; g ) as the joint goal-state density induced by p ( g ) and the policy π .

Original: Finally, given d π , we can express the objective function (2) as J ( π ) =
Revised: Finally, given d π , we can express the objective function (2) as J ( π ) =

Original: 11 − γ E
Revised: 11 − γ E

Original: ( s,g )
Revised: ( s,g )

Original: ∼ d π ( s,g ) [ r ( s ; g )] .
Revised: ∼ d π ( s,g ) [ r ( s ; g )] .

Original: Offline GCRL.
Revised: Offline GCRL.

Original: In offline GCRL, the agent cannot interact with the environment M ; instead,it is equipped with a static dataset of logged transitions D := { τ i } Ni =1 , where each trajectoryτ ( i ) =
Revised: In offline GCRL, the agent cannot interact with the environment M ; instead,it is equipped with a static dataset of logged transitions D := { τ i } Ni =1 , where each trajectoryτ ( i ) =

Original: ( s ( i ) 0 , a ( i ) 0 , r
Revised: ( s ( i ) 0 , a ( i ) 0 , r

Original: ( i ) 0 , s ( i ) 1 , ... ; g ( i ) ) with s ( i ) 

Revised: ( i ) 0 , s ( i ) 1 , ... ; g ( i ) ) with s ( i )

Original: ∼ µ 0 and g ( i ) is the commanded goal of the trajectory.
Revised: ∼ µ 0 and g ( i ) is the commanded goal of the trajectory.

Original: Note that trajectories need not to be generated from a goal-directed agent, in which case g ( i ) can berandomly drawn from p ( g ) .
Revised: Note that trajectories need not to be generated from a goal-directed agent, in which case g ( i ) can berandomly drawn from p ( g ) .

Original: We denote the empirical goal-conditioned state-action occupancies of
Revised: We denote the empirical goal-conditioned state-action occupancies of

Original: D O as d O ( s, a ; g ) .
Revised: D O as d O ( s, a ; g ) .

Original:  Goal-Conditioned f -Advantage Regression
Revised:  Goal-Conditioned f -Advantage Regression

Original: In this section, we introduce Goal-conditioned f -Advantage Regression (GoFAR).
Revised: In this section, we introduce Goal-conditioned f -Advantage Regression (GoFAR).

Original: We first derive thealgorithm in full (Section 4.1), then delve deep into its several appealing properties (Section 4.2-4.4).
Revised: We first derive thealgorithm in full (Section 4.1), then delve deep into its several appealing properties (Section 4.2-4.4).

Original: 4.1 Algorithm
Revised: Algorithm

Original: We first show that goal-conditioned state-occupancy matching is a mathematically principled approachfor solving general GCRL problems, formalizing the teleportation intuition in the introduction. Proposition 4.1. Given any r ( s ; g ) , for each g in the support of p ( g ) , define p ( s ; g ) = e r ( s ; g ) Z ( g ) , where
Revised: Recall the goal-conditioned state-occupancy matching objective stated in (1):

Original: Z ( g ) :

Original: = (cid:82) e r ( s ; g ) ds is the normalizing constant.

Original: Then, the following equality holds:

Original: [Equation]
Revised: [Equation]

Original: where J ( π ) is the GCRL objective (Eq. (2) ) with reward r ( s ; g ) and C : = E g ∼ p
Revised: The key challenge with optimizing this objective offline is that we cannot sample from d π ( s ; g ) .

Original: ( g )

Original: [log Z ( g )] .

Original: See Appendix E.1 for proof.

Original: This proposition states that, for any choice of reward r ( s ; g ) , solving the

Original: GCRL problem with a maximum state-entropy regularization is equivalent to optimizing for the goalconditioned state-occupancy matching objective with target distribution p ( s ; g ) :=

Original: e r ( s ; g ) Z ( g ) . Now, thekey challenge with optimizing this objective offline is that we cannot sample from d π ( s ; g ) . To addressthis issue,following [32], we first derive an offline lower bound involving an f -divergence (see Appendix A for definition) regularization term, which subsequently enables solving this optimizationproblem via its dual using purely offline data:
Revised: To address this issue, we first derive an offlinelower bound involving an f -divergence (see Appendix A for definition) regularization term, whichsubsequently enables solving this optimization problem via its dual using purely offline data:

Original: Proposition 4.2.
Revised: Proposition 4.1.

Original: Assume 2 for all g in support of p ( g ) , ∀ s, d O ( s ; g ) > 0 if p ( s ; g ) > 0 .
Revised: Assume 2 for all g in support of p ( g ) , ∀ s, d O ( s ; g ) > 0 if p ( s ; g ) > 0 .

Original: Then, for anyf -divergence that upper bounds the KL-divergence,
Revised: Then, for anyf -divergence that upper bounds the KL-divergence,

Original: [Equation]
Revised: [Equation]

Original: The RHS of (6) can be understood as an f -divergence regularized GCRL objective with rewardfunction r ( s ; g )
Revised: The RHS of (5) can be understood as an f -divergence regularized GCRL objective with rewardfunction r ( s ; g )

Original: = log p ( s ; g )
Revised: = log p ( s ; g )

Original: d O ( s ; g ) .
Revised: d O ( s ; g ) .

Original: Intuitively, this reward encourages visiting states that occur more oftenin the “expert” state distribution p ( s ; g ) than in the offline dataset, and the f -divergence regularizationthen ensures that the learned policy is supported by the offline dataset.
Revised: Intuitively, this reward encourages visiting states that occur more oftenin the “expert” state distribution p ( s ; g ) than in the offline dataset, and the f -divergence regularizationthen ensures that the learned policy is supported by the offline dataset.

Original: This choice of reward functioncan be estimated in practice by training a discriminator [15] c : S × G → (0 , 1) using the offlinedata: 
Revised: This choice of reward functioncan be estimated in practice by training a discriminator [15] c : S × G → (0 , 1) using the offlinedata: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can in fact obtain a looser lower bound that does not require training a discriminator:
Revised: We can in fact obtain a looser lower bound that does not require training a discriminator:

Original: [Equation]
Revised: [Equation]

Original: Because p ( s ; g ) ∝ e r ( s ; g ) , we may substitute r ( s ; g ) (when the offline dataset contains reward labels)for log p ( s ; g )  and bypass having totrain a discriminator for reward;
Revised: Here, the reward function is r ( s ; g ) = log p ( s ; g ) , which encompasses standard GCRL rewardfunctions [37] and is related to a probabilistic interpretation of GCRL; see Appendix E.1for detail.

Original: Now, for either choice of lower bound, we may pose the optimization problem with respect to validchoices of state-action occupancies directly, introducing the Bellman flow constraint (4):
Revised: Now, for either choice of lower bound, we may pose the optimization problem with respect to validchoices of state-action occupancies directly, introducing the Bellman flow constraint (4):

Original: [Equation]
Revised: [Equation]

Original: This reformulation does not solve the fundamental problem that (9) still requires sampling fromd ( s ; g ) ; however, it has now written the problem in a way amenable to simplification using toolsfrom convex analysis.
Revised: This reformulation does not solve the fundamental problem that (8) still requires sampling fromd ( s ; g ) ; however, it has now written the problem in a way amenable to simplification using toolsfrom convex analysis.

Original: Now, we show that its dual problem can be reduced to an unconstrainedminimization problem over the dual variables which serve the role of a value function; importantly,the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution:
Revised: Now, we show that its dual problem can be reduced to an unconstrainedminimization problem over the dual variables which serve the role of a value function; importantly,the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution:

Original: Proposition 4.3.
Revised: Proposition 4.2.

Original: The dual problem to (9) is
Revised: The dual problem to (8) is

Original: [Equation]
Revised: [Equation]

Original: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and
Revised: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and

Original: T V ( s, a ; g ) =
Revised: T V ( s, a ; g ) =

Original: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .
Revised: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .

Original: Given the optimal V ∗ , the primal optimal d ∗ satisfies:
Revised: Given the optimal V ∗ , the primal optimal d ∗ satisfies:

Original: [Equation]
Revised: [Equation]

Original: A proof is given in Appendix B. Crucially, as neither expectation in (10) depends on samples fromd , this objective can be estimated entirely using offline data, making it suitable for offline GCRL.
Revised: A proof is given in Appendix B. Crucially, as neither expectation in (9) depends on samples fromd , this objective can be estimated entirely using offline data, making it suitable for offline GCRL.

Original: For tabular MDPs, we show that for a suitable choice of f -divergence, the optimal V ∗ in fact admitsclosed-form solutions; see Appendix D for details.
Revised: For tabular MDPs, we show that for a suitable choice of f -divergence, the optimal V ∗ in fact admitsclosed-form solutions; see Appendix D for details.

Original: In the continuous control setting, we can optimize(10) using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.
Revised: In the continuous control setting, we can optimize(9) using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.

Original: Then, once we have obtained the optimal (resp.
Revised: Then, once we have obtained the optimal (resp.

Original: converged) V ∗ , we propose learning the policy viathe following supervised regression update:
Revised: converged) V ∗ , we propose learning the policy viathe following supervised regression update:

Original: [Equation]
Revised: [Equation]

Original: We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluatedat the dual optimal advantage , r ( s ; g ) + γ T V ∗ ( s, a ; g ) − V ∗ ( s ; g ) ; we refer to this weighting termas f -advantage .
Revised: We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluatedat the dual optimal advantage , r ( s ; g ) + γ T V ∗ ( s, a ; g ) − V ∗ ( s ; g ) ; we refer to this weighting termas f -advantage .

Original: Hence, we name our overall method Goal-conditioned f -Advantage Regression(GoFAR) ; an abbreviated high-level pseudocode is provided in Algorithm 1 and a detailed versionis provided in Appendix C.
Revised: Hence, we name our overall method Goal-conditioned f -Advantage Regression(GoFAR) ; an abbreviated high-level pseudocode is provided in Algorithm 1 and a detailed versionis provided in Appendix C.

Original: In practice, we implement GoFAR with χ 2 -divergence, a choice off -divergence that is stable for off-policy optimization [49, 32, 26]; see Appendix C for details.1: (Optionally) Train discriminator-based reward (7) 2: Train (optimal) dual value function V ∗ ( s ; g ) (10) 3: Train policy π via f -Advantage Regression (12)
Revised: In practice, we implement GoFAR with χ 2 -divergence, a choice off -divergence that is stable for off-policy optimization [50, 32, 26]; see Appendix C for details.

Original: Note that (12) forgos directly minimizing (1) offline, which has been foundto suffer from training instability [21].
Revised: Note that (11) forgos directly minimizing (1) offline, which has been foundto suffer from training instability [21].

Original: Instead, it naturally incorporates theprimal-dual optimal solutions in a regression loss.
Revised: Instead, it naturally incorporates theprimal-dual optimal solutions in a regression loss.

Original: Now, we will show thatthis policy objective has several theoretical and practical benefits for offline GCRL that make it particularly appealing.
Revised: Now, we will show thatthis policy objective has several theoretical and practical benefits for offline GCRL that make it particularly appealing.

Original: 4.2 Optimal Goal-Weighting Property
Revised: 4.2 Optimal Goal-Weighting Property

Original: We show that optimizing (12) automatically obtains the optimal goal-weighting distribution .
Revised: We show that optimizing (11) automatically obtains the optimal goal-weighting distribution .

Original: That is,
Revised: That is,

Original: GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (9).
Revised: GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (8).

Original: In particular, this property is achieved without any explicit hindsight relabeling (see Appendix A fora technical definition), a mechanism that prior works heavily depend on.To this end, we first definep ( g | s, a ) to be the conditional distribution of goals in the offline dataset conditioned on state-actionpair ( s, a ) .
Revised: In particular, this property is achieved without any explicit hindsight relabeling (see Appendix A for atechnical definition), a mechanism that prior works heavily depend on.

Original: Then, according to Bayes rule, we have that
Revised: To this end, we first note thatby Bayes rule, p ( g )

Original: [Equation]
Revised: d O ( s, a ; g ) = d O ( s, a ) p ( g | s, a ) .

Original: Using this equality, we can rewrite the policy objective (12) as follows:
Revised: Then, we can rewrite the policy objective (11)as follows:

Original: [Equation]
Revised: [Equation]

Original: 15) where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Thus, we see that GoFAR’s f -advantage weighting scheme is equivalent to performing supervisedpolicy regression where goals are sampled from ˜ p
Revised: Thus, we see that GoFAR’s f -advantage weighting scheme is equivalent to performing supervisedpolicy regression where goals are sampled from ˜ p

Original: ( g | s, a ) .
Revised: ( g | s, a ) .

Original: Now, combining (16) and Bayes rulegives
Revised: Now, combining (14) and Bayes rulegives

Original: [Equation]
Revised: [Equation]

Original: Thus, we can replace the nested expectations in (15) and obtain that GoFAR policy update amountsto supervised regression of the state-action occupancy distribution of the optimal policy to theregularized GCRL problem (9):
Revised: Thus, we can replace the nested expectations in (13) and obtain that GoFAR policy update amountsto supervised regression of the state-action occupancy distribution of the optimal policy to theregularized GCRL problem (8): [Equation]

Original: [Equation]
Revised: [Equation]

Original: This derivation makes clear GoFAR’s connection with the hindsight goal relabeling mechanism [3]that is ubiquitous in GCRL:
Revised: This derivation makes clear GoFAR’s connection with the hindsight goal relabeling mechanism [3]that is ubiquitous in GCRL:

Original: GoFAR automatically performs the optimal goal-weighting policy updatewithout any explicit goal relabeling.
Revised: GoFAR automatically performs the optimal goal-weighting policy updatewithout any explicit goal relabeling.

Original: Furthermore, our derivation also suggests why hindsight relabeling is sub-optimal without further assumption on the reward function [10]: it heuristically chooses˜ p ( g | s, a ) to be the empirical trajectory-wise future achieved goal distribution (i.e., HER), whichgenerally does not coincide with the goals that the optimal policy would reach; see Appendix E.

Revised: Furthermore, our derivation also suggests why hindsight relabeling is sub-optimal without further assumption on the reward function [10]: it heuristically chooses˜ p ( g | s, a ) to be the empirical trajectory-wise future achieved goal distribution (i.e., HER), whichgenerally does not coincide with the goals that the optimal policy would reach; see Appendix E.

Original: forfurther discussion.
Revised: forfurther discussion.

Original: 4.3 Uninterleaved Optimization and Performance Guarantee
Revised: 4.3 Uninterleaved Optimization and Performance Guarantee

Original: An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the valuenetwork and the policy network.
Revised: An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the valuenetwork and the policy network.

Original: This can be observed by noting that GoFAR’s advantage term (11)is computed using V ∗ , the optimal solution of the dual problem.
Revised: This can be observed by noting that GoFAR’s advantage term (10)is computed using V ∗ , the optimal solution of the dual problem.

Original: This has the practical significanceof disentangling the value-function update (10) from the policy update (12), as we do not need tooptimize the latter until the former has converged.
Revised: This has the practical significanceof disentangling the value-function update (9) from the policy update (11), as we do not need tooptimize the latter until the former has converged.

Original: This disentanglement is in sharp contrast to prior
Revised: This disentanglement is in sharp contrast to prior

Original: GCRL works [47, 6, 11, 3], which typically involve alternating updates to the critic Q-network andthe policy network, a training procedure that has found to be unstable in the offline setting [23].
Revised: GCRL works [48, 6, 11, 3], which typically involve alternating updates to the critic Q-network andthe policy network, a training procedure that has found to be unstable in the offline setting [23].

Original: Thisis unavoidable for prior works because their advantage functions are estimated using the Q-estimateof the current policy, whereas our advantage term naturally falls out from  primal-dual optimality.
Revised: Thisis unavoidable for prior works because their advantage functions are estimated using the Q-estimateof the current policy, whereas our advantage term naturally falls out from the primal-dual optimalitycondition.

Original: The uninterleaved and relabeling-free nature of GoFAR also allows us to derive strong performanceguarantees.
Revised: The uninterleaved and relabeling-free nature of GoFAR also allows us to derive strong performanceguarantees.

Original: Because V ∗ is fixed in (12), this policy objective amounts to a weighted supervisedlearning problem.
Revised: Because V ∗ is fixed in (11), this policy objective amounts to a weighted supervisedlearning problem.

Original: Therefore, we can extend and adapt mature theoretical results for analyzing
Revised: Therefore, we can extend and adapt mature theoretical results for analyzing

Original: Behavior Cloning [1, 42, 46] as well as finite sample error guarantees for weighted regression [7] toobtain statistical guarantees on GoFAR’s performance with respect to the optimal π ∗ for (9):
Revised: Behavior Cloning [1, 43, 47] as well as finite sample error guarantees for weighted regression [7] toobtain statistical guarantees on GoFAR’s performance with respect to the optimal π ∗ for (8):

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Assume sup s,a,g d ∗ ( s,a ; g )
Revised: Assume sup s,a,g d ∗ ( s,a ; g )

Original: d O ( s,a ; g ) ≤ M and sup | r ( s, g )
Revised: d O ( s,a ; g ) ≤ M and sup | r ( s, g )

Original: | ≤ R max .
Revised: | ≤ R max .

Original: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .
Revised: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .

Original: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,

Original: GoFAR (18) will return a policy ˆ π such that:
Revised: GoFAR (16) will return a policy ˆ π such that:

Original: [Equation]
Revised: [Equation]

Original: Notably, the error shrinks as the size of the offline data N increases, requiring no dependency onaccess to data from the “expert” distribution d ∗ .
Revised: Notably, the error shrinks as the size of the offline data N increases, requiring no dependency onaccess to data from the “expert” distribution d ∗ .

Original: This provides a theoretical basis for GoFAR’sempirical scalability as we are guaranteed to obtain good results when the offline data becomes moreexpansive.
Revised: This provides a theoretical basis for GoFAR’sempirical scalability as we are guaranteed to obtain good results when the offline data becomes moreexpansive.

Original: In contrast, prior regression-based GCRL methods cannot be easily reduced to a simpleweighted regression with respect to the desired goal distribution p ( g ) , so they only obtain weakerresults under stronger assumptions on the offline data (e.g, full state-space coverage) as well as thepolicy; see Appendix E.3 for discussion.
Revised: In contrast, prior regression-based GCRL methods cannot be easily reduced to a simpleweighted regression with respect to the desired goal distribution p ( g ) , so they only obtain weakerresults under stronger assumptions on the offline data (e.g, full state-space coverage) as well as thepolicy; see Appendix E.3 for discussion.

Original: 4.4 Goal-Conditioned Planning
Revised: 4.4 Goal-Conditioned Planning

Original: Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zeroshot transfer to other domains of the same task.
Revised: Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zeroshot transfer to other domains of the same task.

Original: The key insight is that GoFAR’s value functionobjective (10) does not depend on actions assuming deterministic transitions.
Revised: The key insight is that GoFAR’s value functionobjective (9) does not depend on actions assuming deterministic transitions.

Original: This is because givena transition ( s, a, s ′ , g ) ∼ d
Revised: This is because givena transition ( s, a, s ′ , g ) ∼ d

Original: O , T V ( s, a ; g ) =
Revised: O , T V ( s, a ; g ) =

Original: V ( s ′ ; g ) , so we can rewrite the second term in (10)as E ( s,a,s ′ ,g )
Revised: V ( s ′ ; g ) , so we can rewrite the second term in (9)as E ( s,a,s ′ ,g )

Original: ∼ d
Revised: ∼ d

Original: O
Revised: O

Original: [ f ⋆ ( r ( s, g ) + γV ( s ′ ; g ) − V
Revised: [ f ⋆ ( r ( s, g ) + γV ( s ′ ; g ) − V

Original: s ; g ))] , which does not depend on the action.
Revised: s ; g ))] , which does not depend on the action.

Original: This property enables us to learn a goal-centric value function that is independent of the agent’s statespace because the agent’s actions are not relevant.
Revised: Thisproperty enables us to learn a goal-centric value function that is independent of the agent’s statespace because the agent’s actions are not relevant.

Original: Specifically, we propose the following objective:
Revised: Specifically, we propose the following objective:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can think of this objective as learning a value function with the inductive bias that the firstoperation transforms the state input to the goal space via ϕ .
Revised: We can think of this objective as learning a value function with the inductive bias that the firstoperation transforms the state input to the goal space via ϕ .

Original: Since V is now independent of the agent,we can use f -advantage regression to instead learn a goal-conditioned planner : 
Revised: Since V is now independent of the agent [Equation] we can use f -advantage regression to instead learn a goal-conditioned planner : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where π now outputs the next subgoal ϕ ( s ′ ) conditioned on the current achieved goal ϕ ( s ) and thedesired goal g .
Revised: where π now outputs the next subgoal ϕ ( s ′ ) conditioned on the current achieved goal ϕ ( s ) andthe desired goal g .

Original: In our experiments, we show how this planner can achieve hierarchical control throughzero-shot transfer subgoal plans to a new target domain.
Revised: In our experiments, we show how this planner can achieve hierarchical controlthrough zero-shot transfer subgoal plans to a new target domain that shares the same goal space andby empowering a target-domain low-level controller that is only effective for nearby goals to reachdistant goals.

Original:  Experiments
Revised:  Experiments

Original: We pose the following questions and provide affirmative answers in our experiments:
Revised: We pose the following questions and provide affirmative answers in our experiments:

Original: Is GoFAR effective for offline GCRL?
Revised: Is GoFAR effective for offline GCRL?

Original: What components are important for performance?2.
Revised: What components are important for performance?2.

Original: Is GoFAR more robust to stochastic environments than hindsight relabeling methods?3.
Revised: Is GoFAR more robust to stochastic environments than hindsight relabeling methods?3.

Original: Can GoFAR be applied to a real-robotics system?4.
Revised: Can GoFAR be applied to a real-robotics system?4.

Original: Can GoFAR learn a goal-conditioned planner for zero-shot cross-embodiment transfer?5.

Revised: Can GoFAR learn a goal-conditioned planner for zero-shot cross-embodiment transfer?5.

Original: Offline GCRL
Revised: Offline GCRL

Original: Tasks.
Revised: Tasks.

Original: In our simulation experiments, we consider six distinct environments.
Revised: In our simulation experiments, we consider six distinct environments.

Original: They include fourrobot manipulation environments[39]: FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide,and two dexterous manipulation environments: HandReach [39], and D’ClawTurn [2].
Revised: They include four robotmanipulation environments[40]: FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide, andtwo dexterous manipulation environments: HandReach [40], and D’ClawTurn [2].

Original: All tasksusesparse reward, and their respective goal distributions are defined over valid configurations in eitherthe robot space or object space, depending on whether the task involves object manipulation.
Revised: The dexterousmanipulation tasks are particularly challenging due to the high-dimensional action space that makescredit assignment difficult.

Original: Theoffline dataset for each task is collected by either a random policy or a mixture of 90% random policyand 10% expert policy [32, 21], depending on whether random data provides enough coverage of thedesired goal distribution.
Revised: The offline dataset for each task is collected by either a random policy ora mixture of 90% random policy and 10% expert policy [32, 21], depending on whether random dataprovides enough coverage of the desired goal distribution.

Original: The first five tasks and their datasets are taken from [47]. See Appendix G.3for dataset details and Appendix F for detailed task descriptions and figures.
Revised: See Appendix G.3 for dataset details and Appendix F for detailed task descriptions and figures.

Original: [Table] Algorithms.
Revised: Algorithms.

Original: We compare tostate-of-art offline GCRL algorithms, consisting of bothregression-based and actor-criticmethods.
Revised: We compare to state-of-art offline GCRL algorithms, consisting of both regression-basedand actor-critic methods.

Original: The regression-basedmethods are: (1) GCSL [14],which incorporates hindsight relabeling in conjunction with behavior cloning to clone actionsthat lead to a specified goal, and (2) WGCSL [47], which improves upon GCSL by incorporatingdiscount factor and advantage weighting into the supervised policy learning update.
Revised: The regression-based methods are: (1) GCSL [14], which incorporateshindsight relabeling in conjunction with behavior cloning to clone actions that lead to a specifiedgoal, and (2) WGCSL [48], which improves upon GCSL by incorporating discount factor andadvantage weighting into the supervised policy learning update.

Original: The actor-criticmethods are (1) DDPG [3], which adapts DDPG [29] to the goal-conditioned setting by incorporating hindsight relabeling, and
Revised: The actor-critic methods are (1) DDPG [3], which adapts DDPG [29] to the goal-conditioned setting by incorporating hindsightrelabeling, and

Original: (2) ActionableModel (AM) [6], which incorporates conservative Q-Learning [24] as well as goal-chaining on top of an actor-critic method.
Revised: (2) ActionableModel (AM) [6], which incorporates conservative Q-Learning [24] aswell as goal-chaining on top of an actor-critic method; in this work, we use DDPG as the actor-criticmethod for AM.

Original: We use tuned hyperparameters for each baseline on all tasks; in particular, we search the best HERratio from { 0 . 2 , 0 . 5 , 1 . 0 } for eachbaseline on each task separately and report the best-performingone.
Revised: We use tuned hyperparameters for each baseline on all tasks and additionally search the best HERratio from { 0 . 2 , 0 . 5 , 1 . 0 } for each task separately.

Original: For GoFAR, we use identical hyperparameters as WGCSL for the shared network componentsand do not tune further.
Revised: For GoFAR, we use identical hyperparameters as WGCSL for the shared network components and do not tune further.

Original: We train each method for 10 seeds, and each training run uses 400k minibatchupdates of size 512.
Revised: We train each method for 3seeds, and each training run uses 400k minibatch updates of size 512.

Original: Complete architecture and hyperparameter table as well as additional trainingdetails are provided in Appendix G. 
Revised: Complete architecture andhyperparameter table as well as additional training details are provided in Appendix G. [Table]

Original: Evaluations and Results.
Revised: Evaluations and Results.

Original: We report the discounted return using the sparse binary task reward.
Revised: Wereport the discounted return using the sparse binary task reward.

Original: Thismetric rewards algorithms that reach goals as fast as possible and stay in the goal region thereafter. In
Revised: This metric rewards algorithmsthat reach goals as fast as possible and stay in the goal regionthereafter.

Original: Appendix H, we also report the final success rate using the samesparse binary criterion.
Revised: Note that this metricfavors the baseline algorithms asthey use the same binary rewardlabel and therefore have knowledge of the success criterion thatis used for evaluation. In contrast, GoFAR’s reward signal comes from its discriminator.

Original: Because thetask reward is binary, these metrics do not take account into how precisely a goal is being reached.
Revised: Because thetask reward is binary, this metric does not take account into how precisely a goal is being reached.

Original: Therefore, we additionally 
Revised: Therefore, we additionally report the final distance to goal in Appendix H.

Original: The full discounted return results  are shown in Table 1;( ⋆ ) indicates statistically significant improvement over thesecond best method under a two-sample t -test. The fraction inside () indicates thebest HER rate for each baseline.
Revised: The full discounted return results 3 are shown in Table 1; the fraction inside () indicates the best HERrate for each baseline.

Original: As shown, GoFAR attains the bestoverall performance across sixtasks, and the results are statistically significant on three tasks, including the two more challengingdexterous manipulation tasks.
Revised: Aside from the simplest FetchReach, on which actor-critic methods slightlyoutperform GoFAR, GoFAR attain the best performance on all tasks and is significantly better on themore challenging dexterous manipulation tasks.

Original: In Table 6 in Appendix H, wefind GoFAR is also superior on the finaldistance metric.{+final
+}
Revised: In Table 6 in Appendix H, we also report the {+final
+}

Revised: The error bar version is included in Appendix H. distance metric, on which GoFAR is once again superior.

Original: In other words, GoFAR reaches the goals fastest and most precisely.
Revised: In other words, GoFAR reaches the goalsfastest and most precisely.

Original: Ablations.
Revised: Ablations.

Original: Recall that GoFAR achieves optimal goal-weighting, and does not require the HER heuristic that has been key to prior GCRL approaches.
Revised: Recall that GoFAR achieves optimal goal-weighting, and does not require the HER heuristic that has been key to prior GCRL approaches.

Original: We now experimentally confirm that the baselines arenot performant without HER.
Revised: We now experimentally confirm that the baselines arenot performant without HER.

Original: Conversely, we investigate whether HER can help GoFAR.
Revised: Conversely, we investigate whether HER can help GoFAR.

Original: Additionally,we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in (8).
Revised: Additionally,we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in (7).

Original: Finally, we replace χ 2 -divergencewith KL to understand whether thef -divergence lower bound in (6) isnecessary.
Revised: Finally, we replace χ 2 -divergencewith KL to understand whether thef -divergence lower bound in (5) isnecessary.

Original: The results are shown in
Revised: The results are shown in

Original: Figure 3 (task-specific breakdown isincluded in Appendix H).
Revised: Figure 3 (task-specific breakdown isincluded in Appendix H).

Original: Comparedto their original performances (colored dash lines), baselines without
Revised: Comparedto their original performances (colored dash lines), baselines without

Original: HER suffer severely, especially GCSLand WGCSL: both these methods infact require 100% relabeling and relysolely on HER for positive learningsignals.
Revised: HER suffer severely, especially GCSLand WGCSL: both these methods infact require 100% relabeling and relysolely on HER for positive learningsignals.

Original: In sharp contrast, GoFAR isalso a regression-based method, yet itperforms very well without HER and its performance is unaffected by adding HER.
Revised: In sharp contrast, GoFAR isalso a regression-based method, yet itperforms very well without HER and its performance is unaffected by adding HER.

Original: These findingsare consistent with the theoretical results of Section 4.2.
Revised: These findingsare consistent with the theoretical results of Section 4.2.

Original: GoFAR (Binary) performs slightly worsethan GoFAR, which is to be expected due to the looser bound; however, it is still better than allbaselines, which use the same binary reward and are aided by HER.
Revised: GoFAR (Binary) performs slightly worsethan GoFAR, which is to be expected due to the looser bound; however, it is still better than allbaselines, which use the same binary reward and are aided by HER.

Original: Again, this ablation highlightsthe merit of our optimization approach.
Revised: Again, this ablation highlightsthe merit of our optimization approach.

Original: Finally, GoFAR (KL) suffers from numerical instability andfails to learn, showing that our general f -divergence lower bound (6) is not only of theoretical valuebut also practically significant.
Revised: Finally, GoFAR (KL) suffers from numerical instability andfails to learn, showing that our general f -divergence lower bound (5) is not only of theoretical valuebut also practically significant.

Original: 5.2 Robustness in Stochastic Offline GCRL Settings
Revised: 5.2 Robustness in Stochastic Offline GCRL Settings

Original:  For this experiment, we create noisy variants ofthe FetchReach environment by adding varyinglevels of white Gaussian noise to policy actionoutputs.
Revised: [Figure] For this experiment, we create noisy variants ofthe FetchReach environment by adding varyinglevels of white Gaussian noise to policy actionoutputs.

Original: FetchReach is well-suited for this experiment because all baselines attain satisfactoryperformance and even outperform GoFAR in Table 1.
Revised: FetchReach is well-suited for this experiment because all baselines attain satisfactoryperformance and even outperform GoFAR in Table 1.

Original: Furthermore, to isolate the effect of noisyactions, we use GoFAR with binary reward sothat all methods now use the same reward input and perform comparatively without addednoise.
Revised: Furthermore, to isolate the effect of noisyactions, we use GoFAR with binary reward sothat all methods now use the same reward input and perform comparatively without addednoise.

Original: We consider zero-mean Gaussian noiseof σ value from 0 .
Revised: We consider zero-mean Gaussian noiseof σ value from 0 .

Original: 5 , 1 , and 1 . 5 .
Revised: 5 , 1 , and 1 . 5 .

Original: For each noise level, we re-collect offline data by executing randomactions in the respective noisy environment and train on these noisy offline random data.
Revised: For each noise level, we re-collect offline data by executing randomactions in the respective noisy environment and train on these noisy offline random data.

Original: The average discounted returns over differentnoise levels are illustrated in Figure 4.
Revised: The average discounted returns over differentnoise levels are illustrated in Figure 4.

Original: As shown,at noise level 0 .
Revised: As shown,at noise level 0 .

Original: 5 , while GoFAR’s performanceis not effected, all baselines already exhibitdegraded performance.
Revised: 5 , while GoFAR’s performanceis not effected, all baselines already exhibitdegraded performance.

Original: In particular, DDPGsharply degrades, underperforming GoFAR despite better original performance.
Revised: In particular, DDPGsharply degrades, underperforming GoFAR despite better original performance.

Original: At 1 . 0 , thegap continues to widen, and AM notably collapses, highlighting its sensitivity to noise.
Revised: At 1 . 0 , thegap continues to widen, and AM notably collapses, highlighting its sensitivity to noise.

Original: This is expected as AM’s relabeling mechanism alsosamples future goals from other trajectories and labels the selected goals with their current Q-valueestimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity,thus contributing to the instability observed in this experiment.
Revised: This is expected as AM’s relabeling mechanism alsosamples future goals from other trajectories and labels the selected goals with their current Q-valueestimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity,thus contributing to the instability observed in this experiment.

Original: It is only at noise level 1 . 5 that
Revised: It is only at noise level 1 . 5 that

Original: GoFAR’s performance degradation becomes comparable to those of other methods.
Revised: GoFAR’s performance degradation becomes comparable to those of other methods.

Original: Thus, GoFAR’s relabeling-free optimization indeed confers greater robustness to environmental stochasticity; incontrast, all baselines suffer from “false positive” relabeled goals due to such disturbances.
Revised: Thus, GoFAR’srelabeling-free optimization indeed confers greater robustness to environmental stochasticity; incontrast, all baselines suffer from “false positive” relabeled goals due to such disturbances.

Original: 5.3 Real-World Robotic Dexterous Manipulation
Revised: 5.3 Real-World Robotic Dexterous Manipulation

Original: Now, we evaluate on a real D’Claw [2] dexterous manipulation robot (Figure 5 (left)).
Revised: Now, we evaluate on a real D’Claw [2] dexterous manipulation robot (Figure 5 (left)).

Original: The task is tohave the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix Ffor detail.
Revised: The task is tohave the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix Ffor detail.

Original: The offline dataset contains 400K transitions, collected by executing purely random actionsin the environment, which provides sufficient coverage of the goal space.
Revised: The offline dataset contains 400K transitions, collected by executing purely random actionsin the environment, which provides sufficient coverage of the goal space.

Original: Compared to the simulated
Revised: Compared to the simulated

Original: D’Claw, an additional challenge is the inherent stochasticity on a real robot system due to imperfectactuation and hardware wear and tear.
Revised: D’Claw, an additional challenge is the inherent stochasticity on a real robot system due to imperfectactuation and hardware wear and tear.

Original: Given our conclusion in the stochastic environment experimentabove, we should expect GoFAR to perform better.
Revised: Given our conclusion in the stochastic environment experimentabove, we should expect GoFAR to perform better.

Original: The training curve of final distances averaged over 3 seedsis shown in Figure 6.
Revised: The training curve of final distances averagedover 3 seeds is shown in Figure 6.

Original: We see that GoFAR exhibits smoothprogress over training and significantly outperforms allbaselines.
Revised: We see that GoFAR exhibits smooth progress over trainingand significantly outperforms all baselines.

Original: At the end of training, GoFAR is able to rotatethe valve on average within 10 degrees of the specifiedgoal angle.
Revised: Atthe end of training, GoFAR is able to rotatethe valve on average within 10 degrees of thespecified goal angle.

Original: In contrast, most baselines do not make anyprogress on this difficult task and learns policies that do notmanage to turn the valve at all.
Revised: In contrast, most baselinesdo not make any progress on this difficult taskand learns policies that do not manage to turnthe valve at all.

Original: AM is the only baseline thatlearns a non-trivial policy; however, its training fluctuatessignificantly, and the learned policy often executes extremeactions during evaluation that fail to reach the desired goal.
Revised: AM is the only baseline thatlearns a non-trivial policy; however, its trainingfluctuates significantly, and the learned policyoften executes extreme actions during evaluationthat fail to reach the desired goal.

Original: We provide qualitative analysis in Appendix H.3, and videos are included in the supplement.
Revised: We provide qualitative analysis in Appendix H.3, and the fullvideos are included in the supplementary material.

Original: 5.4 Zero-Shot Transfer Across Robots
Revised: 5.4 Zero-Shot Transfer Across Robots

Original: [Table]
Revised: [Table]

Original: For this experiment, we consider zero-shot transfer fromthe FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Pandarobot [12]; the environments are illustrated in Figure 5.
Revised: For this experiment, we consider zero-shot transfer fromthe FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Pandarobot [12]; the environments are illustrated in Figure 5.

Original: In the target domain, we assume a pre-programmed orlearned low-level goal-reaching controller π low ( a | s, g )that can push objects over short distances but fails for longer tasks.
Revised: In the target domain, we assume a pre-programmed orlearned low-level goal-reaching controller π low ( a | s, g )that can push objects over short distances but fails for longer tasks.

Original: For these distant goals, we trainthe goal-conditioned GoFAR value function V ∗ ( ϕ ( s ); g ) and planner π ( ϕ ( s ′ )
Revised: For these distant goals, we trainthe goal-conditioned GoFAR value function V ∗ ( ϕ ( s ); g ) and planner π ( ϕ ( s ′ )

Original: | ϕ ( s ) , g ) accordingto (20) and (21), to set goals for π low .
Revised: | ϕ ( s ) , g ) accordingto (18) and (19), to set goals for π low .

Original: We use the same offline FetchPush data as in Section 5.1.
Revised: We use the same offline FetchPush data as in Section 5.1.

Original: Wecompare two approaches: (1) naively using the Low-Level Controller , and (2) using the GoFARplanner to guide the low-level controller ( GoFAR Hierarchical Controller ).
Revised: Wecompare two approaches: (1) naively using the Low-Level Controller , and (2) using the GoFARplanner to guide the low-level controller ( GoFAR Hierarchical Controller ).

Original: See Appendix G.4for additional details.
Revised: See Appendix G.4for additional details.

Original: We report the success rate over 100 random distant test goals in Table 2.
Revised: We report the success rate over 100 random distant test goals in Table 2.

Original: Wesee that naively commanding the low-level controller with these distant goals indeed results in verylow success rate.
Revised: Wesee that naively commanding the low-level controller with these distant goals indeed results in verylow success rate.

Original: When the controller is augmented with GoFAR planner, the success rate nearlydoubles.
Revised: When the controller is augmented with GoFAR planner, the success rate nearlydoubles.

Original: Note that our planner itself is much superior: if all subgoals are perfectly reached, GoFAR
Revised: Note that our planner itself is much superior: if all subgoals are perfectly reached, GoFAR

Original: Planner (Oracle) achieves 84% success.
Revised: Planner (Oracle) achieves 84% success.

Original: See Appendix G.4 for additional experimental details.
Revised: See Appendix G.4 for additional experimental details.

Original: These experiments validate GoFAR’s ability to zero-shot transferring subgoal plans to enhance thecapabability of low-level controllers.
Revised: These experiments validate GoFAR’s ability to zero-shot transferring subgoal plans to enhance thecapabability of low-level controllers.

Original: Note that we have no access to any data from the target domain,and this planning capability naturally emerges from our training objectives and does not require anychange in the algorithm.
Revised: Note that we have no access to any data from the target domain,and this planning capability naturally emerges from our training objectives and does not require anychange in the algorithm.

Original: We show qualitative results in Appendix H.4 and videos in the supplement.
Revised: We show qualitative results in Appendix H.4 and videos in the supplement.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented GoFAR, a novel regression-based offline GCRL algorithm derived from a stateoccupancy matching perspective.
Revised: We have presented GoFAR, a novel regression-based offline GCRL algorithm derived from a stateoccupancy matching perspective.

Original: GoFAR is relabeling-free and enjoys uninterleaved training, makingit both theoretically and practically advantageous compared to prior state-of-art.
Revised: GoFAR is relabeling-free and enjoys uninterleaved training, makingit both theoretically and practically advantageous compared to prior state-of-art.

Original: Furthermore, GoFARsupports training a goal-conditioned planner with promising zero-shot transfer capability.
Revised: Furthermore, GoFARsupports training a goal-conditioned planner with promising zero-shot transfer capability.

Original: Throughextensive experiments, we have validated the practical utility of GoFAR in various challengingproblem settings, showing significant improvement over prior state-of-art.
Revised: Throughextensive experiments, we have validated the practical utility of GoFAR in various challengingproblem settings, showing significant improvement over prior state-of-art.

Original: We believe that GoFAR’sstrong theoretical foundation and empirical performance make it an ideal candidate for scalable skilllearning in the real world.
Revised: We believe that GoFAR’sstrong theoretical foundation and empirical performance make it an ideal candidate for scalable skilllearning in the real world.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We include the
Revised: [Yes] We include the

Original: code and data for our simulation experiments in the supplementary material.
Revised: code and data for our simulation experiments in the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Appendix G.
Revised: [Yes] See Appendix G.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Appendix H.
Revised: [Yes] See Appendix H.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix G.
Revised: [Yes] See Appendix G.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: We include videos in our supplementary material.
Revised: We include videos in our supplementary material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A] Part I
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: Appendix

Original: These appendices contain material supplementary to submission 7863: "Offline Goal-Conditioned 579 Reinforcement Learning via f -Advantage Regression"

Original: Table of Contents

Original: A Additional Technical Background

Original: A.1 f

Original: [Table]

Original: Divergence and Fenchel Duality . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Hindsight Goal-Relabeling . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [-B Proofs 
-]
Revised: [-B Proofs 
-]

Original: B.1 Proof of Proposition 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Proof of Proposition 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.

Original: [Table]

Original: of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table]

Original: GoFAR Technical Details

Original: C

Original: .1 Discriminator Training

Original: C

Original: .2 GoFAR with common f -Divergences

Original: C

Original: .3 Full Pseudocode

Original: D

Original: GoFAR for Tabular MDPs

Original: E Additional Technical Discussion

Original: [Table]

Original: .1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL

Original: E.2 Optimality Conditions for Hindsight Relabeling

Original: E.3 Theoretical Comparison to

Original: Prior Regression-based GCRL methods . . . . . . .

Original: [-F Task Descriptions 
-]
Revised: [-F Task Descriptions 
-]

Original: F.1 Fetch Tasks . .

Original: F.2 Hand Reach . .

Original: F.3 D’ClawTurn (Simulation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.4 D’ClawTurn (Real) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G

Original: Experimental

Original: [-Details 
-]
Revised: [-Details 
-]

Original: G.1 Baseline Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Architectures and Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.3 Offline GCRL Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.4 Zero-Shot Transfer Experiments . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H Additional Results

Original: H.1 Offline GCRL Full Results . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.2 Ablations . .

Original: H.3 Real-World Dexterous Manipulations . .

Original: [Table]

Original: A Additional Technical Background

Original: We provide some technical definitions that are needed in our proofs in Appendix B and discussionsof hindsight relabeling in Section 4.1 of the main text as well as Appendix E.2.

Original: A.1 f -Divergence and Fenchel Duality

Original: These definitions are adapted from [32, 34].

Original: Definition A.1 ( f -divergence) .

Original: For any continuous, convex function f and two probability distributions p, q ∈ ∆( X ) over a domain X , the f -divergence of p computed at q is defined as

Original: [Equation]

Original: Some common choices of f -divergence includes the KL-divergence and the χ 2 -divergence, whichcorresponds to choosing f ( x ) = x log x and f ( x ) = 12 ( x − 1) 2 , respectively.

Original: Definition A.2 (Fenchel conjugate) .

Original: Given a vector space X with inner-product ⟨· , ·⟩ , the Fenchelconjugate f ⋆ : X ⋆ → R of a convex and differentiable function f : Ω → R is

Original: [Equation]

Original: 23) and any maximizer x ∗ of f ⋆ ( y ) satisfies x ∗ = f ′ ⋆ ( y ) .

Original: For an f -divergence, under mild realizability assumptions [8] on f , the Fenchel conjugate of D f ( p ∥ q )at y : X → R is

Original: [Equation]

Original: [Equation]

Original: 25) and any maximizer p ∗ of D

Original: ⋆,f ( y ) satisfies p ∗ ( x ) =

Original: q ( x ) f ′ ⋆ ( y ( x )) .

Original: These optimality conditions canbe seen as extensions of the KKT-condition.

Original: A.2 Hindsight Goal-Relabeling

Original: We provide a mathematical formalism of hindsight goal relabeling [3].

Original: Definition A.3.

Original: Given a state s t from a trajectory τ = { s 0 , a 0 , r 0 , ..., s T ; g } , hindsight goalrelabeling is the goal-relabeling distribution

Original: [Equation]

Original: where q is some categorical distribution taking values in { ϕ ( s t ) , ..., ϕ ( s T ) } .

Original: That is, the relabeled goal is selected from some distribution goals that are reached in the future inthe same trajectory.

Original: The most canonical choice of q , known as hindsight experience replay (HER),selects q to be the uniform distribution.

Original: Once a goal ˜ g is chosen, the reward label is also re-computedusing the reward function assumed by the algorithm: r t := r ( s t , ˜ g ) .

Original: B Proofs

Original: In this section, we restate propositions and theorems in the paper and present their proofs.

Original: B.1 Proof of Proposition 4.

Original: Proposition B.1.

Original: Assume for all g in support of p ( g ) , ∀ s, d O ( s ; g ) >

Original: if p ( s ; g ) > 0 .

Original: Then, for anyf -divergence that upper bounds the KL-divergence,

Original: [Equation]

Original: Proof.

Original: We first present and prove some technical lemma needed to prove this result.

Original: The followinglemmas and proofs are adapted from [32]; in particular, we extend these known results to thegoal-conditioned setting.

Original: Lemma B.1.

Original: For any pair of valid occupancy distributions d 1 and d 2 , we have

Original: [Equation]

Original: Proof.

Original: This lemma hinges on proving the following lemma first

Original: [Equation]

Original: Lemma B

Original: [Equation]

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: Using this result

Original: [Equation]

Original: we can prove Lemma B

Original: [Equation]

Original: [Equation]

Original: we assume that d O ( s ; g ) > 0 whenever p ( s ; g ) > 0 .

Original: [Equation]

Original: where the last step follows from Lemma B.1.

Original: Then, for any D f ≥ D KL , we have that

Original: [Equation]

Original: Then, since E ( s,g ) ∼ d π ( s,g ) (cid:104) log

Original: [Equation]

Original: ≥ 0 , we also obtain the following looser bound:

Original: [Equation]

Original: B.2 Proof of Proposition 4.

Original: Proposition B.2.

Original: The dual problem to (9) is

Original: [Equation]

Original: where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and

Original: T V ( s, a ; g ) =

Original: E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .

Original: Given the optimal V ∗ , the primal optimal d ∗ satisfies:

Original: [Equation]

Original: Proof.

Original: We begin by writing the Lagrangian dual of the primal problem:

Original: [Equation]

Original: where p ( g ) V ( s ; g ) is the Lagrangian vector.

Original: Then, we note that

Original: [Equation]

Original: g

Original: [Equation]

Original: Using this, we can rewrite (35) as

Original: [Equation]

Original: And finally,

Original: [Equation]

Original: Now, we make the key observation that the inner maximization problem in (38) is in fact the Fenchelconjugate of D f ( d ( s, a, g ) ∥

Original: d O ( s, a, g )) at r ( s, g ) + γ T V ( s, a, g ) − V ( s, g ) .

Original: Therefore, we canreduce (38) to an unconstrained minimization problem over the dual variables

Original: [Equation]

Original: and consequently, we can relate the dual-optimal V ∗ to the primal-optimal d ∗ using Fenchel duality(see Appendix A:

Original: [Equation]

Original: as desired.

Original: B.3 Proof of Theorem

Original: Theorem B.3.

Original: Assume sup s,a,g d ∗ ( s,a ; g )

Original: d O ( s,a ; g ) ≤ M and sup | r ( s, g )

Original: | ≤ R max .

Original: Consider a policy classΠ : { S → ∆( A ) } such that π ∗ ∈ Π .

Original: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,

Original: GoFAR (18) will return a policy ˆ π such that:

Original: [Equation]

Original: N (41) where V π := E ( s,g ) ∼ ( µ 0 ,g )

Original: [ V ( s ; g )] .

Original: Proof.

Original: We begin by deriving an upper bound using the performance difference lemma [1]:

Original: [Equation]

Original: Then, using standard algebraic manipulations, we have:

Original: [Equation]

Original: Then, since sup s,a,g d ∗ ( s,a ; g ) d O ( s,a ; g ) ≤ M , we can use Hoeffding’s inequality with weighted empiricalloss [7] to obtain that:

Original: [Equation]

Original: C GoFAR Technical Details

Original: In this section, we provide additional technical details of GoFAR that are omitted in the main text.

Original: These include (1) detail of the GoFAR discriminator training, (2) mathematical expressions of GoFARspecialized to common f -Divergences, and (3) a full pseudocode.

Original: C.1 Discriminator Training

Original: Training the discriminator 7 in practice requires choosing p ( s ; g ) .

Original: For simplicity, we set p ( s ; g ) to bethe Dirac distribution centered at g : I ( ϕ ( s ) = g ) ; this precludes having to choose hyperparametersfor p ( s ; g ) .

Original: Once the discriminator has converged, we can retrieve the reward function r ( s ; g )

Original: = log p ( s ; g )

Original: d O ( s ; g )

Original: =− log (cid:16) 1 c ∗ ( s ; g ) − 1 (cid:17) , since c ∗ ( s ; g ) =

Original: d O ( s ; g ) p ( s ; g )+

Original: d O ( s ; g ) .

Original: C.2 GoFAR with common f -Divergences

Original: GoFAR requires choosing a f -divergence.

Original: Here, we specialize GoFAR to χ 2 -divergence as well as

Original: KL-divergence as examples.

Original: Our practical implementation uses χ 2 -divergence, which we found to besignificantly more stable than KL-divergence (see Section 5.1).

Original: Example 1 (GoFAR with χ 2 -divergence) .

Original: f ( x ) = 12 ( x − 1) 2 , and we can show that f ⋆ ( x ) =12 ( x + 1) 2 and f ′

Original: ⋆ ( x ) = x

Original: + 1 .

Original: Hence, the GoFAR objective amounts tomin

Original: [Equation]

Original: (45) and

Original: [Equation]

Original: [Equation]

Original: Example 2 (GoFAR with KL-divergence) .

Original: We have f ( x ) = x log x and that D

Original: ⋆,f ( y ) =

Original: log E

Original: x ∼ q [exp y ( x )] [4].

Original: Hence, the KL-divergence GoFAR objective ismin

Original: [Equation]

Original: [Equation]

Original: 47) and

Original: [Equation]

Original: Now, we provide the full pseudocode for GoFAR implemented using χ 2 -divergence in Algorithm 2.

Original: C.3 Full Pseudocode

Original: 1: Require : Offline dataset d O , choice of f -divergence f , choice of p ( s ; g ) 2: Randomly initialize discriminator c ψ , value function V θ , and policy π ϕ .

Original: Train Discriminator (Optional) 4: for number of discriminator iterations do 5:

Original: Sample minibatch { s id , g i } Ni =1 ∼ d O 6:

Original: Sample { s jg } Mj =1 ∼ p ( s ; g i ) ∀ i ∈ 1 . . .

Original: N 7:

Original: Discriminator objective: L c ( ψ ) = 1 N (cid:80) Ni

Original: =1 [log(1 − c ψ ( s jd , g i ))

Original: + 1 M (cid:80) Mj

Original: =1 [log c ψ ( s jg , g i )]] 8:

Original: Update c ψ using SGD: c ψ ← c ψ − α c ∇L c ( ψ ) 9: end for 10: //

Original: Train Dual Value Function 11: for number of value iterations do 12:

Original: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d O , { s i 0 }

Original: Mi =1 ∼ µ 0 , { g i 0 } Mi =1 ∼ d O 13: If discriminator, obtain reward: r ( s i t ; g i t ) = − log (cid:16) 1 c ψ ( s i t ; g i t ) − 1 (cid:17) ∀ i = 1 . . .
Revised: spent on participant compensation?

Original: N 14:

Original: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d O 15:

Original: Value objective: L V ( θ ) = 1 − γM (cid:80) Mi

Original: =1 [ V θ ( s i 0 ; g i 0 )]

Original: + 1 N (cid:80) Ni =1 (cid:2)

Original: f ⋆ ( r it + γV

Original: ( s it +1 ; g it ) − V

Original: ( s it ; g it )) (cid:3) 16:

Original: Update V θ using SGD: V θ ← V θ − α V ∇L V ( θ ) 17: end for 18: // Train Policy With f -Advantage Regression 19: for number of policy iterations do 20:

Original: Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d

Original: O 21:

Original: If discriminator, obtain reward: r ( s it ; g it )

Original: = − log (cid:16) 1 c ψ ( s it ; g it ) − 1 (cid:17) ∀ i = 1 . . .

Original: N 22:

Original: If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d

Original: O 23:

Original: Policy objective: L π ( ϕ ) =

Original: (cid:80) Ni =1 (cid:2)(cid:0)

Original: f ′ ⋆ (cid:0) r it + γV

Original: θ ( s it +1 ; g it ) − V θ

Original: ( s it ; g it (cid:1)(cid:1) log π ( a | s, g ) (cid:3) 24:

Original: Update π ϕ using SGD: π ϕ ← π ϕ − α π ∇L ( ϕ ) 25: end for

Original: D GoFAR for Tabular MDPs

Original: In Section 4.1 of the main text, we have stated that in tabular MDPs, GoFAR’s optimal dual valuefunction (10) admits closed-form solution when we choose χ 2 -divergence.

Original: Here, we provide aderivation of this result.

Original: Recall the dual problem (1)min

Original: [Equation]

Original: To derive a closed-form solution, we rewrite the problem in vectorized notation; we borrow ournotations from [32].

Original: We first augment the state-space by concatenating the state dimensions andthe goal dimensions so that the new state space ˜ S has dimension S + G .

Original: Then, the new transitionfunction, with slight abuse of notation, T (( s ′ , g ′ )

Original: | ( s, g ) , a ) =

Original: T ( s | s, a ) I ( g ′ =

Original: g ) ; the new initialstate distribution is thus µ 0 ( s, g )

Original: = µ 0 ( s ) p ( g ) .

Original: Therefore, ˜ T ∈ R | S

Original: ||

Original: G ||

Original: A |×| S || G | + and µ 0 ∈ R | S || G | + .

Original: We assume that the offline dataset D O is collected by a behavior policy π b .

Original: We constructa surrogate MDP ˆ M using maximum likelihood estimation; that is, ˆ T (( s ′ , g ′ )

Original: | ( s, g ) , a ) =

Original: n ( s,a,s ′ )

Original: n ( s,a ) I

Original: ( g ′ = g ) , and we impute ˆ T (( s ′ , g ) | ( s, g ) , a ) = 1 S when n ( s, a ) = 0 .

Original: Then, usingˆ M , we can compute d O ∈ R | S

Original: ||

Original: G ||

Original: A | + using linear programming and define reward r ∈ R | S || G | +as r ( s ; g )

Original: = log p ( s ; g )

Original: d O ( s ; g ) , where p ( s ; g ) ∈ R | S || G | + .

Original: Now, define T ∈ R | S

Original: ||

Original: G ||

Original: A |×| S || G | such that( T V )( s, a ; g ) =

Original: (cid:80) s ′ T (( s ′ , g )

Original: | ( s, g ) , a ) V ( s ′ ; g ) , where V ∈ R | S || G | + is the dual optimizationvariables.

Original: We also define B ∈ R | S

Original: ||

Original: G ||

Original: A |×| S || G | such that ( B V )( s, a ; g ) =

Original: V ( s ; g ) .

Original: Finally, we define

Original: D =

Original: [Equation]

Original: ||

Original: [Equation]

Original: | .

Original: Now, we can rewrite the dual problem as follow:

Original: [Equation]

Original: Now, we recognize that (50) is equivalent to Equation 49 in [32]

Original: [Equation]

Original: as we have reduced goal-conditioned

Original: RL to regular RL with an augmented state-space.

Original: Now, using the same derivation as in [32]

Original: [Equation]

Original: we havethat

Original: [Equation]

Original: and we can recover d ∗ ( s, a ; g ) :

Original: [Equation]

Original: Given d ∗

Original: [Equation]

Original: we may extract the optimal policy π ∗ by marginalizing over actions:

Original: [Equation]

Original: E Additional Technical Discussion

Original: E.1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL

Original: Suppose the GCRL problem comes with a reward function r ( s ; g ) .

Original: We also show that there is anequivalent goal-conditioned state-occupancy matching problem with a target distribution p ( s ; g )defined based on r ( s ; g ) .

Original: Proposition E.1.

Original: (Proposition 4.1 in the paper)

Original: Given any r ( s ; g ) , for each g in the support of p ( g ) ,define p ( s ; g ) =

Original: e r ( s ; g ) Z ( g ) , where Z ( g ) :

Original: = (cid:82) e r ( s ; g ) ds is the normalizing constant.

Original: Then, the followingequality holds:

Original: [Equation]

Original: [Equation]

Original: where J ( π ) is the GCRL objective (Eq.

Original: (2) ) with reward r ( s ; g ) and C :

Original: =

Original: E g ∼ p

Original: ( g ) [log Z ( g )] is aconstant.

Original: Proof.

Original: We have that

Original: [Equation]

Original: Rearranging the inequality gives the desired result.

Original: A constant term C appear in the equality to account for the need for normalizing e r ( s ; g ) to make it aproper distribution.

Original: This, however, does not change the optimal solution for the goal-conditionedstate-occupancy matching objective.

Original: Therefore, we have shown that for any choice of reward r ( s ; g ) ,solving the GCRL problem with a maximum state-entropy regularization is equivalent to optimizingfor the goal-conditioned state-occupancy matching objective with target distribution p ( s ; g ) :=

Original: e r ( s ; g ) Z ( g ) .

Original: E.2 Optimality Conditions for Hindsight Relabeling

Original: In section 4.2, we have stated that HER is not optimal for most choices of reward functions.

Original: In thissection, we investigate conditions under which hindsight relabeling methods such as HER would beoptimal.

Original: Let the goal-relabeling distribution for HER be p HER ( g | s, a ) ; we do not specify the functional formof p HER ( g | s, a ) for generality (see 26).

Original: Then, in order for this distribution to be optimal, then itmust satisfy

Original: [Equation]

Original: Then, the choice of r ( s ; g ) such that this equality holds is the reward function for which HER wouldbe optimal.

Original: However, solving for r ( s ; g ) is generally challenging and we leave it to future work forinvestigating whether doing so is possible for general f -divergence coupled with neural networks.

Original: This optimality condition is related to a prior work [10], which has found that hindsight relabeling isoptimal in the sense of maximum-entropy inverse RL [50] for a peculiar choice of reward function(see Equation 9 in [10]), which cannot be implemented in practice.

Original: Our result is more general as itapplies to any choice of f -divergence, and is not restricted to the form of maximum-entropy inverse

Original: RL.

Original: E.3 Theoretical Comparison to Prior Regression-based GCRL methods

Original: In section 4.3, we have stated that GoFAR’s theoretical guarantee (Theorem 4.1) is stronger in naturecompared to prior regression-based GCRL methods.

Original: Here, we provide an in-depth discussion.

Original: Both GCSL [14] and WGCSL [47] prove that their objectives are lower bounds of the true RLobjective (Theorem 3.1 in [14] and Theorem 1 in [47], respectively); however, in both works, thelower bounds are loose due to constant terms that do not depend on the policy and hence do notvanish to zero.

Original: In contrast, GoFAR’s objective (6) is, by construction, a lower bound on the RLobjective, as it simply incorporates a f -divergence regularization.

Original: If the offline data d O is on-policy ,then our lower bound is an equality.

Original: In contrast, even with on-policy data, the lower bounds in both

Original: GCSL and WGCSL are still loose due to the unavoidable constant terms.

Original: GCSL also proves a sub-optimality guarantee (Theorem 3.2 in [14]) under the assumption of fullstate-space coverage.

Original: Though full state-space coverage has been considered in some prior offline

Original: RL works [24, 31], it is much stronger than the concentrability assumption in our Theorem 4.1,which only applies to d ∗ .

Original: Furthermore, this guarantee is not statistical in nature, and instead directlymakes a strong assumption on the maximum total-variance distance between π and optimal π ∗ for the

Original: GCSL objective, which is difficult to verify in practice.

Original: In contrast, our bound suggests asymptoticoptimality: given enough offline data, the solution to GoFAR’s policy objective will converge toπ ∗ .

Original: Finally, WGCSL proves a policy improvement guarantee (Proposition 1 in [47]) under theirexponentially weighted advantage; the improvement is not a strict equality, and consequently there isno convergence guarantee to the optimal policy.

Original: Furthermore, this result is not directly dependent ontheir use of an advantage function, so it is not clear the precise role of their advantage function intheir algorithm.

Original: F Task Descriptions

Original: In this section, we describe the tasks in our experiments in Section 5.

Original: F.1 Fetch Tasks

Original: [Figure]

Original: The Fetch environments involve the Fetch robot with the following specifications, developed by

Original: Plappert et al.

Original: • Seven degrees of freedom

Original: [Figure]

Original: • Two pronged parallel gripper

Original: [Figure]

Original: • Three dimensional goal representing Cartesian coordinates of target

Original: [Figure]
Revised: [N/A]

Original: • Sparse, binary reward signal: 0 when at goal with tolerance 5cm and -1 otherwisejk

Original: [Figure]

Original: • 25 Hertz simulation frequency

Original: [Figure]

Original: • Four dimensional action space

Original: [Figure]

Original: – Three Cartesian dimensions

Original: [Figure]

Original: – One dimension to control gripper

Original: [Figure]

Original: F.1.1 Fetch Reach

Original: The task is to place the end effector at the target goal position.

Original: Observations consist of the endeffector’s positional state, whether the gripper is closed, and the end effector’s velocity.

Original: The reward isgiven by:

Original: [Equation]

Original: Push

Original: The task is to push an object to the target goal position.

Original: Observations consist of the end effector’sposition, velocity, and gripper state as well as the object’s position, rotational orientation, linearvelocity, and angular velocity.

Original: The reward is given by:

Original: [Equation]

Original: F.1.3 Fetch Slide

Original: In this task, the goal position lies outside of the robot’s reach and the robot must slide the puck-likeobject across the table to the goal.

Original: Observations consist of the end effector’s position, velocity,and gripper state as well as the object’s position, rotational orientation, linear velocity, and angularvelocity.

Original: The reward is given by:

Original: [Equation]

Original: The task is to grasp the object and hold it at the goal, which could be on or above the table.

Original: Observations consist of the end effector’s position, velocity, and gripper state as well as the object’sposition, rotational orientation, linear velocity, and angular velocity.

Original: The reward is given by:

Original: [Equation]

Original: F.2 Hand Reach

Original: Uses a 24 DoF robot hand with a 20 dimensional action space.

Original: Observations consist of each of the 24joints’ positions and velocities.

Original: The goal space is 15 dimensional corresponding to the positions ofeach of its five fingers.

Original: The goal is achieved when the mean distance of the fingers to their goals isless than 1cm.

Original: The reward is binary and sparse: 0 if the goal is reached and -1 otherwise, i.e.r ( s, a, g

Original: [Equation]

Original: =

Original: [Equation]

Original: (cid:32)

Original: [Equation]

Original: (cid:33)

Original: F.3 D’ClawTurn (Simulation)

Original: First introduced by Ahn et al.

Original: [2], the D’Claw environment has a 9 DoF three-fingered robotic hand.

Original: The turn task consists of turning the valve to a desired angle.

Original: The initial angle is randomly chosenfrom [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − 2 ∗ π 3 , 2 ∗ π 3 ] .

Original: The observation space is 21D,consisting of the current joint angles θ t , their velocities ˙ θ , angle between current and goal angle, andthe previous action.

Original: The environment terminates after 80 steps.

Original: The reward function is defined as:

Original: [Equation]

Original: F.4 D’ClawTurn (Real)

Original: To make real-world data collection easier, we slightly modify the initial and target angle distributions.

Original: The initial angle is randomly chosen from [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − π 2 , π 2 ] .

Original: Using this task distribution, collecting 400K transitions with random actions takes about 15 hours.

Original: In

Original: Figure 8, we also include a larger picture of the robot platform.

Original: G Experimental Details

Original: In this section, we provide experimental details omitted in Section 5 of the main text.

Original: These include(1) technical details of the baseline methods, (2) hyperparameter and architecture details for allmethods, (3) offline GCRL dataset details, and finally, (4) experimental details of the zero-shottransfer experiment.

Original: G.1 Baseline Implementation Details

Original: DDPG.

Original: We use an open-source implementation of DDPG, which has already tuned DDPG on theset of Fetch tasks.

Original: We implement all other methods on top of this implementation, keeping identicalarchitectures and hyperparameters when appropriate.

Original: The critic objective is

Original: [Equation]

Original: (57) where ¯ Q denotes the stop-gradient operation.

Original: The policy objective is

Original: [Equation]

Original: DDPG updates the critic and the policy in an alternating fashion.

Original: ActionableModel.

Original: We implement AM on top of DDPG.

Original: Specifically, we add a CQL loss in the criticupdate:

Original: [Equation]

Original: where d ˜ O is the distribution of the relabelled dataset.

Original: In practice, we sample 10 random actionsfrom the action-space to approximate this expectation.

Original: Furthermore, we implement goal-chaining,where for half of the relabeled transitions in each minibatch update, the relabelled goals are randomlysampled from the offline dataset.

Original: We found goal-chaining to not be stable in some environments, inparticular, FetchPush, FetchPickAndPlace, and FetchSlide.

Original: Therefore, to obtain better results, weremove goal-chaining for these environments in our experiments.

Original: GCSL.

Original: We implement GCSL by removing the DDPG critic component and changing the policy lossto maximum likelihood:

Original: [Equation]

Original: WGCSL.

Original: We implement WGCSL on top of GCSL by including a Q-function.

Original: The Q-functionis trained using TD error as in DDPG.

Original: To compute the advantage term, we follow the original

Original: WGCSL algorithm by setting V ( s t ; g ) =

Original: Q ( s t , π ( s t +1 , g ); g ) and then computing A ( s t , a t , g )

Original: =

Original: r ( s t ; g ) + γV ( s t +1 ; g ) − V ( s t ; g ) .

Original: Using this, the WGCSL policy objective is

Original: [Equation]

Original: (61) where we clip exp( · ) for numerical stability.

Original: The original WGCSL uses different HER rates forthe critic and the actor training.

Original: To make the implementation simple and consistent with all otherapproaches, we use the same HER rate for both components.

Original: With the exception of AM, all baselines set the goal-relabeling distribution d ˜ O to be the uniformdistribution over future states in the same trajectory (See Equation (26)).

Original: G.2 Architectures and Hyperparameters

Original: Each algorithm uses their own set of fixed hyperparameters for all tasks.

Original: WGCSL, GCSL, and DDPGare already tuned on our set of tasks [39, 47], so we use the reported values from prior works; AM, inour implementation, shares same networks as DDPG, so we use DDPG’s values.

Original: For GoFAR, weuse identical hyperparameters as WGCSL because they share similar network components; GoFARadditionally trains a discriminator, for which we use the same architecture and learning rate as thevalue network.

Original: We impose a small discriminator gradient penalty [16] to prevent overfitting.

Original: For allexperiments, We train each method for 3 seeds, and each training run uses 400k minibatch updates ofsize 512.

Original: The architectures and hyperparameters for all methods are reported in Table 3.

Original: G.3 Offline GCRL Experiments

Original: Datasets.

Original: For each environment, the offline dataset composition is determined by whether datacollected by random actions provides sufficient coverage of the desired goal distribution.

Original: For

Original: FetchReach and D’ClawTurn, we find this to be the case and choose the offline dataset to be 1 millionrandom transitions.

Original: For the other four tasks, random data does not capture meaningful goals, so wecreate a mixture dataset with 100K transitions from a trained DDPG-HER agent and 900K randomtransitions; the transitions are not labeled with their sources.

Original: This mixture setup has been consideredin prior works [21, 32] and is reminiscent of real-world datasets, where only a small portion of thedataset is task-relevant but all transitions provide useful information about the environment.

Original: G.4 Zero-Shot Transfer Experiments

Original: We use GoFAR (Binary) variant for trainning the GoFAR planner.

Original: The low-level controller is trainedusing an online DDPG algorithm on a narrow goal distribution, set to be closed to the object’s initialpositions.

Original: GoFAR Hierarchical Controller operates by first generating a sequence of subgoals ( g 1 , ..., g T ) usingπ high by recursively feeding the newest generated goal and conditioning on the final goal g .

Original: Then, at

Original: [Table]

Original: each time step t , the low-level controller executes action π low ( a t | s t , g t ) .

Original: The high-level subgoalsare not re-planned during low-level controller execution.

Original: We note that this is a simple planningalgorithm

Original: [Table]

Original: and improvement in performance can be expected by considering more sophisticatedplanning approaches.

Original: H Additional ResultsH.1 Offline GCRL Full Results

Original: In this section

Original: [Table]

Original: we provide the full results table for discounted return, final distance, and success ratemetrics, including error bars over 10 random seeds.

Original: The number inside the parenthesis indicatesthe best HER rate for the baseline methods on the task.

Original: Star ( ⋆ ) indicate statistically significantimprovement over the second best-performing method under a 2-sample t -test.

Original: [Table]

Original: [Table]

Original: [Table]

Original: H.2 Ablations

Original: [Table]

Original: We also include the full task-breakdown table of GoFAR ablations presented in Figure 3 for completeness.

Original: As shown in 7, GoFAR and GoFAR (HER) perform comparatively on all tasks.

Original: GoFAR (binary)is slightly worse across tasks, and GoFAR (KL) collapses due to the use of an unstable f -divergence.

Original: [Table]

Original: H.3 Real-World Dexterous Manipulations

Original: In our qualitative analysis, we visualize all methods on a specific task instance of turning the valveprong (marked by the red strip) clockwise for 90 degree; the goal location is marked by the greenstrip.

Original: The robot initial pose is randomized.

Original: As shown in Figure 9, GoFAR reaches the goal with threerandom initial poses, whereas all baselines fail.

Original: See the figure captions for detail.

Original: Policy videos areincluded in the supplementary material.

Original: [Figure]

Original: H.4 Zero-Shot Plan Transfer

Original: We visualize GoFAR hierarchical controller and the plain low-level controller on three distinct goalsin Figure 10.

Original: See the figure caption for detail.

Original: Policy videos are included in the supplementarymaterial.

Original: [Figure]