How Far I’ll Go: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression
Abstract
Offline goal-conditioned reinforcement learning (GCRL) promises general-purpose skill learning in the form of reaching diverse goals from purely offline datasets.
We propose Go al-conditioned f - A dvantage R egression (GoFAR), a novel regressionbased offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a stateoccupancy matching problem between a dynamics-abiding imitator agent and an expert agent that directly teleports to the goal.
In contrast to prior approaches,
GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks.
These distinct features confer GoFAR with much better offline performance and stability as well as statistical performance guarantee that is unattainable for prior methods.
Furthermore, we demonstrate that GoFAR’s training objectives can be re-purposed to learn an agent-independent goal-conditioned planner from purely offline source-domain data, which enables zero-shot transfer to new target domains.
Through extensive experiments, we validate GoFAR’s effectiveness in various problem settings and tasks, significantly outperforming prior state-of-art.
Notably, on a real robotic dexterous manipulation task, while no other method makes meaningful progress, GoFAR acquires complex manipulation behavior that successfully accomplishes diverse goals. Project website: https://jasonma2016.github.io/GoFAR/
Introduction
Goal-conditioned reinforcement learning [19, 47, 43] (GCRL) aims to learn a repertoire of skills in the form of reaching distinct goals.
Offline GCRL [6, 51] is particularly promising because it enables learning general goal-reaching policies from purely offline interaction datasets without any environment interaction [30, 26], which can be expensive in the real-world.
As offline datasets contain diverse goals and become increasingly prevalent [9, 20, 6], policies learned this way can acquire a large set of useful primitives for downstream tasks [32].
A central challenge in GCRL is the sparsity of reward signal [3]; without any additional knowledge about the environment, an agent at a state typically only accrues positive binary reward when the state lies within the goal neighborhood.
This sparse reward problem is exacerbated in the offline setting, in which the agent cannot explore the environment to discover more informative states about desired goals.
Therefore, designing an effective offline GCRL algorithm is a concrete yet challenging path towards general-purpose and scalable policy learning.
In this paper, we present a novel offline GCRL algorithm,
Go al-conditioned f - A dvantage R egression (GoFAR), first casting GCRL as a state-occupancy matching [29, 35] problem and then deriving a regression-based policy objective.
In particular, GoFAR begins with the following goal-conditioned state-matching objective:
[Equation]
1) where d π ( s ; g ) is the goal-conditioned state-occupancy distribution of policy π and p ( s ; g ) is the distribution of states that satisfy a particular goal g .
This objective casts goal-conditioned offline RL as an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expert who can teleport to the goal in one step; see Figure 2.
Posing GCRL this way is mathematically principled, as we show that this objective is equivalent to a probabilistic interpretation of GCRL that additionally encourages maximizing state entropy.
More importantly, this objective admits elegant optimization using purely offline data by considering an f -divergence regularized objective and exploiting ideas from convex duality theory [45, 4].
In particular, we obtain the dual optimal value function V ∗ from a single unconstrained optimization problem, using which we construct the optimal advantage function (we refer to this as f-advantage ) that serves as importance weighting for a regression-based policy training objective; see Figure 1 for a schematic illustration.
There are several distinct features to our approach.
First and foremost is GoFAR’s lack of goal relabeling.
Hindsight goal relabeling [3] (known as HER in the literature) relabels trajectory goals to be states that were actually achieved instead of the originally commanded goals.
This heuristic is critical for alleviating the sparse-reward problem in prior GCRL methods, but is unnecessary for GoFAR.
GoFAR updates its policy as if all data is already coming from the optimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling.
GoFAR’s relabeling-free training is of significant practical benefits.
First, it enables more stable and simpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot be easily performed offline [52].
Second, hindsight relabeling suffers from hindsight bias in stochastic environments [48], as achieved goals may be accomplished due to noise; this is exacerbated in the offline setting due to inability to collect more data.
By bypassing hindsight relabeling, GoFAR promises to be more robust and scalable.
Second, as suggested in Figure 1, GoFAR’s value and policy training steps are completely uninterleaved : we do not need to update the policy until the value function has converged.
This confers greater offline training stability [24, 35] compared to prior works, which mostly involve alternating updates to a critic Q-network and policy network.
Furthermore, it enables an algorithmic reduction of GoFAR to weighted regression [7], which allows us to obtain strong finite-sample statistical guarantee on GoFAR’s performance; prior regression-based GCRL approaches [15, 51] do not enjoy this reduction and obtain much weaker theoretical guarantees.
Finally, we show that GoFAR can also be used to learn a goal-conditioned planner [39, 5].
The key insight lies in observing that, under a mild assumption, GoFAR’s dual value function objective does not depend on the action and can learn from state-only offline data.
This enables learning a goal-centric value function and thereby a near-optimal goal-conditioned planner that is capable of zero-shot transferring to new domains of the same task that shares the goal space.
In our experiments, we illustrate that GoFAR planner can indeed plan effective subgoals in a new target domain and enable a low-level controller to reach distant goals that it is not designed for.
We extensively evaluate GoFAR on a variety of offline GCRL environments of varying task complexity and dataset composition, and show that it outperforms all baselines in all settings.
Notably, GoFAR is more robust to stochastic environments than methods that depend on hindsight relabeling.
We additionally demonstrate that GoFAR learns complex manipulation behavior on a real-world highdimensional dexterous manipulation task [2], for which most baselines fail to make any progress.
Finally, we showcase GoFAR’s planning capability in a cross-robot zero-shot transfer task.
To summarize, our contributions are:
1. GoFAR, a novel offline GCRL algorithm derived from goal-conditioned state-occupancy matching 2.
Detailed technical derivation and analysis of GoFAR’s distinct features: (1) relabeling-free, (2) uninterleaved optimization, and (3) planning capability 3.
Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities
Related Work
Offline Goal-Conditioned Reinforcement Learning.
One core challenge of (offline) GCRL [19, 47] is the sparse-reward nature of goal-reaching tasks.
A popular strategy is hindsight goal relabeling [3] (or HER), which relabels off-policy transitions with future goals they have achieved instead of the original commanded goals.
Existing offline GCRL algorithms [6, 51] adapt HER-based online GCRL algorithms to the offline setting by incorporating additional components conducive to offline training. [6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [25] as well as goal-chaining, which expands the pool of candidate relabeling goals to the entire offline dataset.
Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shown to be a simple and effective method [15].
[51] improves upon [15] by incorporating discount-factor and advantage function weighting [41, 42], and shows improved performance in offline GCRL.
Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting an online algorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matching and derives an elegant algorithm from the dual formulation that is naturally suited for the offline setting and carries many distinct properties that make it more stable, scalable, and versatile.
State-Occupancy Matching.
Posing reinforcement learning as learning an occupancy distribution has been shown to be an effective paradigm for learning from observations [35, 53], exploration [29, 10], and safe RL [28, 34]; state-occupancy matching, in general, has been explored in the imitation learning literature [18, 49, 14, 21].
At a technical level, our strategy of estimating the state-occupancy ratio is related to the DICE technique [36, 38], which has recently been extended to the offline setting [27, 22, 35].
The closest work to ours is [35], which proposes a f -divergence based stateoccupancy matching objective for offline imitation learning.
Our work differs in that we consider a goal-conditioned setting and do not require a separate dataset of demonstrations for policy learning.
Furthermore, we derive several new results (Section 4.2-4.4) that are of particular interest in the GCRL setting.
Problem Formulation
Goal-Conditioned Reinforcement Learning.
We consider an infinite-horizon Markov decision process (MDP) [44] M = ( S, A, R, T, µ 0 , γ ) with state space S , action space A , deterministic rewards r ( s, a ) , stochastic transitions s ′ ∼ T ( s, a ) , initial state distribution µ 0 ( s ) , and discount factor γ ∈ (0 , 1] .
A policy π : S → ∆( A ) outputs a distribution over actions to use in a given state.
In goal-conditioned RL, the MDP additionally assumes a goal space G :=
{ ϕ ( s ) | s ∈ S } , where the state-to-goal mapping ϕ : S → G is known.
Now, the reward function r ( s ; g ) 1 as well as the policy π ( a | s, g ) depend on the commanded goal g ∈ G .
Given a distribution over desired goals p ( g ) , the objective of goal-conditioned RL is to find a policy π that maximizes the discounted cumulative return:
[Equation]
[Equation]
The goal-conditioned state-action occupancy distribution d π ( s, a ; g ) : S ×
A × G → [0 , 1] of π is d π ( s, a [Equation] g ) := [Equation]
) ∞
[Equation]
[Equation]
which captures the relative frequency of state-action visitations for a policy π conditioned on goal g .
The state-occupancy distribution then marginalizes over actions: d π ( s ; g )
= (cid:80) a d π ( s, a ; g ) .
Then, it follows that π ( a | s, g ) =
d π ( s,a ; g ) d
π ( s ; g ) .
A state-action occupancy distribution must satisfy the Bellman flow constraint in order for it to be an occupancy distribution for some stationary policy π : (cid:88)
[Equation]
We write d π ( s, g )
=
p ( g ) d π ( s ; g ) as the joint goal-state density induced by p ( g ) and the policy π .
Finally, given d π , we can express the objective function (2) as J ( π ) =
11 − γ E
( s,g )
∼ d π ( s,g ) [ r ( s ; g )] .
Offline GCRL.
In offline GCRL, the agent cannot interact with the environment M ; instead, it is equipped with a static dataset of logged transitions D := { τ i } Ni =1 , where each trajectory τ ( i ) =
( s ( i ) 0 , a ( i ) 0 , r
( i ) 0 , s ( i ) 1 , ... ; g ( i ) ) with s ( i )
∼ µ 0 and g ( i ) is the commanded goal of the trajectory.
Note that trajectories need not to be generated from a goal-directed agent, in which case g ( i ) can be randomly drawn from p ( g ) .
We denote the empirical goal-conditioned state-action occupancies of
D O as d O ( s, a ; g ) .
Goal-Conditioned f -Advantage Regression
In this section, we introduce Goal-conditioned f -Advantage Regression (GoFAR).
We first derive the algorithm in full (Section 4.1), then delve deep into its several appealing properties (Section 4.2-4.4).
4.1 Algorithm
Recall the goal-conditioned state-occupancy matching objective stated in (1):
[Equation]
The key challenge with optimizing this objective offline is that we cannot sample from d π ( s ; g ) .
To address this issue, we first derive an offline lower bound involving an f -divergence (see Appendix A for definition) regularization term, which subsequently enables solving this optimization problem via its dual using purely offline data:
Proposition 4.1.
Assume 2 for all g in support of p ( g ) , ∀ s, d O ( s ; g ) > 0 if p ( s ; g ) > 0 .
Then, for any f -divergence that upper bounds the KL-divergence,
[Equation]
The RHS of (5) can be understood as an f -divergence regularized GCRL objective with reward function r ( s ; g )
= log p ( s ; g )
d O ( s ; g ) .
Intuitively, this reward encourages visiting states that occur more often in the “expert” state distribution p ( s ; g ) than in the offline dataset, and the f -divergence regularization then ensures that the learned policy is supported by the offline dataset.
This choice of reward function can be estimated in practice by training a discriminator [16] c : S × G → (0 , 1) using the offline data:
[Equation]
We can in fact obtain a looser lower bound that does not require training a discriminator:
[Equation]
Here, the reward function is r ( s ; g ) =
log p ( s ; g ) , which encompasses standard GCRL reward functions [40] and is related to a probabilistic interpretation of GCRL; see Appendix E.1for detail. Now [Equation]
for either choice of lower bound [Equation] we may pose the optimization problem with respect to valid choices of state-action occupancies directly, introducing the Bellman flow constraint (4):
[Equation]
This reformulation does not solve the fundamental problem that (8) still requires sampling from d ( s ; g ) ; however, it has now written the problem in a way amenable to simplification using tools from convex analysis.
Now, we show that its dual problem can be reduced to an unconstrained minimization problem over the dual variables which serve the role of a value function; importantly, the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution:
Proposition 4.2.
The dual problem to (8) is
[Equation]
where f ⋆ denotes the convex conjugate function of f , V ( s ; g ) is the Lagrangian vector, and
T V ( s, a ; g ) =
E s ′ ∼ T ( ·| s [Equation] a ) [ V ( s ′ ; g )] .
Given the optimal V ∗ , the primal optimal d ∗ satisfies:
[Equation]
A proof is given in Appendix B. Crucially, as neither expectation in (9) depends on samples from d , this objective can be estimated entirely using offline data, making it suitable for offline GCRL.
For tabular MDPs, we show that for a suitable choice of f -divergence, the optimal V ∗ in fact admits closed-form solutions; see Appendix D for details.
In the continuous control setting, we can optimize (9) using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.
Then, once we have obtained the optimal (resp.
converged) V ∗ , we propose learning the policy via the following supervised regression update:
[Equation]
We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluated at the dual optimal advantage , r ( s ; g ) + γ T V ∗ ( s, a ; g ) − V ∗ ( s ; g ) ; we refer to this weighting term as f -advantage .
Hence, we name our overall method Goal-conditioned f -Advantage Regression (GoFAR) ; an abbreviated high-level pseudocode is provided in Algorithm 1 and a detailed version is provided in Appendix C.
In practice, we implement GoFAR with χ 2 -divergence, a choice of f -divergence that is stable for off-policy optimization [53, 35, 27]; see Appendix C for details. Algorithm (Abbreviated); 3-disjoint steps ) Train discriminator
based 2: Train (optimal) dual value function V ∗ ( s g ) (9) 3: Train policy π via f -Advantage Regression Note that (11) forgos directly minimizing (1) offline, which has been found to suffer from training instability [22].
Instead, it naturally incorporates the primal-dual optimal solutions in a regression loss.
Now, we will show that this policy objective has several theoretical and practical benefits for offline GCRL that make it particularly appealing.
4.2 Optimal Goal-Weighting Property
We show that optimizing (11) automatically obtains the optimal goal-weighting distribution .
That is,
GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (8).
In particular, this property is achieved without any explicit hindsight relabeling (see Appendix A for a technical definition), a mechanism that prior works heavily depend on.
To this end, we first note that by Bayes rule, p ( g ) d O ( s, a ; g ) =
d O ( s, a ) p ( g | s, a ) .
Then, we can rewrite the policy objective (11) as follows:
[Equation]
where
[Equation]
Thus [Equation] we see that GoFAR’s f -advantage weighting scheme is equivalent to performing supervised policy regression where goals are sampled from ˜ p
( g | s, a ) .
Now, combining (14) and Bayes rule gives
[Equation]
Thus, we can replace the nested expectations in (13) and obtain that GoFAR policy update amounts to supervised regression of the state-action occupancy distribution of the optimal policy to the regularized GCRL problem (8):
[Equation]
This derivation makes clear GoFAR’s connection with the hindsight goal relabeling mechanism [3] that is ubiquitous in GCRL:
GoFAR automatically performs the optimal goal-weighting policy update without any explicit goal relabeling.
Furthermore, our derivation also suggests why hindsight relabeling is sub-optimal without further assumption on the reward function [11]: it heuristically chooses ˜ p ( g | s, a ) to be the empirical trajectory-wise future achieved goal distribution (i.e., HER), which generally does not coincide with the goals that the optimal policy would reach; see Appendix [-E.
 forfurther-]E.2 for further discussion.
4.3 Uninterleaved Optimization and Performance Guarantee
An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the value network and the policy network.
This can be observed by noting that GoFAR’s advantage term (10) is computed using V ∗ , the optimal solution of the dual problem.
This has the practical significance of disentangling the value-function update (9) from the policy update (11), as we do not need to optimize the latter until the former has converged.
This disentanglement is in sharp contrast to prior
GCRL works [51, 6, 12, 3], which typically involve alternating updates to the critic Q-network and the policy network, a training procedure that has found to be unstable in the offline setting [24].
This is unavoidable for prior works because their advantage functions are estimated using the Q-estimate of the current policy, whereas our advantage term naturally falls out from the primal-dual optimality condition.
The uninterleaved and relabeling-free nature of GoFAR also allows us to derive strong performance guarantees.
Because V ∗ is fixed in (11), this policy objective amounts to a weighted supervised learning problem.
Therefore, we can extend and adapt mature theoretical results for analyzing
Behavior Cloning [1, 46, 50] as well as finite sample error guarantees for weighted regression [7] to obtain statistical guarantees on GoFAR’s performance with respect to the optimal π ∗ for (8):
Theorem 4.1.
Assume sup s,a,g d ∗ ( s,a ; g )
d O ( s,a ; g ) ≤ M and sup | r ( s, g )
| ≤ R max .
Consider a policy class Π : { S → ∆( A ) } such that π ∗ ∈ Π .
Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
GoFAR (16) will return a policy ˆ π such that:
[Equation]
Notably, the error shrinks as the size of the offline data N increases, requiring no dependency on access to data from the “expert” distribution d ∗ .
This provides a theoretical basis for GoFAR’s empirical scalability as we are guaranteed to obtain good results when the offline data becomes more expansive.
In contrast, prior regression-based GCRL methods cannot be easily reduced to a simple weighted regression with respect to the desired goal distribution p ( g ) , so they only obtain weaker results under stronger assumptions on the offline data (e.g, full state-space coverage) as well as the policy; see Appendix E.3 for discussion.
4.4 Goal-Conditioned Planning
Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zeroshot transfer to other domains of the same task.
The key insight is that GoFAR’s value function objective (9) does not depend on actions assuming deterministic transitions.
This is because given a transition ( s, a, s ′ , g ) ∼ d
O , T V ( s, a ; g ) =
V ( s ′ ; g ) , so we can rewrite the second term in (9) as E ( s,a,s ′ ,g ) ∼ d
O [ f ⋆ ( r ( s, g ) + γV ( s ′ ; g ) − V
s ; g ))] , which does not depend on the action.
This property enables us to learn a goal-centric value function that is independent of the agent’s state space because the agent’s actions are not relevant.
Specifically, we propose the following objective:
[Equation]
We can think of this objective as learning a value function with the inductive bias that the first operation transforms the state input to the goal space via ϕ .
Since V is now independent of the agent, we can use f -advantage regression to instead learn a goal-conditioned planner :
[Equation]
19) where π now outputs the next subgoal ϕ ( s ′ ) conditioned on the current achieved goal ϕ ( s ) and the desired goal g .
In our experiments, we show how this planner can achieve hierarchical control through zero-shot transfer subgoal plans to a new target domain that shares the same goal space and by empowering a target-domain low-level controller that is only effective for nearby goals to reach distant goals.
Experiments
We pose the following questions and provide affirmative answers in our experiments 3 :
Is GoFAR effective for offline GCRL?
What components are important for performance?
Is GoFAR more robust to stochastic environments than hindsight relabeling methods? 3.
Can GoFAR be applied to a real-robotics system?
Can GoFAR learn a goal-conditioned planner for zero-shot cross-embodiment [-transfer?5.
-]transfer?
5.1 Offline GCRL
Tasks.
In our simulation experiments, we consider six distinct environments.
They include four robot manipulation environments[43]: FetchReach, FetchPickAndPlace, FetchPush, and FetchSlide, and two dexterous manipulation environments: HandReach [43], and D’ClawTurn [2].
The dexterous manipulation tasks are particularly challenging due to the high-dimensional action space that makes credit assignment difficult.
The offline dataset for each task is collected by either a random policy or a mixture of 90% random policy and 10% expert policy [35, 22], depending on whether random data provides enough coverage of the desired goal distribution.
See Appendix G.3 for dataset details and
Appendix F for detailed task descriptions and figures.
Algorithms.
We compare to state-of-art offline GCRL algorithms, consisting of both regression-based and actor-critic methods.
The regression-based methods are: (1) GCSL [15], which incorporates hindsight relabeling in conjunction with behavior cloning to clone actions that lead to a specified goal, and (2) WGCSL [51], which improves upon GCSL by incorporating discount factor and advantage weighting into the supervised policy learning update.
The actor-critic methods are (1)
DDPG [3], which adapts DDPG [31] to the goal-conditioned setting by incorporating hindsight relabeling, and (2) ActionableModel (AM) [6], which incorporates conservative Q-Learning [25] as well as goal-chaining on top of an actor-critic method; in this work, we use DDPG as the actor-critic method for AM.
We use tuned hyperparameters for each baseline on all tasks and additionally search the best HER ratio from { 0 . 2 , 0 . 5 , 1 . 0 } for each task separately.
For GoFAR, we use identical hyperparameters as
WGCSL for the shared network components and do not tune further.
We train each method forseeds, and each training run uses 400k minibatch updates of size 512.
Complete architecture and hyperparameter table as well as additional training details are provided in Appendix G.
[Table]
Evaluations and Results.
We report the discounted return using the sparse binary task reward.
This metric rewards algorithms that reach goals as fast as possible and stay in the goal region thereafter.
Note that this metric favors the baseline algorithms as they use the same binary reward label and therefore have knowledge of the success criterion that is used for evaluation.
In contrast, GoFAR’s reward signal comes from its discriminator.
Because the task reward is binary, this metric does not take account into how precisely a goal is being reached.
Therefore, we additionally report the final distance to goal in Appendix H.
The full discounted return results 4 are shown in Table 1; the fraction inside () indicates the best HER rate for each baseline.
Aside from the simplest FetchReach, on which actor-critic methods slightly outperform GoFAR, GoFAR attain the best performance on all tasks and is significantly better on the more challenging dexterous manipulation tasks.
In Table 5 in Appendix H, we also report the [-final
 The error bar version is included in Appendix H.-]final distance metric, on which GoFAR is once again superior.
In other words, GoFAR reaches the goals fastest and most precisely.
Ablations.
Recall that GoFAR achieves optimal goal-weighting, and does not require the HER heuristic that has been key to prior GCRL approaches.
We now experimentally confirm that the baselines are not performant without HER.
Conversely, we investigate whether HER can help GoFAR.
Additionally, we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in (7).
Finally, we replace χ 2 -divergence with KL to understand whether the f -divergence lower bound in (5) is necessary.
The results are shown in
Figure 3 (task-specific breakdown is included in Appendix H).
Compared to their original performances (colored dash lines), baselines without
HER suffer severely, especially GCSL and WGCSL: both these methods in fact require 100% relabeling and rely solely on HER for positive learning signals.
In sharp contrast, GoFAR is also a regression-based method, yet it performs very well without HER and its performance is unaffected by adding HER.
These findings are consistent with the theoretical results of Section 4.2.
GoFAR (Binary) performs slightly worse than GoFAR, which is to be expected due to the looser bound; however, it is still better than all baselines, which use the same binary reward and are aided by HER.
Again, this ablation highlights the merit of our optimization approach.
Finally, GoFAR (KL) suffers from numerical instability and fails to learn, showing that our general f -divergence lower bound (5) is not only of theoretical value but also practically significant.
5.2 Robustness in Stochastic Offline GCRL Settings
evaluation.
For this experiment, we create noisy variants of the FetchReach environment by adding varying levels of white Gaussian noise to policy action outputs.
FetchReach is well-suited for this experiment because all baselines attain satisfactory performance and even outperform GoFAR in Table 1.
Furthermore, to isolate the effect of noisy actions, we use GoFAR with binary reward so that all methods now use the same reward input and perform comparatively without added noise.
We consider zero-mean Gaussian noise of σ value from 0 .
5 , 1 , and 1 . 5 .
For each noise level, we re-collect offline data by executing random actions in the respective noisy environment and train on these noisy offline random data.
The average discounted returns over different noise levels are illustrated in Figure 4.
As shown, at noise level 0 .
5 , while GoFAR’s performance is not effected, all baselines already exhibit degraded performance.
In particular, DDPG sharply degrades, underperforming GoFAR despite better original performance.
At 1 . 0 , the gap continues to widen, and AM notably collapses, highlighting its sensitivity to noise.
This is expected as AM’s relabeling mechanism also samples future goals from other trajectories and labels the selected goals with their current Q-value estimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity, thus contributing to the instability observed in this experiment.
It is only at noise level 1 . 5 that
GoFAR’s performance degradation becomes comparable to those of other methods.
Thus, GoFAR’s relabeling-free optimization indeed confers greater robustness to environmental stochasticity; in contrast, all baselines suffer from “false positive” relabeled goals due to such disturbances.
5.3 Real-World Robotic Dexterous Manipulation
Now, we evaluate on a real D’Claw [2] dexterous manipulation robot (Figure 5 (left)).
The task is to have the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix F for detail.
The offline dataset contains 400K transitions, collected by executing purely random actions in the environment, which provides sufficient coverage of the goal space.
Compared to the simulated
D’Claw, an additional challenge is the inherent stochasticity on a real robot system due to imperfect actuation and hardware wear and tear.
Given our conclusion in the stochastic environment experiment above, we should expect GoFAR to perform better.
The training curve of final distances averaged over 3 seeds is shown in Figure 6.
We see that
GoFAR exhibits smooth progress over training and significantly outperforms all baselines.
At the end of training, GoFAR is able to rotate the valve on average within 10 degrees of the specified goal angle.
In contrast, most baselines do not make any progress on this difficult task and learns policies that do not manage to turn the valve at all.
AM is the only baseline that learns a non-trivial policy; however, its training fluctuates significantly, and the learned policy often executes extreme actions during evaluation that fail to reach the desired goal.
We provide qualitative analysis in Appendix H.3, and the full videos are included on the project website.
5.4 Zero-Shot Transfer Across Robots
[Table]
For this experiment, we consider zero-shot transfer from the FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Panda robot [13]; the environments are illustrated in Figure 5.
In the target domain, we assume a pre-programmed or learned low-level goal-reaching controller π low ( a | s, g ) that can push objects over short distances but fails for longer tasks.
For these distant goals, we train the goal-conditioned GoFAR value function V ∗ ( ϕ ( s ); g ) and planner π ( ϕ ( s ′ )
| ϕ ( s ) , g ) according to (18) and (19), to set goals for π low .
We use the same offline FetchPush data as in Section 5.1.
We compare two approaches: (1) naively using the Low-Level Controller , and (2) using the GoFAR planner to guide the low-level controller ( GoFAR Hierarchical Controller ).
See Appendix G. for additional details.
We report the success rate over 100 random distant test goals in Table 2.
We see that naively commanding the low-level controller with these distant goals indeed results in very low success rate.
When the controller is augmented with GoFAR planner, the success rate nearly doubles.
Note that our planner itself is much superior: if all subgoals are perfectly reached, GoFAR
Planner (Oracle) achieves 84% success.
See Appendix G.4 for additional experimental details.
These experiments validate GoFAR’s ability to zero-shot transferring subgoal plans to enhance the capabability of low-level controllers.
Note that we have no access to any data from the target domain, and this planning capability naturally emerges from our training objectives and does not require any change in the algorithm.
We show qualitative results in Appendix H.4 and videos on the project website.
Conclusion
We have presented GoFAR, a novel regression-based offline GCRL algorithm derived from a stateoccupancy matching perspective.
GoFAR is relabeling-free and enjoys uninterleaved training, making it both theoretically and practically advantageous compared to prior state-of-art.
Furthermore, GoFAR supports training a goal-conditioned planner with promising zero-shot transfer capability.
Through extensive experiments, we have validated the practical utility of GoFAR in various challenging problem settings, showing significant improvement over prior state-of-art.
We believe that GoFAR’s strong theoretical foundation and empirical performance make it an ideal candidate for scalable skill learning in the real world.
Acknowlegement
We thank Edward Hu and Oleh Rybkin for their feedback on an earlier draft of the paper, Kun Huang for assistance on the real-robot experiment setup, and Selina Li for assistance on graphic design. This work is funded in part by an Amazon Research Award, gift funding from NEC Laboratories America, NSF Award CCF-1910769, NSF Award CCF-1917852 and ARO Award W911NF-201-0080. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Table of Contents A Additional Technical Background
[-A Additional Technical Background 
-]
[Table]
[Table] [Table]
[-B Proofs 
-][Table] B.
[Table]
Proof of Proposition 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.
[Table]
C GoFAR Technical Details
[Table] [Table] [Table]
[Table]
[Table] for Tabular MDPs[-
-]
E Additional Technical Discussion[-
-]
[Table]
and
[Table] [Table] Hindsight Relabeling
[Table]
Theoretical Comparison to Prior Regression-based GCRL methods
F Task Descriptions
[Table]
[Table] [Table] [Table] [Table]
G.
[Table]
[-Details 
 G.1-] Baseline Implementation Details G.
[Table] Architectures and Hyperparameters G. [Table]
Offline GCRL Experiments
G.4 Zero-Shot Transfer Experiments
[Table]
Results[-
-]
H 1 Offline GCRL
H Ablations
H.3 Real-World Dexterous Manipulations
H.4 Zero-Shot Plan Transfer
A Additional Technical Background
We provide some technical definitions that are needed in our proofs in Appendix B and discussions of hindsight relabeling in Section 4.1 of the main text as well as Appendix E.2.
A.1 f -Divergence and Fenchel Duality
These definitions are adapted from [35, 37].
Definition A.1 ( f -divergence) .
For any continuous, convex function f and two probability distributions p, q ∈ ∆( X ) over a domain X , the f -divergence of p computed at q is defined as
[Equation]
Some common choices of f -divergence includes the KL-divergence and the χ 2 -divergence, which corresponds to choosing f ( x ) = x log x and f ( x ) = 12 ( x − 1) 2 , respectively.
Definition A.2 (Fenchel conjugate) .
Given a vector space X with inner-product ⟨· , ·⟩ , the Fenchel conjugate f ⋆ : X ⋆ → R of a convex and differentiable function f : Ω → R is
[Equation]
21) and any maximizer x ∗ of f ⋆ ( y ) satisfies x ∗ = f ′ ⋆ ( y ) .
For an f -divergence, under mild realizability assumptions [8] on f , the Fenchel conjugate of D f ( p ∥ q ) at y : X → R is
[Equation] [Equation]
23) and any maximizer p ∗ of D
⋆,f ( y ) satisfies p ∗ ( x ) =
q ( x ) f ′ ⋆ ( y ( x )) .
These optimality conditions can be seen as extensions of the KKT-condition.
A.2 Hindsight Goal-Relabeling
We provide a mathematical formalism of hindsight goal relabeling [3].
Definition A.3.
Given a state s t from a trajectory τ = { s 0 , a 0 , r 0 , ..., s T ; g } , hindsight goalrelabeling is the goal-relabeling distribution
[Equation]
where q is some categorical distribution taking values in { ϕ ( s t ) , ..., ϕ ( s T ) } .
That is, the relabeled goal is selected from some distribution goals that are reached in the future in the same trajectory.
The most canonical choice of q , known as hindsight experience replay (HER), selects q to be the uniform distribution.
Once a goal ˜ g is chosen, the reward label is also re-computed using the reward function assumed by the algorithm: r t := r ( s t , ˜ g ) .
B Proofs
In this section, we restate propositions and theorems in the paper and present their proofs.
B.1 Proof of Proposition 4.
Proposition B.1.
Assume for all g in support of p ( g ) , ∀ s, d O ( s ; g ) >
if p ( s ; g ) > 0 .
Then, for any f -divergence that upper bounds the KL-divergence,
[Equation]
Proof.
We first present and prove some technical lemma needed to prove this result.
The following lemmas and proofs are adapted from [35]; in particular, we extend these known results to the goal-conditioned setting.
Lemma B.1.
For any pair of valid occupancy distributions d 1 and d 2 , we have
[Equation]
[Equation]
Proof.
This lemma hinges on proving the following lemma first
[Equation]
Lemma B
[Equation] [Equation]
Proof.
[Equation]
Using this result
[Equation]
we can prove Lemma B
[Equation]
[Equation]
we assume that d O ( s ; g ) > 0 whenever p ( s ; g ) > 0 .
[Equation]
where the last step follows from Lemma B
[Equation]
Then, for any D f ≥ D KL [Equation] we have that −
[Equation]
Then, since E ( s,g ) ∼ d π ( s,g ) (cid:104) log
[Equation]
≥ 0 , we also obtain the following looser bound:
[Equation]
B.2 Proof of Proposition 4.
Proposition B.2.
The dual problem to (8) is
[Equation]
[Equation]
where f ⋆ denotes the convex conjugate function of f [Equation] V ( s ; g ) is the Lagrangian vector
[Equation]
and T V ( s, a ; g ) = E s ′ ∼ T ( ·| s,a ) [ V ( s ′ ; g )] .
Given the optimal V ∗ [Equation] the primal optimal d ∗ satisfies:
[Equation]
Proof.
We begin by writing the Lagrangian dual of the primal problem:
[Equation]
where p ( g ) V ( s ; g ) is the Lagrangian vector.
Then, we note that (cid:88)
[Equation]
Using this
[Equation]
we can rewrite (33) as
[Equation]
And finally,
[Equation]
Now [Equation] we make the key observation that the inner maximization problem in (36) is in fact the Fenchel conjugate of D f ( d ( s, a, g ) ∥
d O [Equation] s, a, g )) at r ( s, g ) + γ T V ( s, a, g ) − V ( s, g ) .
Therefore, we can reduce (36) to an unconstrained minimization problem over the dual variables
[Equation]
) and consequently, we can relate the dual-optimal V ∗ to the primal-optimal d ∗ using Fenchel duality (see Appendix A:
[Equation]
as desired.
B.3 Proof of Theorem
Theorem B.3.
Assume sup s,a,g d ∗ ( s,a ; g )
d O ( s,a ; g ) ≤ M and sup | r ( s, g )
| ≤ R max .
Consider a policy class Π : { S → ∆( A ) } such that π ∗ ∈ Π .
Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ ,
GoFAR (16) will return a policy ˆ π such that:
[Equation]
(39) where V π := E ( s,g ) ∼ ( µ 0 ,g )
[ V ( s ; g )] .
Proof.
We begin by deriving an upper bound using the performance difference lemma [1]:
[Equation]
[Equation]
Then, using standard algebraic manipulations [Equation] we have:
[Equation]
where the last step follows since sup s,a,g d ∗ ( s,a ; g )
d O ( s,a ; g ) ≤ M .
Now, we consider the simplified case when both π ∗ and ˆ π are deterministic; the stochastic case achieves the same bound with a more involved proof strategy (see Theorem 15.2 in [1].
The following proof snippet is adapted from [50].
First, we have that
[Equation]
Then, we can define the expected risk of a policy π to be
[Equation]
[Equation]
and its empirical risk to be
[Equation]
where ( s ( i ) , a ( i ) , g ( [Equation] ) ) ∼ d O (44) Now, for any error threshold ϵ > 0 , we want to bound the probability that L (ˆ π ) > ϵ .
Because ˆ π admits zero empirical risk, we have that
[Equation]
By an union bound argument, we have that
[Equation]
Setting the RHS to be equal to δ and combining with (41), we obtain that
[Equation]
[Equation]
C GoFAR Technical Details
In this section, we provide additional technical details of GoFAR that are omitted in the main text.
These include (1) detail of the GoFAR discriminator training, (2) mathematical expressions of GoFAR specialized to common f -Divergences, and (3) a full pseudocode.
C.1 Discriminator Training
Training the discriminator 6 in practice requires choosing p ( s ; g ) .
For simplicity, we set p ( s ; g ) to be the Dirac distribution centered at g : I ( ϕ ( s ) = g ) ; this precludes having to choose hyperparameters for p ( s ; g ) .
Once the discriminator has converged, we can retrieve the reward function r ( s ; g )
= log p ( s ; g )
d O ( s ; g )
= − log (cid:16) 1 c ∗ ( s ; g ) − 1 (cid:17) , since c ∗ ( s ; g ) =
[Equation]
C.2 GoFAR with common f -Divergences
GoFAR requires choosing a f -divergence.
Here, we specialize GoFAR to χ 2 -divergence as well as
KL-divergence as examples.
Our practical implementation uses χ 2 -divergence, which we found to be significantly more stable than KL-divergence (see Section 5.1).
Example 1 (GoFAR with χ 2 -divergence) .
f ( x ) = 12 ( x − 1) 2 , and we can show that f ⋆ ( x ) = 12 ( x + 1) 2 and f ′
⋆ ( x ) = x
+ 1 .
Hence, the GoFAR objective amounts to min
[Equation]
[Equation] 48) and
[Equation]
Example 2 (GoFAR with KL-divergence) .
We have f ( x ) = x log x and that D
⋆,f ( y ) =
log E
x ∼ q [exp y ( x )] [4].
Hence, the KL-divergence GoFAR objective is min
[Equation]
50) and
[Equation]
Now, we provide the full pseudocode for GoFAR implemented using χ 2 -divergence in Algorithm 2.
1: Require : Offline dataset d O , choice of f -divergence f , choice of p ( s ; g ) 2: Randomly initialize discriminator c ψ , value function V θ , and policy π ϕ .
Train Discriminator (Optional) 4: for number of discriminator iterations do 5:
Sample minibatch { s id , g i } Ni =1 ∼ d O 6:
Sample { s jg } Mj =1 ∼ p ( s ; g i ) ∀ i ∈ 1 . . .
N 7:
Discriminator objective: L c ( ψ ) = 1 N (cid:80) Ni
=1 [log(1 − c ψ ( s jd , g i ))
+ 1 M (cid:80) Mj
=1 [log c ψ ( s jg , g i )]] 8:
Update c ψ using SGD: c ψ ← c ψ − α c ∇L c ( ψ ) 9: end for 10: //
Train Dual Value Function 11: for number of value iterations do 12:
Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d O , { s i 0 }
Mi =1 ∼ µ 0 , { g i 0 }
Mi =1 ∼ d O 13:
If discriminator, obtain reward: r ( s it ; g it )
= − log (cid:16) 1 c ψ ( s it ; g it ) − 1 (cid:17) ∀ i = 1 . . .
N 14:
If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d O 15:
Value objective: L V ( θ ) = 1 − γ M (cid:80) Mi
=1 [ V θ ( s i 0 ; g i 0 )]
+ 1 N (cid:80) Ni =1 (cid:2)
f ⋆ ( r i t + γV
( s i t +1 ; g i t ) − V ( s i t ; g i t ))
(cid:3) 16:
Update V θ using SGD: V θ ← V θ − α V ∇L V ( θ ) 17: end for 18: // Train Policy With f -Advantage Regression 19: for number of policy iterations do 20:
Sample minibatch of offline data { s it , a it , s it +1 , g it } Ni =1 ∼ d
O 21:
If discriminator, obtain reward: r ( s it ; g it )
= − log (cid:16) 1 c ψ ( s it ; g it ) − 1 (cid:17) ∀ i = 1 . . .
N 22:
If no discriminator, obtain reward: { r ( s it ; g it ) } Ni =1 ∼ d
O 23:
Policy objective: L π ( ϕ ) =
(cid:80) Ni =1 (cid:2)(cid:0)
f ′ ⋆ (cid:0) r it + γV
θ ( s it +1 ; g it ) − V θ
( s it ; g it (cid:1)(cid:1) log π ( a | s, g ) (cid:3) 24:
Update π ϕ using SGD: π ϕ ← π ϕ − α π ∇L ( ϕ ) 25: end for
D GoFAR for Tabular MDPs
In Section 4.1 of the main text, we have stated that in tabular MDPs, GoFAR’s optimal dual value function (9) admits closed-form solution when we choose χ 2 -divergence.
Here, we provide a derivation of this result.
Recall the dual problem (1) min
[Equation]
To derive a closed-form solution, we rewrite the problem in vectorized notation; we borrow our notations from [35].
We first augment the state-space by concatenating the state dimensions and the goal dimensions so that the new state space ˜ S has dimension S + G .
Then, the new transition function, with slight abuse of notation, T (( s ′ , g ′ )
| ( s, g ) , a ) =
T ( s | s, a ) I ( g ′ =
g ) ; the new initial state distribution is thus µ 0 ( s, g )
= µ 0 ( s ) p ( g ) .
Therefore, ˜ T ∈ R | S
||
G ||
A |×| S || G | + and µ 0 ∈ R | S || G | + .
We assume that the offline dataset D O is collected by a behavior policy π b .
We construct a surrogate MDP ˆ M using maximum likelihood estimation; that is, ˆ T (( s ′ , g ′ )
| ( s, g ) , a ) =
n ( s,a,s ′ )
n ( s,a ) I
( g ′ = g ) , and we impute ˆ T (( s ′ , g ) | ( s, g ) , a ) = 1 S when n ( s, a ) = 0 .
Then, using ˆ M , we can compute d O ∈ R | S
||
G ||
A | + using linear programming and define reward r ∈ R | S || G | + as r ( s ; g )
= log p ( s ; g )
d O ( s ; g ) , where p ( s ; g ) ∈ R | S || G | + .
Now, define T ∈ R | S
||
G ||
A |×| S || G | such that ( T V )( s, a ; g ) =
(cid:80) s ′ T (( s ′ , g )
| ( s, g ) , a ) V ( s ′ ; g ) , where V ∈ R | S || G | + is the dual optimization variables.
We also define B ∈ R | S
||
G ||
A |×| S || G | such that ( B V )( s, a ; g ) =
V ( s ; g ) .
Finally, we define
D = diag
[Equation]
d O ) ∈
[Equation]
| S
||
G || A |×| S || G
||
A | .
Now, we can rewrite the dual problem as follow:
[Equation]
Now [Equation] we recognize that (53) is equivalent to Equation 49 in [35]
[Equation]
as we have reduced goal-conditioned
RL to regular RL with an augmented state-space.
Now
[Equation]
using the same derivation as in [35]
[Equation]
we have that
[Equation]
and we can recover d ∗ ( s, a ; g ) :
[Equation]
Given d ∗
[Equation]
we may extract the optimal policy π ∗ by marginalizing over actions:
[Equation]
E Additional Technical Discussion E.1 Connecting Goal-Conditioned State-Matching and Probabilistic GCRL
In Section 4. [Equation] we have hinted at a connection between GoFAR’s goal-conditioned state-matching objective and a probabilistic interpretation of GCRL.
In this section [Equation] we develop this connection mathematically.
We begin by proving the following result:
Proposition E.1.
Choose r ( s ; g )
=
p ( s ; g ) , the density of states that satisfy goal g .
Then,
[Equation]
Proof.
We have that
[Equation]
[Equation]
Rearranging the inequality gives the desired result.
This result shows that our objective lower bounds the log-transformation of the original GCRL objective using reward function r ( s ; g )
=
p ( s ; g ) , with the addition of a state entropy maximization term, which helps learning an unique and non-degenerate solution.
If we push log( · ) inside the expectations of J ( π ) (2) through Jensen’s inequality, we in fact obtain equality.
The resulting expression is
[Equation]
[Equation]
This expression can be understood as a probabilistic interpretation of GCRL.
To see this, for any r ( s ; g ) , we can choose p ( s ; g ) =
e r ( s ; g ) , corresponding to a softmax distribution with inputs r ( s ; g ) [40].
Therefore, solving the original GCRL objective plus maximizing the state entropy is equivalent to minimizing state-occupancy divergence with respect to the “expert” state-distribution e r ( s ; g ) .
E.2 Optimality Conditions for Hindsight Relabeling
In section 4.2, we have stated that HER is not optimal for most choices of reward functions.
In this section, we investigate conditions under which hindsight relabeling methods such as HER would be optimal.
Let the goal-relabeling distribution for HER be p HER ( g | s, a ) ; we do not specify the functional form of p HER ( g | s, a ) for generality (see 24).
Then, in order for this distribution to be optimal, then it must satisfy
[Equation]
Then, the choice of r ( s ; g ) such that this equality holds is the reward function for which HER would be optimal.
However, solving for r ( s ; g ) is generally challenging and we leave it to future work for investigating whether doing so is possible for general f -divergence coupled with neural networks.
This optimality condition is related to a prior work [11], which has found that hindsight relabeling is optimal in the sense of maximum-entropy inverse RL [54] for a peculiar choice of reward function (see Equation 9 in [11]), which cannot be implemented in practice.
Our result is more general as it applies to any choice of f -divergence, and is not restricted to the form of maximum-entropy inverse
RL.
E.3 Theoretical Comparison to Prior Regression-based GCRL methods
In section 4.3, we have stated that GoFAR’s theoretical guarantee (Theorem 4.1) is stronger in nature compared to prior regression-based GCRL methods.
Here, we provide an in-depth discussion.
Both GCSL [15] and WGCSL [51] prove that their objectives are lower bounds of the true RL objective (Theorem 3.1 in [15] and Theorem 1 in [51], respectively); however, in both works, the lower bounds are loose due to constant terms that do not depend on the policy and hence do not vanish to zero.
In contrast, GoFAR’s objective (5) is, by construction, a lower bound on the goal-conditioned state-matching objective, as it simply incorporates a f -divergence regularization.
If the offline data d O is on-policy , then our lower bound is an equality.
In contrast, even with on-policy data, the lower bounds in both GCSL and WGCSL are still loose due to the unavoidable constant terms.
GCSL also proves a sub-optimality guarantee (Theorem 3.2 in [15]) under the assumption of full state-space coverage.
Though full state-space coverage has been considered in some prior offline
RL works [25, 33], it is much stronger than the concentrability assumption in our Theorem 4.1, which only applies to d ∗ .
Furthermore, this guarantee is not statistical in nature, and instead directly makes a strong assumption on the maximum total-variance distance between π and optimal π ∗ for the
GCSL objective, which is difficult to verify in practice.
In contrast, our bound suggests asymptotic optimality: given enough offline data and assumptions satisfied, the solution to GoFAR’s policy objective will converge to π ∗ .
Finally, WGCSL proves a policy improvement guarantee (Proposition 1 in [51]) under their exponentially weighted advantage; the improvement is not a strict equality, and consequently there is no convergence guarantee to the optimal policy.
Furthermore, this result is not directly dependent on their use of an advantage function, so it is not clear the precise role of their advantage function in their algorithm.
F Task Descriptions
In this section, we describe the tasks in our experiments in Section 5.
F.1 Fetch Tasks
The Fetch environments involve the Fetch robot with the following specifications, developed by
Plappert et al.
• Seven degrees of freedom • Two pronged parallel gripper • Three dimensional goal representing Cartesian coordinates of target • Sparse, binary reward signal: 0 when at goal with tolerance 5cm and -1 otherwisejk • 25 Hertz simulation frequency • Four dimensional action space – Three Cartesian dimensions – One dimension to control gripper
F.1.1 Fetch Reach
The task is to place the end effector at the target goal position.
Observations consist of the end effector’s positional state, whether the gripper is closed, and the end effector’s velocity.
The reward is given by:
[Equation]
2 Fetch Push
The task is to push an object to the target goal position.
Observations consist of the end effector’s position, velocity, and gripper state as well as the object’s position, rotational orientation, linear velocity, and angular velocity.
The reward is given by:
[Equation]
In this task, the goal position lies outside of the robot’s reach and the robot must slide the puck-like object across the table to the goal.
Observations consist of the end effector’s position, velocity, and gripper state as well as the object’s position, rotational orientation, linear velocity, and angular velocity.
The reward is given by:
[Equation]
The task is to grasp the object and hold it at the goal, which could be on or above the table.
Observations consist of the end effector’s position, velocity, and gripper state as well as the object’s position, rotational orientation, linear velocity, and angular velocity.
The reward is given by:
[Equation]
F.2 Hand Reach
Uses a 24 DoF robot hand with a 20 dimensional action space.
Observations consist of each of thejoints’ positions and velocities.
The goal space is 15 dimensional corresponding to the positions of each of its five fingers.
The goal is achieved when the mean distance of the fingers to their goals is less than 1cm.
The reward is binary and sparse: 0 if the goal is reached and -1 otherwise, i.e.
[Equation]
F.3 D’ClawTurn (Simulation)
First introduced by Ahn et al.
[2], the D’Claw environment has a 9 DoF three-fingered robotic hand.
The turn task consists of turning the valve to a desired angle.
The initial angle is randomly chosen from [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − 2 ∗ π 3 , 2 ∗ π 3 ] .
The observation space is 21D, consisting of the current joint angles θ t , their velocities ˙ θ , angle between current and goal angle, and the previous action.
The environment terminates after 80 steps.
The reward function is defined as:
[Equation]
F.4 D’ClawTurn (Real)
To make real-world data collection easier, we slightly modify the initial and target angle distributions.
The initial angle is randomly chosen from [ − π 3 , π 3 ] ; the target angle is randomly chosen from [ − π 2 , π 2 ] .
Using this task distribution, collecting 400K transitions with random actions takes about 15 hours.
In
Figure 8, we also include a larger picture of the robot platform.
G Experimental Details
In this section, we provide experimental details omitted in Section 5 of the main text.
These include (1) technical details of the baseline methods, (2) hyperparameter and architecture details for all methods, (3) offline GCRL dataset details, and finally, (4) experimental details of the zero-shot transfer experiment.
G.1 Baseline Implementation Details
DDPG.
We use an open-source implementation of DDPG, which has already tuned DDPG on the set of Fetch tasks.
We implement all other methods on top of this implementation, keeping identical architectures and hyperparameters when appropriate.
The critic objective is
[Equation] [Equation]
where ¯ Q denotes the stop-gradient operation.
The policy objective is
[Equation]
DDPG updates the critic and the policy in an alternating fashion.
ActionableModel.
We implement AM on top of DDPG.
Specifically, we add a CQL loss in the critic update:
[Equation]
where d ˜ O is the distribution of the relabelled dataset.
In practice, we sample 10 random actions from the action-space to approximate this expectation.
Furthermore, we implement goal-chaining, where for half of the relabeled transitions in each minibatch update, the relabelled goals are randomly sampled from the offline dataset.
We found goal-chaining to not be stable in some environments, in particular, FetchPush, FetchPickAndPlace, and FetchSlide.
Therefore, to obtain better results, we remove goal-chaining for these environments in our experiments.
GCSL.
We implement GCSL by removing the DDPG critic component and changing the policy loss to maximum likelihood:
[Equation]
WGCSL.
We implement WGCSL on top of GCSL by including a Q-function.
The Q-function is trained using TD error as in DDPG.
To compute the advantage term, we follow the original
WGCSL algorithm by setting V ( s t ; g ) =
Q ( s t , π ( s t +1 , g ); g ) and then computing A ( s t , a t , g )
=
r ( s t ; g ) + γV ( s t +1 ; g ) − V ( s t ; g ) .
Using this, the WGCSL policy objective is
[Equation]
where we clip exp( · ) for numerical stability.
The original WGCSL uses different HER rates for the critic and the actor training.
To make the implementation simple and consistent with all other approaches, we use the same HER rate for both components.
G.2 Architectures and Hyperparameters
Each algorithm uses their own set of fixed hyperparameters for all tasks.
WGCSL, GCSL, and DDPG are already tuned on our set of tasks [43, 51], so we use the reported values from prior works; AM, in our implementation, shares same networks as DDPG, so we use DDPG’s values.
For GoFAR, we use identical hyperparameters as WGCSL because they share similar network components; GoFAR additionally trains a discriminator, for which we use the same architecture and learning rate as the value network.
We impose a small discriminator gradient penalty [17] to prevent overfitting.
For all experiments, We train each method for 3 seeds, and each training run uses 400k minibatch updates of size 512.
The architectures and hyperparameters for all methods are reported in Table 3.
[Table]
G.3 Offline GCRL Experiments
Datasets.
For each environment, the offline dataset composition is determined by whether data collected by random actions provides sufficient coverage of the desired goal distribution.
For
FetchReach and D’ClawTurn, we find this to be the case and choose the offline dataset to be 1 million random transitions.
For the other four tasks, random data does not capture meaningful goals, so we create a mixture dataset with 100K transitions from a trained DDPG-HER agent and 900K random transitions; the transitions are not labeled with their sources.
This mixture setup has been considered in prior works [22, 35] and is reminiscent of real-world datasets
[Table]
where only a small portion of the dataset is task-relevant but all transitions provide useful information about the environment
[Table]
G.4 Zero-Shot Transfer Experiments
We use GoFAR (Binary) variant for trainning the GoFAR planner.
The low-level controller is trained using an online DDPG algorithm on a narrow goal distribution, set to be closed to the object’s initial positions.
GoFAR Hierarchical Controller operates by first generating a sequence of subgoals ( g 1 , ..., g T ) using π high by recursively feeding the newest generated goal and conditioning on the final goal g .
Then, at each time step t , the low-level controller executes action π low ( a t | s t , g t ) .
The high-level subgoals are not re-planned during low-level controller execution.
We note that this is a simple planning algorithm, and improvement in performance can be expected by considering more sophisticated planning approaches
[Table]
H Additional Results H.1 Offline GCRL
In this section, we provide the full results table for dicounted return and final distance metrics, including error bars over 3 random seeds
[Table]
[Table] [Table]
H.2 Ablations
We also include the full task-breakdown table of GoFAR ablations presented in Figure 3 for completeness.
As shown in 6, GoFAR and GoFAR (HER) perform comparatively on all tasks.
GoFAR (binary) is slightly worse across tasks
[Table]
and GoFAR (KL) collapses due to the use of an unstable f -divergence
[Table] [Table]
H.3 Real-World Dexterous Manipulations
In our qualitative analysis
[Table]
we visualize all methods on a specific task instance of turning the valve prong (marked by the red strip) clockwise for 90 degree; the goal location is marked by the green strip.
The robot initial pose is randomized.
As shown in Figure 9, GoFAR reaches the goal with three random initial poses, whereas all baselines fail.
See the figure captions for detail.
Policy videos are included on the project website.
(a) GoFAR robustly achieved the goal with three random initial poses; in the first two runs, it demonstrates “recovery” behavior, as the robot would initially overshoot and then turn the valve counterclockwise.
In the last run, the robot initially undershoots and then turns again to reach the goal.
(b) Baselines fail to turn the volve prong (marked by the red strip) to the goal angle (marked by the green strip).
AM is the only method that is able to rotate the prong to some degree, though it overshoots in this case and exhibits unnatural behavior.
H.4 Zero-Shot Plan Transfer
We visualize GoFAR hierarchical controller and the plain low-level controller on three distinct goals in Figure 10.
See the figure caption for detail.
Policy videos are included on the project website.
[Figure]