R ETHINKING G OAL -C ONDITIONED S UPERVISED L EARNING AND I TS C ONNECTION TO O FFLINE RL
A BSTRACT
Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms.
A recent work, called Goal-Conditioned Supervised
Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences.
In this paper, we revisit the theoretical property of GCSL — optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel ofﬂine goal-conditioned RL algorithm.
The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) bestadvantage weight.
Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme.
The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and ofﬂine settings.
To evaluate algorithms in the ofﬂine goal-conditioned RL setting, we provide a benchmark including a
range of point and simulated robot domains.
Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art ofﬂine methods in the fully ofﬂine goal-conditioned setting.
I NTRODUCTION
Reinforcement learning (RL) enables automatic skill learning and has achieved great success in various tasks (Mnih et al., 2015;
Lillicrap et al., 2016; Haarnoja et al., 2018; Vinyals et al., 2019).
Recently, goal-conditioned RL is gaining attention from the community, as it encourages agents to reach multiple goals and learn general policies (Schaul et al., 2015).
Previous advanced works for goal-conditioned RL consist of a variety of approaches based on hindsight experience replay (Andrychowicz et al., 2017; Fang et al., 2019), exploration (Florensa et al., 2018;
Ren et al., 2019; Pitis et al., 2020), and imitation learning (Sun et al., 2019;
Ding et al., 2019; Sun et al., 2020;
Ghosh et al., 2021).
However, these goal-conditioned methods need intense online interaction with the environment, which could be costly and dangerous for real-world applications.
A direct solution to learn goal-conditioned policy from ofﬂine data is through imitation learning.
For instance, goal-conditioned supervised learning (GCSL) (Ghosh et al., 2021) iteratively relabels collected trajectories and imitates them directly.
It is substantially simple and stable, as it does not require expert demonstrations or value estimation.
Theoretically, GCSL guarantees to optimize over a lower bound of the objective for goal reaching problem.
Although hindsight relabeling (Andrychowicz et al., 2017) with future reached states can be optimal under certain conditions (Eysenbach et al., 2020), it would generate non-optimal experiences in more general ofﬂine goal-conditioned RL set- ting, as discussed in Appendix B.1.
As a result, GCSL suffers from the same issue as other behavior cloning methods by assigning uniform weights for all experiences and results in suboptimal policies.
To this end, leveraging the simplicity and stability of GCSL, we generalize it to the ofﬂine goal-conditioned RL setting and propose an effective and theoretically grounded method, named
Weighted GCSL (WGCSL).
We ﬁrst revisit the theoretical foundation of GCSL by additionally considering discounted rewards, and this allows us to obtain a weighted supervised learning objective.
To learn a better policy from the ofﬂine dataset and promote learning efﬁciency, we introduce a more general weighting scheme by considering the importance of different relabeling goals and the expected return estimated with a value function.
Theoretically, the discounted weight for relabeling goals contributes to optimizing a tighter lower bound than GCSL.
Based on the discounted weight, we show that additionally re-weighting with an exponential function over advantage value guarantees monotonic policy improvement for goal-conditioned RL.
Moreover, the introduced weights build a natural connection between WGCSL and ofﬂine RL, making WGCSL available for both online and ofﬂine setting.
Another major challenge in goal-conditioned RL is the multi-modality problem (Lynch et al., 2020), i.e., there are generally many valid trajectories from a state to a goal, which can present multiple counteracting action labels and even impede learning.
To tackle the challenge, we further introduce the best-advantage weight under the general weighting scheme, which contributes to the asymptotic performance when the data is multi-modal.
Although WGCSL has the cost of learning the value function, we empirically show that it is worth for its remarkable improvement.
For the evaluation of ofﬂine goal-conditioned RL algorithms, we provide a public benchmark and ofﬂine datasets including a set of challenging multi-goal manipulation tasks with a robotics arm or an anthropomorphic hand.
Experiments 1 conducted in the introduced benchmark show that WGCSL signiﬁcantly outperforms other state-of-the-art baselines in the fully ofﬂine goalconditioned settings, especially in the difﬁcult anthropomorphic hand task and when learning from randomly collected datasets with sparse rewards.
P RELIMINARIES
Markov Decision Process and ofﬂine RL The RL problem can be described as a Markov Decision Process (MDP), denoted by a tuple ( S , A , P , r, γ ) , where S and A are the state and action spaces; P describes the transition probability as S ×
A × S → [0 , 1] ; r : S ×
A → R is the reward function and γ ∈ (0 , 1] is the discount factor; π : S → A denotes the policy, and an optimal policy π ∗ satisﬁes
π
[Equation]
=
[Equation]
max
[Equation]
E s
[Equation]
( s 1 ) ,
[Equation]
t ∼
[Equation] [Equation]
| s t ) ,s
[Equation]
1 ∼P ( ·
[Equation]
t ,
[Equation]
t ) [ (cid:80) ∞ t =1 γ t − 1 r ( s t , a t )] , where ρ ( s 1 ) is the distribution of initial states.
For ofﬂine RL problems, the agent can only access a static dataset
D , and is not allowed to interact with the environment during the training process.
The ofﬂine data can be collected by some unknown policies.
Goal-conditioned RL Goal-conditioned RL further considers a goal space G .
The policy π : S × G → A and the reward function r : S × G × A → R are both conditioned on goal g ∈ G .
The agent learns to maximize the expected discounted cumulative return J ( π ) =
E g ∼ p
( g ) ,s 1 ∼
[Equation]
( s 1 ) ,a t ∼ π ( ·| s t ) ,s t +1 ∼P ( ·| s t ,a t ) (cid:2) (cid:80) ∞ t =1 γ t − 1 r ( s t , a t , g ) (cid:3) , where p ( g ) is the distribution over goals g .
We study goal-conditioned RL with sparse rewards, and the reward function is typically deﬁned as:
[Equation]
t ) − g
||
22 <
some threshold 0 , otherwise , where φ : S → G is a mapping from states to goals.
For theoretical analysis in Section 3, we consider
MDPs with discrete state/goal space, such that the reward function can be written as r ( s t , a t , g ) =
1[ φ ( s t )
=
g ] , which provides a positive signal only when s t achieves g .
The value function V :
S × G → R is deﬁned as V π ( s, g ) =
E a t ∼ π,s t +1 ∼P ( ·| s t ,a t ) (cid:2) (cid:80)
∞ t =1 γ t − 1 r ( s t , a t , g ) | s 1 = s (cid:3) .
Goal-conditioned Supervised Learning Different with goal-conditioned RL that maximizes discounted cumulative return, GCSL considers the goal-reaching problem, i.e., maximizing the last-step reward E g ∼ p
( g ) ,τ ∼ π (cid:2) r ( s T , a T , g ) (cid:3) for trajectories τ of horizon T .
GCSL iterates between two process, relabeling and imitating.
Trajectories in the replay buffer are relabeled with
[Figure]
hindsight methods (Kaelbling, 1993; Andrychowicz et al., 2017) to form a relabeled dataset
D relabel = { ( s t , a t , g (cid:48) ) } , where g (cid:48) =
φ ( s i ) for i ≥ t denotes the relabeled goal.
The policy is optimized through supervised learning to mimic those relabeled transitions: θ =
[Equation]
W EIGHTED G OAL -C ONDITIONED S UPERVISED L EARNING
In this section, we will revisit GCSL and introduce the general weighting scheme which generalizes
GCSL to
ofﬂine goal-conditioned RL.
3. R EVISITING G OAL - CONDITIONED S UPERVISED L EARNING
As an imitation learning method, GCSL sticks the agent’s policy to the relabeled data distribution, therefore it naturally alleviates the problem of out-of-distribution actions.
Besides, it has the potential to reach any goal in the ofﬂine dataset with hindsight relabeling and the generalization ability of neural networks.
Despite its advantages, GCSL has a major disadvantage for ofﬂine goalconditioned RL, i.e., it only considers the last step reward r ( s T , a T , g ) and generally results in suboptimal policies.
A Motivating Example
We provide training results of GCSL in the PointReach task in Figure 2.
The objective of the task is to move a point from the starting position to the desired goal as quickly as possible.
The ofﬂine dataset is collected using a random policy.
As shown in Figure 2, GCSL learns a suboptimal policy which detours to reach goals.
This is because GCSL only considers the last step reward r ( s T , a T , g ) .
As long as the trajectories reaching the goal at the end of the episode, they are all considered equally to GCSL.
To improve the learned policy, a straightforward way is to evaluate the importance of samples or trajectories for policy learning using importance weights.
As a comparison, Weighted GCSL (WGCSL), which uses a novel weighting scheme, learns the optimal policy in the PointReach task.
We will derive the formulation of
WGCSL in the following analysis.
Connection between Goal-Conditioned RL and SL
GCSL has been proved to optimize a lower bound on the goal-reaching objective (Ghosh et al., 2021), which is different from the goal-conditioned RL objective that optimizes the cumulative return.
In this section, we will introduce a surrogate function for goal-conditioned RL and then derive the connection between goal-conditioned RL and weighted goal-conditioned supervised learning (WGCSL).
For overall consistency, we provide the formulation of WGCSL as below:
[Equation]
where p ( g ) is the distribution of desired goals and π b refers to the data collecting policy.
The weight w t,i is subscripted by both the time step t and the index i of the relabeled goal.
When w t,i = 1 ,
J WGCSL reduces to GCSL (Ghosh et al., 2021), and we denote the corresponding objective as
J GCSL for comparison convenience.
Note that in Eq. 1, φ ( s i ) , i ≥ t implies that any future visited states after s t can be relabeled as goals.
Theorem 1.
Assume a ﬁnite-horizon discrete MDP, a stochastic discrete policy π which selects actions with non-zero probability and a sparse reward function r ( s t , a t , g ) =
1[ φ ( s t )
= g ] , where φ is the state-to-goal mapping and 1[ φ ( s t ) =
g ] is an indicator function.
Given trajectories τ
=
( s 1 , a 1 , · · · , s T , a T ) and discount factor γ ∈ (0 , 1] , let the weight w t,i = γ
i − t , t ∈ [1 , T ] , i ∈
[ t, T ] , then the following bounds hold:
[Equation]
where J
[Equation] [Equation]
is a surrogate function of J ( π ) .
We defer the proof to Appendix B.2, where we also show that under mild conditions, (1) J surr is a lower bound of log J , and (2) J surr shares the same gradient direction with J at π b .
Theorem 1 reveals the connection between the goal-conditioned RL objective and the WGCSL/GCSL objective.
Meanwhile, it suggests that GCSL with the discount weight γ
i − t is a tighter lower bound compared to the unweighted version.
We name this weight as discounted relabeling weight (DRW) as it intuitively assigns smaller weights on longer trajectories reaching the same relabeled goal.
3. A M ORE G ENERAL W EIGHTING S CHEME
DRW can be viewed as a weight evaluating the importance of relabeled goals.
In this subsection, we consider a more general weighting scheme using a weight function measuring the importance of state-action-goal combination, revealed by the following corollary.
Corollary 1.
Suppose function h ( s, a, g ) ≥ 1 over the state-action-goal combination.
Given trajectory τ = ( s 1 , a 1 , · · · , s T , a T ) , let w t,i = γ
i − t h
( s t , a t , φ ( s i )) , t ∈ [1 , T ] , i ∈ [ t, T ] , then the following bound holds:
[Equation]
The proof are provided in Appendix B.3.
Naturally, the Q-value in RL with some constant shift is a candidate of the function family h ( s, a, g ) .
Inspired by prior ofﬂine RL approaches (Wang et al., 2018; Peng et al., 2019; Nair et al., 2020), we choose h ( s, a, g ) =
exp( A ( s, a, g )+ C ) , an exponential function over the goal-conditioned advantage, which we refer to as goal-conditioned exponential advantage weight (GEAW).
The constant C is used to ensure that h ( s, a, g ) ≥ 1 .
The intuition is that GEAW assigns larger weight for samples with higher values.
In addition, the exponential advantage weighted formulation has been demonstrated to be a closed-form solution of an ofﬂine RL problem, where the learned policy is constrained to stay close to the behavior policy (Wang et al., 2018).
Compared to DRW, GEAW evaluates the importance of state-action-goal samples, taking advantage of the universal value function (Schaul et al., 2015).
To tackle the multi-modality challenge in goal-conditioned RL (Lynch et al., 2020), we further introduce the best-advantage weight (BAW) based on the learned value function.
BAW has the following form:
[Equation]
where ˆ A is a threshold, and (cid:15) min is a small positive value.
Implementation details of BAW can be found in Section 4.
In our implementation, ˆ A gradually increases.
Therefore, BAW gradually leads the policy toward the modal with the highest return rather than sticks to a position between multiple modals, especially when ﬁtting using a Gaussian policy.
Combining all three introduced weights, we have the weighting form: w t,i
[Equation]
s t ,
[Equation]
t ,
[Equation]
s
[Equation] [Equation]
( A
[Equation]
s t , a t , φ ( s i ))) .
The constant
C only introduces a coefﬁcient independent of t, i for all transitions, therefore, we simply omit it for further analysis.
3. P OLICY I MPROVEMENT VIA WGCSL
We formally show that combining the three introduced weights, WGCSL can consistently improve the policy learned from the ofﬂine dataset.
First, we assume there exists a policy π relabel that can produce the relabeled experiences D relabel .
GCSL is essentially equivalent to imitating π relabel :
[Equation]
Following (Wang et al., 2018), solving WGCSL is equivalent to imitating a new goal-conditioned policy ˜ π deﬁned as:
[Equation]
Then, we have the following proposition:
Proposition 1.
Under certain conditions, ˜ π is uniformly as good as or better than π relabel .
That is, ∀ s t , φ ( s i ) ∈ D relabel , i ≥ t , V ˜ π
( s t , φ ( s i )) ≥ V π relabel ( s t , φ ( s i )) , where D relabel contains the experiences after goal relabeling.
The proof can be found in Appendix B.4.
Proposition 1 implies that WGCSL can generate a uniformly non-worse policy using the relabeled data D relabel .
To be rigorous, we also need to discuss the relationship between the relabeled return and the original return.
In fact, there is a monotonic improvement for the relabeled return over the original one when relabeling with inverse RL (Eysenbach et al., 2020).
Below, we provide a simpler relabeling strategy which also offers monotonic value improvement.
Proposition 2.
Assume the state space can be perfectly mapped to the goal space, ∀ s t ∈ S, φ ( s t ) ∈
G , and the dataset has sufﬁcient coverage.
We deﬁne a relabeling strategy for ( s t , g ) as
[Equation]
We also assume that when g (cid:48) (cid:54) = g , the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g (cid:48) in the original dataset D .
Then, π relabel is uniformly as good as or better than π b , i.e., ∀ s t , g, ∈ D, V π relabel ( s t , g ) ≥
V π b ( s t , g ) , where π b is the behavior policy forming the dataset D .
The proof is provided in Appendix B.5.
In Proposition 2, the relabeling strategy is slightly different from the strategy of relabeling with random future states (Andrychowicz et al., 2017).
When there is no successful future states, the relabeling strategy in Proposition 2 randomly samples from the future visited states for relabeling, otherwise it keeps the original goal.
For random datasets, there are rare successful states and the relabeling strategy in Proposition 2 acts similarly to relabeling with random future states.
Combining Propositions 1 and 2, we know that under certain assumptions, after performing relabeling and weighted supervised learning, the policy can be monotonically improved.
In fully ofﬂine settings, WGCSL is able to learn a better policy than the policy learned by GCSL and the behavior policy generating the ofﬂine dataset.
A LGORITHM
In this section, we summarize our proposed algorithm.
Denote the relabeled dataset as D relabel =
( s t , a t , r (cid:48) t , g (cid:48) ) , t ∈ [1 , T ] , g (cid:48) =
φ ( s i ) , r (cid:48) t
= r
( s t , a t , φ ( s i )) , i ≥ t , and we maximize the following
WGCSL objective based on the relabeled data
[Equation]
where the weight w t,i is composed of three parts conforming to the general weighting scheme
[Equation]
(1) γ i − t is the discounted relabeling weight (DRW) as introduced in Section 3.1.
(2) exp clip ( A ( s t , a t , φ ( s i )))
=
clip(exp( A ( s t , a t , φ ( s i ))) , 0 , M )
is the goal-conditioned exponential advantage weight (GEAW) with a clipped range (0 , M ] , where M is some positive number for numerical stability.
In our experiments, M is set as 10 for all the tasks.
[Figure]
[Figure]
) (c) (d) (e) (f) (g) (h) (i) (j)
(3) (cid:15)
( A ( s t , a t , φ ( s i ))) is the best-advantage weight (BAW) with the form in Eq.
In our experiments, ˆ A is set as N percentile of recent advantage values and (cid:15) min is set as 0.05.
Motivated by curriculum learning (Bengio et al., 2009), N gradually increases from 0 to 80 , i.e., we learn from all the experiences in the beginning stage and converge to learning from samples with the top 20% advantage values.
The advantage value in Eq.
3 can be estimated by A ( s t , a t , g (cid:48) )
=
r ( s t , a t , g (cid:48) )
+ γV ( s t +1 , g (cid:48) )
−
V ( s t , g (cid:48) ) , and we learn a Q-value function Q ( s t , a t , g (cid:48) ) by minimizing the TD error
[Equation]
where we relabel original goals g as g (cid:48)
=
φ ( s i ) with a probability of 80% and remain the original goal unchanged for the rest 20% time.
For simplicity, in our experiments, once it falls into the 80% probability of goal relabeling, we use purely relabeling with random future states, no matter whether there exist successful future states in the trajectory or not, to avoid parsing all the future states as discussed in Appendix B.5.
ˆ Q refers to the target network which is slowly updated to stabilize training.
The same value function training method is adopted in (Andrychowicz et al., 2017; Fang et al., 2019).
The value function V ( s t , g (cid:48) ) is estimated using Q-value function as V ( s t , g (cid:48) )
=
Q ( s t , π ( s t +1 , g (cid:48) ) , g (cid:48) ) .
The overall framework of WGCSL is presented in Figure 1 and the entire algorithm is provided in Appendix C.
E XPERIMENTS
In our experiments, we demonstrate our proposed method on a diverse collection of continuous sparse-reward goal-conditioned tasks.
5. E NVIRONMENTS AND E XPERIMENTAL S ETTINGS
As shown in Figure 3, the introduced benchmark includes two point-based environments, and eight simulated robot environments.
In all tasks, the rewards are sparse and binary:
the agent receives a reward of +1 if it achieves a desired goal and a reward of 0 otherwise.
More details about those environments are provided in Appendix F. For the evaluation of ofﬂine goal-conditioned algorithms, we collect two types of ofﬂine dataset, namely ’random’ and ’expert’, following prior ofﬂine RL work (Fu et al., 2020).
Each dataset has the sample size of 2 × 10 6 for hard tasks (FetchPush,
FetchSlide, FetchPick and HandReach) and 1 × 10 5 for other relatively easy tasks.
The ’random’ dataset is collected by a uniform random policy, while the ’expert’ dataset is collected by the ﬁnal policy trained using online HER.
We also add Gaussian noise with zero mean and 0 .
2 standard deviation to increase the diversity of the ’expert’ dataset.
This may lead to the consequence that the deterministic policy learned by behavior cloning achieves a higher average return than that in the dataset, which has also been observed in previous works (Kumar et al., 2019; Fujimoto et al., 2019).
More details about the two datasets are provided in Appendix D. Beyond the ofﬂine setting, we also evaluate our method in the online setting and the results are reported in Appendix E.8.
As for the implementation of GCSL and WGCSL, we utilize the Diagonal Gaussian policy with a mean vector π θ ( s t , g ) and a constant variance σ 2 for continuous control.
In the ofﬂine setting, σ does not need to be deﬁned explicitly because we evaluate the policy without randomness.
The loss function of GCSL can be rewritten as: L GCSL =
[Equation]
relabel [
[Equation]
Our implemented WGCSL learns a policy to minimize the following loss
[Equation]
[Table]
where w t
[Table]
i is deﬁned in Eq 4.
The policy networks of GCSL and WGCSL are 3-layer MLP with 256 units each layer and relu activation.
Besides
[Table]
WGCSL learns a value network with the same structure except for the input and output layers.
The batch size is set as 128 for the ﬁrst 6 tasks
[Table]
and 512 for 4 harder tasks from FetchPush to HandReach.
We use Adam optimizer with a learning rate of 5 × 10 − 4 .
For all the experiments we repeat for 5 different random seeds and report the average evaluation performance with standard deviation.
5. E XPERIMENTAL R ESULTS
In the ofﬂine goal-conditioned setting, we compare WGCSL with GCSL , HER and modiﬁed versions of most related ofﬂine works such as goal-conditioned MARWIL (Wang et al.,
( Goal MARWIL ), goal-conditioned behavior cloning (Bain & Sammut, 1995) ( Goal BC ), goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019)
( Goal BCQ ), and goalconditioned Conservative Q-learning (Kumar et al., 2020)
( Goal CQL ).
The results are presented in
Table 1 and Figure 4.
Additional comparison results with more baselines and results on 10 % dataset are provided in Appendix E.
Expert Datasets The top row of Figure 4 reports the performance of different algorithms using the expert datasets of four hard tasks.
In all these tasks, WGCSL outperforms other baselines in terms of both learning efﬁciency and ﬁnal performance.
GCSL converges very slowly but can ﬁnally reach beyond the average return of expert dataset in FetchSlide and FetchPick tasks.
Goal MARWIL is more efﬁcient than GCSL and achieves the second best performance.
However, none of the baseline algorithms succeeds in the HandReach task, which might be due to high-dimensional states, goals, actions and the noise used to collect the dataset.
In addition, HER itself has the ability to handle ofﬂine dataset as the relabeled data is far off-policy (Plappert et al., 2018).
But we can conclude from Table 1 that the performance of HER is unstable and inconsistent across different tasks.
In contrast, learning curves of WGCSL are consistently stable.
It is also veriﬁed that techniques used in WGCSL are effective to generate better policies than original and relabeled datasets in ofﬂine goal-conditioned RL setting.
Random Datasets
The bottom row of Figure 4 shows the training results of different methods on random datasets.
We can observe that WGCSL outperforms other baselines by a large margin.
Baselines such as Goal MARWIL, Goal BC, Goal BCQ and Goal CQL can hardly learn to reach goals with the random datasets.
GCSL and HER can learn relatively good policies in some tasks, which proves the importance of goal relabeling.
Interestingly, in tasks such as Reacher, SawyerReach and
FetchReach, results of WGCSL and HER on the random dataset exceeds that of the expert dataset.
The possible reason is that the data coverage of random dataset could be larger than the expert dataset in these tasks, and the agent can learn to reach more goals via goal relabeling.
Ablation Studies We also include ablation studies for ofﬂine experiments in Figure 5.
In our ablation studies, we investigate the effect of the three proposed weights, i.e., DRW, GEAW and BAW, by applying them to GCSL respectively.
The results demonstrate that all three weights are effective compared to the plain GCSL, while GEAW and BAW show larger beneﬁts.
This is reasonable since GEAW and BAW learn additional value networks while DRW does not.
Moreover, the results suggest that the learned policy can be improved by combining these weights.
Value Estimation Off-policy RL algorithms are prone to distributional shift and suffer from value overestimation problem on out-of-distribution actions.
Such problem can be exacerbated in tasks with high-dimensional states and actions.
As shown in Figure 6, DDPG and HER exhibit large estimated values during ofﬂine training.
In contrast, WGCSL has a more robust value approximation through weighted supervised learning, thus achieving higher returns in the difﬁcult HandReach task.
R ELATED W ORK
Solving goal-conditioned RL task is an important yet challenging domain in reinforcement learning.
In those tasks, agents are required to achieve multiple goals rather than a single task (Schaul et al., 2015).
Such setting puts much burden on the generalization ability of the learned policy when dealing with unseen goals.
Another challenge arises in goal-conditioned RL is the sparse reward (Plappert et al., 2018).
HER (Andrychowicz et al., 2017) tackles the sparse reward issue via relabeling the failed rollouts as successful ones to increase the proportion of successful trails during learning.
HER is then extended to deal with demonstration (Nair et al., 2018; Ding et al., 2019) and dynamic goals (Fang et al., 2018), and combined with curriculum learning for exploration-exploitation trade-off (Fang et al., 2019).
Eysenbach et al.
(2020) and Li et al.
(2020) further improved the goal sampling strategy from the perspective of inverse RL.
Zhao et al.
(2021) encouraged agents to control their states to reach goals by optimizing a mutual information objective without external rewards.
Different from prior works, we consider the ofﬂine setting, where the agent cannot access the environment to collect data in the training phase.
Our work has strong connections to imitation learning (IL).
Behavior cloning is one of the simplest
IL methods that learns a policy predicting the expert actions (Bain & Sammut, 1995; Bojarski et al., 2016).
Inverse RL (Ng et al., 2000) extracts a reward function from demonstrations and then uses the reward function to train policies.
Several works also leverage self-imitation to improve the stability and efﬁciency of RL without expert experience.
Self-imitation learning (Oh et al., 2018) imitates the agent’s past advantageous decisions and indirectly drives exploration.
Sun et al.
(2019) introduced a dynamic programming method to learn policies progressively via self-imitation.
Lynch et al.
(2020) employed VAE to jointly learn a latent plan representation and goal-conditioned policy from replay without the need for rewards.
Our work is most relevant to GCSL (Ghosh et al., 2021), which iteratively relabels and imitates its own collected experiences.
The difference is that we further generalize GCSL to the ofﬂine goal-conditioned RL setting via weighted supervised learning.
Our work belongs to the scope of ofﬂine RL.
Learning policies from static ofﬂine datasets is challenging for general off-policy RL algorithms due to error accumulation with distributional shift (Fujimoto et al., 2019; Kumar et al., 2019).
To address such issue, a series of algorithms are proposed to constrain the policy updates to avoid excessive deviation from the data distribution (Wang et al., 2018; Fujimoto et al., 2019; Wu et al., 2019; Yang et al., 2021).
MARWIL (Wang et al., 2018) can learn a better policy than the behavior policy of the dataset with theoretical guarantees.
In addition to policy regularization, other works leverage techniques such as value underestimation (Kumar et al., 2020; Yu et al., 2020), robust value estimation (Agarwal et al., 2020), data augmentation (Sinha &
Garg, 2021; Wang et al., 2021), and Expectile V -Learning (Ma et al., 2021) to handle distribution shift.
As ofﬂine RL can easily overﬁt to one task, generalizing ofﬂine RL to unseen tasks can be even more challenging (Li et al., 2021).
Li et al.
(2019) tackled the challenge by leveraging the triplet loss for robust task inference.
Our method has similar exponential advantage weighted form as MARWIL, however, we tackle goal-conditioned tasks and introduces a general weighting scheme for WGCSL.
A most recent work (Chebotar et al., 2021) also addresses the ofﬂine goal-conditioned problem through underestimating the value of unseen actions.
The difference is that WGCSL is a weighted supervised learning method without explicitly value underestimation.
Therefore, WGCSL is substantially simpler, more stable, and more applicable for real-world tasks.
C ONCLUSION
In this paper, we have proposed a novel algorithm, Weighted Goal-Conditioned Supervised Learning (WGCSL), to solve ofﬂine goal-conditioned RL problems with sparse rewards, by ﬁrst revisiting GCSL and then generalizing it to ofﬂine goal-conditioned RL.
We further derive theoretical guarantees to show that WGCSL can generate monotonically improved policies under certain conditions.
Experiments on an introduced benchmark demonstrate that WGCSL outperforms current ofﬂine goal-conditioned approaches by a great margin in terms of learning efﬁciency and convergent performance.
For future directions, we are interested in building more stable and efﬁcient ofﬂine goal-conditioned RL algorithms based on WGCSL for real-world applications.
A CKNOWLEDGMENTS
This work was partly supported by the Science and Technology Innovation 2030-Key Project under
Grant 2021ZD0201404, in part by Science and Technology Innovation 2030 – “New Generation
Artiﬁcial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (No. 20211300509).
Part of this work was done when Rui Yang and Hao Sun worked as interns at Tencent Robotics X Lab.
E THICS AND R EPRODUCIBILITY S TATEMENTS
Our work provides a way to learn general policy from ofﬂine data.
Learning policy for general purpose is one of the fundamental challenges in artiﬁcial general intelligence (AGI).
The proposed ofﬂine goal-conditioned algorithm can provide new insights into training agents with diverse skills, but currently we cannot foresee any negative social impact of our work.
The algorithm is evaluated on simulated robot environments, thus our experiments would not suffer from discrimination/bias/fairness concerns.
The details of experimental settings and additional results are also provided in the Appendix.
All the implementation code and ofﬂine dataset are available in our code link: https://github.com/YangRui2015/AWGCSL.
A A C OMPREHENSIVE S UMMARY FOR WGCSL
WGCSL is strongly efﬁcient in the ofﬂine goal-conditioned RL due to the following advantages:
(1) WGCSL learns a universal value function (Schaul et al., 2015) which is more generalizable than the value function learned in a single task, thus alleviating the overﬁtting problem in ofﬂine RL.
(2) WGCSL utilizes goal relabeling which contributes to augment huge amount of data ( (cid:0) T 2 (cid:1) times) for ofﬂine training and meanwhile alleviates the sparse reward issue.
(3) WGCSL leverages weighted supervised learning to reweight the action distribution in the dataset, avoiding the out-of-distribution action problem in ofﬂine RL.
In the three weights, GEAW keeps the learned policy close to the ofﬂine dataset implicitly and BAW handles the multi-modality problem in ofﬂine goal-conditioned RL.
WGCSL utilizes the three weights to learn a better policy with theoretical guarantees.
B T HEORETICAL R ESULTS
In our analysis, we will assume ﬁnite-horizon MDPs with discrete state and goal spaces.
B. N ON - OPTIMALITY OF RELABELING STRATEGY
Given a trajectory τ = { s 1 , a 1 , . . .
, s T , a T } , we show that under general ofﬂine goal-conditioned
RL setting with the sparse reward function r ( s t , a t ) =
1[ φ ( s t )
=
g ] , relabeling with future states is not optimal.
This is a direct result from Eq.
7 of (Eysenbach et al., 2020), which has proven that the optimal relabelling strategy is
[Equation]
This is different from simply relabeling with the last state as q ( g | τ ) =
[ g = s T ] in Section 4.
of (Eysenbach et al., 2020) and relabeling with future states as q ( g | s t , a t ) =
[Equation]
1[ g
[Equation]
s i ]
[Equation]
t ∈ [1 , T ] , i ∈
[ t, T ] .
B. P ROOF OF T HEOREM
Theorem.
Assume a ﬁnite-horizon discrete MDP, a stochastic discrete policy π which selects actions with non-zero probability and a sparse reward function r ( s t , a t , g ) =
1[ φ ( s t )
= g ] , where φ is the state-to-goal mapping and 1[ φ ( s t ) =
g ] is an indicator function.
Given trajectory τ = ( s 1 , a 1 , · · · , s T , a T ) and discount factor γ ∈ (0 , 1] , let the weight w t,i = γ
i − t , t ∈ [1 , T ] , i ∈
[ t, T ] , then the following bounds hold:
[Equation]
where
[Equation]
is a surrogate function of J ( π ) .
Proof.
[Equation]
Eq.
5 holds because of the non-positive logarithmic function over π .
Compared to the surrogate function used in Appendix B.1 of (Ghosh et al., 2021), J surr ( π ) further considers the discounted cumulative return.
The theorem shows that WGCSL optimizes over a tighter lower bound of J surr ( π ) compared to GCSL.
Remark.
While Theorem 1 bounds the surrogate function J surr with J WGCSL and J GCSL , here we bridge the gap between J surr and the goal-conditioned RL objective J .
We ﬁrst show that log J is an upper bound of J surr in addition to KL divergence and a constant.
Let
R ( τ ) =
[Equation]
Ti =1 γ
i − 1 · 1[ φ ( s i ) =
g ] and C ( g ) =
(cid:82) π b ( τ | g ) R ( τ ) d τ , and assume C ( g ) > 0 .
Since practically g is sampled from ofﬂine dataset, we can ﬁnd C m = min g C ( g ) .
Similarly to (Kober & Peters, 2014), we have following results using Jensen’s inequality and absorbing constants into ’constant’:
[Equation]
Naturally, in ofﬂine setting, we explicitly or implicitly bound the KL divergence between π b and π via policy constraints or (weighted) supervised learning.
Therefore, we can regard J surr as a lower bound of log J .
Furthermore, We show that J surr ( π ) shares the same gradient with J ( π ) when π = π b after scaling with 1 T .
[Equation]
Therefore, taking a gradient update with a proper step size on J surr can also improve the policy on the objective J .
In our algorithm, the exponential weight poses an implicit constraint on the policy update similar to (Wang et al., 2018; Nair et al., 2020).
With a proper constraint, optimizing on the J surr is equivalent as optimizing on the goal-conditioned objective J .
Intuitively, J surr maximizes the likelihood for state-action pairs with large returns, and therefore it can lead to policy improvement based on the ofﬂine dataset.
B. P ROOF OF C OROLLARY
Corollary.
Suppose function h ( s, a, g ) ≥ 1 over the state-action-goal combination.
Given trajectory τ = ( s 1 , a 1 , · · · , s T , a T ) , let w t,i = γ
i − t h
( s t , a t , φ ( s i )) , t ∈ [1 , T ] , i ∈ [ t, T ] , then the following bounds hold:
[Equation]
Proof.
Similarly to the proof of Theorem 1, we have
[Equation]
B. P ROOF OF P ROPOSITION
Proposition.
Under certain conditions, ˜ π is uniformly as good as or better than π relabel .
That is, ∀ s t
, φ ( s i ) ∈ D relabel and i ≥ t , V ˜ π ( s t , φ ( s i )) ≥ V π relabel ( s t , φ ( s i )) , where D relabel contains the experiences after goal relabeling.
Proof.
We assume the advantage value is for the behavior policy of the relabeled data, i.e., π relabel .
We consider the compound state ˆ s =
( s t , φ ( s i )) from the relabeled distribution D relabel .
When ˆ s is ﬁxed (the index of i, t are ﬁxed), γ i − t is a constant and we have:
[Equation]
f (ˆ s, · ) is a monotonically non-decreasing function for ﬁxed ˆ s , where C (ˆ s ) is a normalizing factor,
M is the positive maximum exponential advantage value, and (cid:15)
( A (ˆ s, a t )) is also a non-decreasing function introduced in Section 3.2 with minimum (cid:15) min > 0 .
Leveraging the Proposition 1 in MARWIL (Wang et al., 2018), we can conclude that V ˜ π
(ˆ s ) ≥ V π relabel (ˆ s ) .
Remark.
The above proposition assumes the advantage value is for π relabel .
However, we empirically ﬁnd that using advantage value of the learned policy π (the value function is learned to minimize L TD in Section 4) works better in ofﬂine goal-conditioned RL setting.
As analyzed in prior works (Nair et al., 2020; Yang et al., 2021), using the value function of π is equivalent to solving the following problem:
[Equation]
In addition, for ofﬂine goal-conditioned RL, learning policies with inter-trajectory information is important for reaching diverse goals.
Therefore, using Q ( s t +1 , π ( s t +1 , g (cid:48) ) , g (cid:48) ) to compute TD target helps value estimation across trajectories compared with Q ( s t +1 , π relabel ( s t +1 , g (cid:48) ) , g (cid:48) ) .
Moreover, π is trained via weighted supervised learning to avoid the OOD action problem.
B. P ROOF OF P ROPOSITION
Proposition.
Assume the state space can be perfectly mapped to the goal space, ∀ s t ∈ S, φ ( s t ) ∈
G , and the dataset has sufﬁcient coverage.
We deﬁne a relabeling strategy for ( s t , g ) as g (cid:48) =
(cid:26) φ ( s i ) , i ∼ U [ t, T ] , if φ ( s i ) (cid:54) =
g, ∀ i ≥ t, g, otherwise.
We also assume that when g (cid:48) (cid:54) = g , the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g (cid:48) in the original dataset D .
Then, π relabel is uniformly as good as or better than π b , i.e., ∀ s t , g, ∈ D, V π relabel ( s t , g ) ≥
V π b ( s t , g ) , where π b is the behavior policy forming the dataset D .
Proof.
Let D = { τ j = ( s i , a i , g j )
Ti =1 } Nj =1 .
We assume that the data of D and D relabel are sufﬁcient and cover the full state and goal space, i.e., (cid:83) ( s i , g j ) =
S × G .
Note that every trajectory has only a single goal, and it does not have to reach the goal within the trajectory.
Therefore, we can partition
D into two subsets D p (positive set) and D n (negative set):
[Equation]
where D p contains trajectories that can reach the goal and D n contains trajectories that cannot reach the goal.
Let D ( s t , g ) =
{ τ = ( s i , a i , g )
Ti = t | τ begin with s t } , and similarly we deﬁne the partition
[Equation]
begin with
[Equation]
t
[Equation]
D
[Equation]
= t | τ begin with s t , ∀ s i ∈ τ, φ ( s i ) (cid:54) =
g } .
Hence, the value V π b can be written as
[Equation]
Given τ =
( s i , a i , g )
Ti = t ∈ D , we relabel the future trajectory as τ relabel =
( s i , a i , g (cid:48) )
Ti = t for value estimation following the strategy
[Equation]
and when g (cid:48) (cid:54) =
g , the strategy only accepts τ relabel with higher return than any trajectory in
D ( s t , g (cid:48) ) .
Let D relabel ( s t , g ) =
{ τ relabel =
( s i , a i , g )
Ti = t | τ relabel begin with s t } be the relabeled trajectories with g as goal.
Hence, the value V π relabel can be written as
[Equation]
For any τ relabel ∈ D relabel ( s t , g ) , we enumerate the situation of its counterpart τ without relabeling.
τ ∈ D p ( s t , g ) .
Following the relabeling strategy
we have g (cid:48)
=
g , such that τ relabel = τ . 2.
τ / ∈ D p ( s t , g ) , then τ has another goal ˜ g (cid:54) =
g , but it reaches g at some intermediate state.
It holds that R τ relabel t >
=
R τt .
Following the relabeling strategy, we further have
R
τ relabel
t ≥ R
[Equation]
t
∀ τ
[Equation]
∈ D
s t , g ) , which guarantees improvements on expected value of all trajectories starting from s t with goal g .
Therefore, we have
[Equation]
Remark.
The relabeling strategy of Proposition 2 requires a complete scan of future states of trajectories, which is costly.
Empirically we ﬁnd that relabeling with random future states as HER has comparable performance, and is very simple and efﬁcient to implement.
We provide the comparison between the relabeling strategy of Proposition 2 (called WGCSL-slow-relabel) and the relabeling strategy that we used in practice in Table 2.
[Table]
C A LGORITHM
We summarize our proposed approach WGCSL into Algorithm 1.
D O FFLINE S ETTINGS
Ofﬂine Datasets In the ofﬂine setting, we only use collected datasets to train agents, without additional interaction with the environment.
The difference between the ofﬂine goal-conditioned
RL dataset and the general ofﬂine RL dataset is that it also needs to save the original desired goals g and the achieved goals φ ( s t ) .
We follow previous works (Fujimoto et al., 2019; Fu et al., 2020) to introduce two different types of datasets:
Given ofﬂine dataset D , the best advantage percentile threshold N ;
Initialize policy π θ and value function Q ;
Initialize a First-In-First-Out queue B = {} to save recent advantage values; for training step = 1 , 2 , . .
, M do
Sample a mini-batch from the ofﬂine dataset: { ( s t , a t , g, r t ) } ∼ D ;
Relabel the mini-batch using future goals: { ( s t , a t , φ ( s i ) , r ( s t , a t , φ ( s i ))) , i ≥ t } ∼ D relabel ;
Update value function Q to minimize TD error L TD on the relabeled mini-batch;
Estimate advantage value A ( s t , a t , φ ( s i )) using value function Q ;
Update queue B with the estimated advantage values;
Get the N percentile advantage value ˆ A from B to compute the BAW;
Update policy π θ to maximize the objective in Eq.
3 with the relabeled mini-batch:
J WGCSL =
E ( s t ,a t ,s i ) ∼ D relabel [ γ i − t log π θ ( a t | s t , φ ( s i ))
exp clip ( A ( s t , a t , φ ( s i ))) · (cid:15) ( A ( s t , a t , φ ( s i )))] end for
[Table]
• Random dataset: using a uniformly random policy to collect trajectories.
• Expert dataset: using the ﬁnal online policy trained by HER with additional Gaussian noise (zero mean and 0 .
2 standard deviation) and random actions (with probability 0.3) following previous work (Plappert et al., 2018) for data collection.
For the dataset of six easy tasks (PointReach, PointRooms, Reacher, SawyerReach, SawyerDoor,
FetchReach), we save a dict of 1 × 10 5 transitions (2000 trajectories) containing observations, actions, desired goals and achieved goals.
Regarding the other four hard tasks (FetchPush, FetchSlide,
FetchPick, HandReach), we store 2 × 10 6 transitions (40000 trajectories) in the ofﬂine dataset.
The rewards can be computed by leveraging the corresponding interface (namely env.compute reward ) of the environment.
The average returns of the two types of datasets are shown in Table 3.
Generally, average returns of the expert datasets are close to that of the optimal policy, while average returns of the random datasets are close to 0.
Baseline Introduction In the ofﬂine goal-conditioned setting, we denote the ofﬂine dataset as D and consider the following baselines:
• GCSL : GCSL (Ghosh et al., 2021) without online interaction, only using ofﬂine samples after relabeling (denoted as D relabel ) to maximize
[Equation]
• Goal MARWIL : goal-conditioned MARWIL (Wang et al., 2018), using ofﬂine samples to maximize
[Equation]
To fairly compare with our method, Goal MARWIL is implemented with actor-critic style as AWAC (Nair et al., 2020) and WGCSL, therefore it can also be treated as ’Goal AWAC’.
Besides, we also clip the exponential advantage value as WGCSL for numerical stability.
• Goal AWR : goal-conditioned AWR (Peng et al., 2019), using ofﬂine samples to maximize
[Equation]
where R t = (cid:80) T − t i =
γ i r t
+ i .
Different from the original AWR that uses TD( λ ) or Monte Carlo return R t to train the value function, we use TD loss to learn value function as WGCSL.
Empirically, we ﬁnd TD target works better than Monte Carlo in our setting.
• Goal BC : goal-conditioned behavior cloning (Bain & Sammut, 1995), using ofﬂine samples to maximize
[Equation]
• Goal BCQ : goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019), which we use the ofﬁcial codebase and concatenate observations, desired goals and achieved goals as states.
The objective function of Goal BCQ is as follows:
[Equation]
where ξ ( s t , a, g, Φ) is the goal-conditioned perturbation model, which outputs an adjustment to an action a within the range [ − Φ , Φ
G w is the VAE ﬁtted on the behavior policy of the ofﬂine dataset.
The policy is optimized through optimizing ξ .
The Q function is learned using the Clipped Double Q-learning in Fujimoto et al ., 2019), and the only difference is that the Q function also includes goal g as input.
• Goal CQL : goal-conditioned Conservative Q-Learning
(Kumar et al., 2020), which we also use the ofﬁcial codebase and concatenate observations, desired goals and achieved goals as states.
The objective function of Goal CQL is as follows:
[Equation]
which jointly optimizes estimated return and policy entropy.
The Q
function
of CQL is learned by minimizing
the following loss:
[Equation]
where π b is the behavior policy for the ofﬂine dataset and B π is the Bellman operator.
• HER : Hindsight Experience Replay (Andrychowicz et al., 2017), using ofﬂine samples after relabeling to maximize
[Equation]
where g (cid:48)
= φ ( s i ) , i ≥ t and the Q function is learned by minimizing TD error:
[Equation]
, g (cid:48) ))
Implementation of Baselines For fair comparison, all the implementation of WGCSL, GCSL,
Goal MARWIL, Goal AWR, Goal BC, and HER use the Diagnal Gaussian policy with a mean vector π θ ( s, g ) and a non-zero constant variance σ 2 , where σ is set as 0.2.
WGCSL, Goal MARWIL,
Goal AWR, and HER share the same network structure, i.e., the value function and the policy network (along with their target networks) are both 3-layer MLPs with 256 units each layer and relu activation.
Besides, GCSL and Goal BC both keep a single policy network without target network and value function.
We also normalize the observations and goals with estimated mean and standard deviation for WGCSL and other baselines, which is helpful for ofﬂine multi-goal manipulation tasks.
The parameter β in Goal MARWIL and Goal AWR are set as 1 by default.
As regards to Goal
BCQ and Goal CQL, we use the same batch size as other algorithms and other parameters are set as default (we ﬁnd that default parameters perform equally well, if not better than others, in the grid search of critical parameters).
All other hyper-parameters of different algorithms are kept the same for each task, such as batch size (128 for the ﬁrst 6 tasks and 512 for 4 harder taks), discount factor γ = 0 . 98 , and Adam optimizer with learning rate 5 ×
10 − 4 .
Implementation of WGCSL
The basic implementation is introduced as above.
To compute the best-advantage weight for WGCSL, we keep a First-In-First-Out queue B of size 5 × 10 4 to store recent calculated advantage values, and the percentile threshold N gradually increases from 0 to 80 .
For each training step, N increases by 0.01 (for 4 harder tasks from FetchPush to HandReach) or 0.15 (for other tasks).
The maximum clipping upper bound M for WGCSL is set as 10.
Though
DRW brings improvements for most of the tasks on expert and random datasets, we ﬁnd DRW slightly affects the performance of harder tasks on random datasets, which may be because in the random datasets we need more timesteps to obtain enough coverage of achieved goals, especially for harder tasks.
Tuning γ for DRW trade-offs between the value and the effective coverage of relabeling goals, and excluding DRW is equivalent to setting γ = 1 for DRW.
Therefore, we only use GEAW and BAW for these tasks on random datasets.
Evaluation Settings For evaluation, we evaluate all the algorithms for 100 episodes without randomness.
We repeat for 5 different random seeds and report the average evaluation performance with standard deviation.
[Figure]
E A DDITIONAL E XPERIMENTAL R ESULTS
In this section, we provide more empirical results in both ofﬂine and online goal-conditioned RL setting, including average return, ﬁnal distance of the ofﬂine setting, results on small ( 10% ) dataset, and the results of online experiments
[Figure]
E. F ULL O FFLINE R ESULTS
We provide the average return curves of all tasks in Figure 7.
In these results, WGCSL performs consistently well across ten environments, especially when handling hard tasks (FetchPush, FetchSlide,
FetchPick, HandReach) and learning from the random ofﬂine dataset.
Other conclusions have been discussed in Section 5.2.
Comparing AWR with MARWIL, the major difference is using Monte
Carlo return to replace the TD estimation for learning the policy.
Hence, we can ﬁnd that TD bootstrap learns more efﬁciently in weighted supervised learning
[Figure]
which is similar to the conclusion of prior work (Nair et al., 2020).
In addition
[Figure]
we present all ﬁnal performance of different algorithms in Table E.
[Figure]
u
[Figure]
v
[Figure]
r
[Figure]
e t u r n
[Figure]
E. C OMPARISON R ESULTS WITH A CTIONABLE M ODELS , HER AND DDPG
In this subsection
[Figure]
we make comparison with Actionable Models (AM) (Chebotar et al., 2021), which applies the conservative estimation idea similar to (Kumar et al., 2020) into ofﬂine goalconditioned RL.
There are some different settings between AM and ours.
For example, we consider cumulative return of the maximum horizon,
i
[Figure]
e., the agent doesn’t terminate the interaction after reaching the goal.
To fairly compare with AM, we implement AM based on HER.
Speciﬁcally, we revise the value update as:
[Equation]
In the formula, AM tries to minimize TD error and minimize unseen Q values at the same time.
The action a is sampled according to exponential Q values.
In our implementation, we sample 20 actions from the action space and compute their exponential Q value in order to sample a .
The results are shown in Figure 8.
We can observe that DDPG and HER are unstable during training, with large variance and performance drops at the end.
In both expert and random dataset, HER outperforms DDPG, which demonstrates the beneﬁts of goal relabeling for ofﬂine goal-conditioned
RL.
Besides, Actionable Model (AM) achieves higher ﬁnal performance than HER and DDPG especially in expert dataset, which indicates the conservative estimation technique is useful for ofﬂine goal-conditioned RL.
Moreover, WGCSL is more efﬁcient than AM, especially in harder tasks with large action space.
Intuitively, WGCSL promotes the probability of visited actions, while AM restrains the probability of unvisited actions.
For tasks with large action space, conservative strategy can be ineffective compared to WGCSL.
[Figure]
E. N ORMALIZED S CORE A CROSS 10 T ASKS
For clarity, we report the normalized score across 10 tasks to compare different algorithms in Figure 9.
We can observe the signiﬁcant improvement of WGCSL over other baselines.
We take an insight into the impact of the goal relabeling technique.
For expert datasets, goal relabeling is not a dominant factor, e.g., the learning curve of Goal BC and GCSL are very close.
In random datasets, goal relabeling is essential to achieve a high performance.
In addition to goal relabeling, WGCSL contains other techniques to jointly promote the performance in ofﬂine goal-conditioned RL tasks.
Other conclusions are the same as prior subsections (Appendix E.1 and Appendix E.2).
E. C OMPARISON OF T RAINING T IME
We further analyze the computation cost of the four most effective algorithms in the random ofﬂine setting, i.e., WGCSL, GCSL, HER and Actionable Models (AM).
The results are shown in Figure 10, from which we can conclude that GCSL needs the minimal training time, while WGCSL and
HER require similar cost.
It needs to be emphasized that WGCSL has only a small amount of additional training time over GCSL, but it achieves signiﬁcantly higher performance.
Therefore, we believe that the cost of learning the value function on top of GCSL is worthwhile.
Moreover,
AM requires the largest computation among these algorithms, which is mainly because it needs to
[Figure] [Figure]
sample several actions and compute their exponential Q value for constraining the large Q value.
These procedures make AM the most time-consuming algorithm.
In contrast, WGCSL can achieve higher performance with less training time.
The training time would vary across different platforms, and we use one single GPU (Tesla P100 PCIe 16GB) and 5 cpu cores (Intel Xeon E5-2680 v4 @ 2.40GHz) to run 5 random seeds parallelly
[Figure]
E. A VERAGE P ROBABILITY OF I MPROVEMENT
In this subsection
[Figure]
we adopt the average probability of improvement (Agarwal et al., 2021), a robust metric to measure how likely it is for one algorithm to outperform another on a randomly selected task.
The results are reported in Figure 11.
As shown in the results, WGCSL robustly outperforms other baselines in both expert and random datasets.
For instance, WGCSL is 100% better thanbaselines on the expert dataset, and 6 baselines on the random dataset.
Note that the most two effective and robust algorithms on both random and expert datasets are WGCSL and Actionable
Models (AM), which are speciﬁcally designed for ofﬂine goal-conditioned RL.
Comparing the two algorithms, WGCSL outperforms AM with a probability of 75% on the expert dataset and 62% on the random dataset.
E. F INAL D ISTANCES IN THE O FFLINE S ETTING
GCSL (Ghosh et al., 2021) uses the average ﬁnal distance to the desired goal as a measure for goal-reaching problems.
We also provide average ﬁnal distances in the ofﬂine setting to evaluate the learned goal-conditioned policy in Figure 12.
From the results we can conclude that WGCSL substantially outperforms other baselines in both random and expert dataset.
Although GCSL learns a suboptimal policy, it also achieves relatively consistent ﬁnal distance across different environments
[Figure]
and ofﬂine datasets.
Besides, Goal MARWIL and Goal BC are two effective baselines when learning from the expert dataset, but they do not perform well on the random dataset.
In addition, HER and
DDPG learn unstable results mainly because of bootstrapping from out-of-distribution actions.
E. R ESULTS ON 10% O FFLINE D ATASET
To evaluate the impact of dataset size on the performance
[Figure]
we select 10% of samples (i
[Figure]
e., 2 × 10transitions) from the ofﬂine dataset of 4 hard tasks (FetchPush, FetchSlide, FetchPick, HandReach).
The results are reported in Figure 13 and Figure 14.
From the results, we can conclude that with only 10% ofﬂine data WGCSL consistently achieves the best performance out of these algorithms.
In Figure 14, we can see that WGCSL is very robust to the dataset size on the expert dataset, while the performance is affected more by the small data size on the random dataset.
The probable reason is that in the ofﬂine setting, the number of samples inﬂuences the data coverage greatly and learning generalizable value function from a small set of data is very difﬁcult.
E. O NLINE E XPERIMENTS
We introduce implementation details and hyper-parameters of the online setting in this subsection.
As GCSL is originally proposed for online goal-conditioned RL, WGCSL can also be applied to online goal-conditioned setting.
In the online training setting, the agent iterates between collecting trajectories from the environment and training with samples from the replay buffer.
We conduct
[Figure] [Figure]
experiments on the ﬁrst six taks, i.e., PointReach (here we call Point2DLargeEnv), PointRooms (here we call Point2D-FourRooms), SawyerReach, FetchReach, Reacher and SawyerDoor.
The buffer size for ﬁrst four tasks is set as 1 × 10 4 , and 5 × 10 4 for the other two tasks.
Before training starts, we ﬁrst randomly sample 20 trajectories to initialize the buffer.
The policies of WGCSL and
GCSL are both Diagonal Gaussian policy with a mean vector π θ ( s, g ) parameterized as θ and a nonzero constant variance σ 2 .
The standard deviation σ is set as 0.2.
All the policy networks and value networks are the same as that used in the ofﬂine setting.
When interacting with the environments, we use additional random actions for exploration with a probability of 0 . 3 .
After data collection, we sample a mini-batch from the replay buffer and perform goal relabeling for WGCSL and GCSL.
Then we train the agent for 1 mini-batch (Point2DLarge, Point2D-FourRooms) or 4 mini-batches for other tasks.
The batch size, optimizer, learning rate keeps the same as the ofﬂine setting.
The maximum clipping upper bound M for WGCSL is set as 10.
We also utilize target nets for WGCSL to stabilize training, and the polyak average coefﬁcient for target nets are 0.9.
With respect to the evaluation phase, we use the deterministic policy π θ ( s t , g ) without randomness.
After each epoch, we evaluate the policy for 100 episodes and compute the average success rate.
For all the experiments we repeat for 10 different random seeds and report the mean performance with standard deviation.
In this subsection, we report the results of WGCSL with the discounted relabeling weight (DRW) and the goal-conditioned exponential advantage weight (GEAW), though the performance can be improved with the best-advantage weight (BAW).
The empirical results regarding the average success rate of the online setting are shown in Figure 15.
From Figure 15 we can conclude that WGCSL achieves better success rate and converges faster than other baselines in most of the environments except SawyerReach.
Therefore, WGCSL is veriﬁed to be highly efﬁcient in both online and ofﬂine goal-conditioend RL, and we hope it could be a stepping stone for future real-world RL applications.
[Figure]
[Figure]
E. T HE I MPACT OF C LIPPING FOR E XPONENTIAL A DVANTAGE W EIGHTS
We study the impact of clipping in WGCSL’s exponential advantage weights by comparing with the following variants:
• WGCSL w/o clip: no clipping weights performed.
• WGCSL clip 5: clipping the exponential advantage weights to (0 , 5] .
• WGCSL clip 10: clipping the exponential advantage weights to (0 , 10] , which is used in our paper by default.
From the results in Figure 16 we can observe that without clipping the advantage estimation can harm the performance in Reacher and SawyerDoor.
Therefore, we perform weight clipping by default.
F D ETAILS OF E NVIRONMENTS
All of the ten tasks have continuous state space, action space and goal space.
The maximum episode horizon is set as 50.
In this section, we introduce these tasks in detail.
PointReach PointReach is taken from the open-source environment multi-world
and it requires the blue point to reach the green circle.
The state space [ − 5 , 5] × [ − 5 , 5] has two dimensions representing Cartesian coordinates of the blue point, and the action space [ − 1 , 1] × [ − 1 , 1] also has two dimensions meaning the horizontal and vertical displacement.
The goal space is the same as state space, which means φ ( s )
= s .
The bule point and the green circle are randomly initialized in the state space.
The allowable error (cid:15) of reaching goals is the radius of the target circle and is set as 1 .
The reward function is deﬁned as:
[Equation]
PointRooms
The PointRooms environment is built on multi-world .
The state space, the action space, the goal space, and the reward function are the same as PointReach.
The difference is that there are four rooms separated by walls.
Reacher The Reacher environment is revised from OpenAI Gym (Brockman et al., 2016).
States are 11-dimensional, which indicate the angles, the positions, and the velocity of the joints.
Actions are 2 -dimensional
and they control the movement of two joints.
Goals are 2 -dimensional vectors representing the expected XY position.
And the state-to-goal mapping is φ ( s ) =
s [ − 3 : − 1] , where the last three dimensions are the XYZ position of the end-effect.
The reward function is deﬁned as:
[Equation]
where the allowable error (cid:15) is set as 0 .
SawyerReach The SawyerReach environment is taken from multi-world .
The Sawyer robot aims to reach a target position with its end-effector.
The observation space is 3-dimensional, representing the 3D Cartesian position of the end-effector.
Correspondingly, the goal space is 3-dimensional and describes the expected position, and the state-to-goal mapping is φ ( s )
= s .
Besides, the action space has 3 dimensions describing the next position of the end-effector.
The reward function is deﬁned as:
[Equation]
where the allowable error (cid:15)
= 0 .
SawyerDoor
The SawyerDoor environment is revised from multi-world .
The Sawyer robot is required to open the door to a desired angle.
The state space (4-dimensional) consists of the Cartesian coordinates of the Sawyer end-effector and the door’s angle.
As in the SawyerReach task, the action space is 3 -dimensional and controls the position of the end-effector.
The desired goals are uniform from 0 to 0.83 radians.
And the reward function is deﬁned as:
[Equation]
where the allowable error (cid:15) is set as 0 .
FetchReach The FetchReach environment is taken from OpenAI Gym (Brockman et al., 2016;
Plappert et al., 2018).
In this environment, a 7-DoF robotic arm is expected to touch a desired location with its two-ﬁnger gripper.
The state space is 10-dimensional, including the gripper’s position and linear velocities.
The action space is 4-dimensional, which represents the gripper’s movements and its status about the opening and closing.
Moreover, the goals are 3-dimensional vectors representing the target places of the gripper.
The state-to-goal mapping is φ ( s ) =
s [0 : 3] , because the ﬁrst 3 dimensions of the state describe the position of the gripper.
The allowable error in FetchReach is (cid:15) = 0 .
The reward function is deﬁned as:
[Equation]
FetchPush Similar to FetchReach, the FetchPush environment is taken from OpenAI Gym where a 7-DoF robotic arm is controlled to move a box to the desired location.
The state space is 25dimensional, including the gripper’s position, linear velocities, and the box’s position, rotation, linear and angular velocities.
The action space is also 4-dimensional as in FetchReach.
Different from
FetchReach, the 3-dimensional goals and achieved goals represent the target and current position of the box.
The state-to-goal mapping is φ ( s ) =
s [3 : 6] , because the 3 dimensions of state describe the position of the box.
The allowable error in FetchPush is (cid:15) = 0 . 05 .
The reward function is deﬁned as:
[Equation]
FetchSlide The FetchSlide environment is like FetchPush where a 7-DoF robotic arm is controlled to slide a box to the desired location.
The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush.
The difference is that the target position is outside of the robot’s reach thus it has to hit the box and makes the box slide and then stop at the desired position.
FetchPick Like FetchPush and FetchSlide environments, the FetchPick environment controls a 7-DoF robotic arm to pick up a box to the desired location.
The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush.
The difference is that the target position may be on the table or in the air, thus the agent needs to pick the box with its gripper.
HandReach In HandReach environment, the agent is required to control a 24-DoF anthropomorphic hand and use its ﬁngers to reach the target place.
Observations is 63-dimensional containing two 24-dimensional vectors about positions and velocities of the joints, and additional 15 dimensions indicating current state of ﬁngertips.
Actions are 20-dimensional vectors that control the non-coupled joints of the hand.
Goals have 15 dimensions for HandReach representing the target Cartesian positions of each ﬁngertip.
The reward function is the same as Fetch tasks except the state-to-goal-mapping ( φ = s [ − 15 :] ) and the allowable threshold ( (cid:15) = 0 . 01 ).
[Table]