R ETHINKING G OAL -C ONDITIONED S UPERVISED L EARNING AND I TS C ONNECTION TO O FFLINE RL
Anonymous authors
Paper under double-blind review
A BSTRACT
Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms.
A recent work, called Goal-Conditioned Supervised
Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences.
In this paper, we revisit the theoretical property of GCSL optimizing a lower bound of the goal reaching objective and extend GCSL as a novel ofﬂine goal-conditioned RL algorithm.
The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned monotonic advantage re-weighting, and (3) bestadvantage weight.
Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme.
The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and ofﬂine goal-conditioned RL.
To evaluate algorithms in the ofﬂine goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains.
Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art ofﬂine methods in the fully ofﬂine goal-conditioned setting.
I NTRODUCTION
Reinforcement learning (RL) enables automatic skill learning and has achieved great success in various tasks (Mnih et al., 2015;
Lillicrap et al., 2016; Haarnoja et al., 2018; Vinyals et al., 2019).
Recently, goal-conditioned RL is gaining attention from the community, as it encourages agents to reach multiple goals and learn general policies (Schaul et al., 2015).
Previous advanced works for goal-conditioned RL consist of a variety of approaches based on hindsight experience replay (Andrychowicz et al., 2017; Fang et al., 2019), exploration (Florensa et al., 2018;
Ren et al., 2019; Pitis et al., 2020), and imitation learning (Sun et al., 2019;
Ding et al., 2019; Ghosh et al., 2021).
However, these goal-conditioned methods need intense online interaction with the environment, which could be costly and dangerous for real-world applications.
A direct solution to learn goal-conditioned policy from ofﬂine data is through imitation learning.
For instance, goal-conditioned supervised learning (GCSL) (Ghosh et al., 2021) iteratively relabels collected trajectories and imitates them directly.
It is substantially simple and stable, as it does not require expert demonstrations or value estimation.
Theoretically, GCSL guarantees to optimize over a lower bound of the objective for goal reaching problem.
Although hindsight relabeling (Andrychowicz et al., 2017) with future reached states can be optimal under certain conditions (Eysenbach et al., 2020), it would generate non-optimal experiences in more general ofﬂine goal-conditioned RL setting with cumulative discounted return (Chebotar et al., 2021).
As a result, GCSL suffers from the same issue as other behavior cloning methods by assigning uniform weights for all experiences and results in suboptimal policies in ofﬂine settings.
To this end, leveraging the simplicity and stability of GCSL, we generalize it to the ofﬂine goal-conditioned RL setting and propose an effective and theoretically grounded method, named
Weighted GCSL (WGCSL).
We ﬁrst revisit the theoretical foundation of GCSL by additionally considering discounted rewards, and this allows us to obtain a weighted supervised learning objective.
To learn a better policy from the ofﬂine dataset and promote learning efﬁciency, we introduce a more general weighting scheme by considering the importance of different relabeling goals and the expected return estimated with a value function.
Theoretically, the discounted weight for relabeling goals contributes to optimizing a tighter lower bound than GCSL.
Based on the discounted weight, we show that additionally re-weighting with a monotonic function over advantage value guarantees monotonic policy improvement for goal-conditioned RL.
Moreover, the introduced weights build a natural connection between WGCSL and ofﬂine RL, making WGCSL available for both online and ofﬂine setting.
Another major challenge in goal-conditioned RL is the multi-modality problem (Lynch et al., 2020), i.e., there are generally many valid trajectories from a state to a goal, which can present multiple counteracting action labels and even impede learning.
To tackle the challenge, we further introduce the best-advantage weight under the general weighting scheme, which contributes to the asymptotic performance when the policy is multi-modal.
Although WGCSL has the cost of learning the value function, we empirically show that it is worth for its remarkable improvement.
For the evaluation of ofﬂine goal-conditioned RL algorithms, we provide a public benchmark and ofﬂine datasets including a set of challenging multi-goal manipulation tasks with a robotics arm or an anthropomorphic hand.
Experiments 1 conducted in the introduced benchmark show that WGCSL signiﬁcantly outperforms other state-of-the-art baselines in the fully ofﬂine goal-conditioned settings, especially in the difﬁcult anthropomorphic hand task and when learning from randomly collected datasets with sparse rewards.
P RELIMINARIES
Markov Decision Process and ofﬂine RL The RL problem can be described as a Markov Decision Process (MDP), denoted by a tuple ( S , A , P , r, γ ) , where S and A are the state and action spaces; P describes the transition probability as S ×
A × S → [0 , 1] ; r : S ×
A → R is the reward function and γ ∈ (0 , 1] is the discount factor; π : S → A denotes the policy, and an optimal policy π ∗ satisﬁes π ∗ = arg max π E s 0 ∼ ρ ( s 0 ) ,a t ∼ π ( a t | s t ) ,s t +1 ∼P
( s t +1 | a t ,s t ) [ (cid:80) ∞ t =0 γ t r ( s t , a t )] , where ρ ( s 0 ) is the distribution of initial states.
For ofﬂine RL problems, the agent can only access a static dataset
D , and is not allowed to interact with the environment during the training process.
The ofﬂine data can be collected by some unknown policies.
Goal-conditioned RL Goal-conditioned RL further considers a goal space G .
The policy π :
S × G → A and the reward function r : S × G ×
A → R are both conditioned on goal g ∈ G .
The agent learns to maximize the discounted cumulative return
[Equation]
where p ( g ) is the distribution over goals g ∈ G .
We study goal-conditioned RL with sparse rewards, and the reward function is typically deﬁned as:
[Equation]
< some threshold 0 , otherwise , where φ : S → G is a mapping function from states to goals (Andrychowicz et al., 2017).
Such reward function can also be written as an indicator function as r ( s t , a t , g ) =
1[ φ ( s t )
=
g ] , which provides a positive signal only when s t achieves g .
The value function V : S × G → R is deﬁned as V π ( s, g ) =
E a t ∼ π,s t +1 ∼P ( ·| s t ,a t ) (cid:2) (cid:80)
∞ t =
γ t r ( s t , a t , g ) | s
= s (cid:3) .
Goal-conditioned Supervised Learning Different with goal-conditioned RL which maximizes discounted cumulative return, GCSL considers the goal-reaching problem, i.e., maximizing the laststep reward E g ∼ p
( g ) ,τ ∼ π (cid:2) r ( s T , a T , g ) (cid:3) for trajectories τ of horizon T .
GCSL iterates between two process, relabeling and imitating.
Trajectories in the replay buffer are relabeled with hindsight methods (Kaelbling, 1993; Andrychowicz et al., 2017) to form a relabeled dataset D τ = { ( s t , a t , g (cid:48) ) } , where g (cid:48) =
φ ( s i ) for i ≥ t denotes the relabeled goal.
The policy is optimized through supervised learning to mimic those relabeled transitions:
[Equation]
[Figure]
W EIGHTED G OAL -C ONDITIONED S UPERVISED L EARNING
In this section, we will revisit GCSL and introduce the general weighting scheme which generalizes
GCSL to ofﬂine goal-conditioned RL.
3. R EVISITING G OAL - CONDITIONED S UPERVISED L EARNING
As an imitation learning method, GCSL sticks the agent’s policy to the relabeled data distribution, therefore it naturally alleviates the problem of out-of-distribution actions.
Besides, it has the potential to reach any goal in the ofﬂine dataset with hindsight relabeling and the generalization ability of neural networks.
Despite its advantages, GCSL has a major disadvantage for ofﬂine goalconditioned RL, i.e., it only considers the last step reward r ( s T , a T , g ) and generally results in suboptimal policies.
A Motivating Example
We provide training results of GCSL in the PointReach task in Figure 2.
The objective of the task is to move a point from the starting position to the desired goal as quickly as possible.
The ofﬂine dataset is collected using a random policy.
As shown in Figure 2, GCSL learns a suboptimal policy which detours to reach goals.
This is because GCSL only considers the last step reward r ( s T , a T , g ) .
As long as the trajectories reaching the goal at the end of the episode, they are all considered equally to GCSL.
To improve the learned policy, a straightforward way is to evaluate the importance of samples or trajectories for policy learning using importance weights.
As a comparison, Weighted GCSL (WGCSL), which uses a novel weighting scheme, learns the optimal policy in the PointReach task.
In the following analysis, we will derive the formulation of WGCSL.
Connection between Goal-Conditioned RL and SL
GCSL has been proved to optimize a lower bound on the goal-reaching objective (Ghosh et al., 2021), which is different from the goal-conditioned RL objective that optimizes the cumulative return.
In this section, we will introduce a surrogate function for the goal-conditioned RL and then derive the connections between goal-conditioned RL and weighted goal-conditioned supervised learning (WGCSL).
For overall consistency, we provide the formulation of WGCSL as below:
[Equation]
where p ( g ) is the distribution of desired goals; π b refers to the data collecting policy.
The weight w i,t is subscripted by both the time step t and the index i of the relabeled goal.
When w i,t = 1 ,
J WGCSL reduces to GCSL (Ghosh et al., 2021), and we denote the corresponding objective as
J GCSL for comparison convenience.
Note that in Eq. 2, φ ( s i ) , i ≥ t implies that any future visited states after s t can be relabeled as goals.
Theorem 1.
Assume a discrete MDP with discrete state/goal spaces, a random discrete policy π which selects actions with non-zero probability and a sparse goal-conditioned reward function r ( s t , a t , g ) =
1[ φ ( s t )
= g ] , where φ is the state-to-goal mapping and 1[ φ ( s t ) =
g ] is an indicator function.
Given trajectory τ = ( s 0 , a 0 , s 1 , a 1 , · · · , s T ) and reward discount γ ∈ (0 , 1] , let the weight w i,t = γ
i − t , t ∈ [0 , T ] , i ∈
[ t, T ] , then the following bounds hold:
[Equation]
where J surr ( π ) =
1 T E g ∼ p
( g ) ,τ ∼ π b ( ·| g ) (cid:104)(cid:80)
Tt =0 log π ( a t | s t , g ) (cid:80)
Ti = t γ i · 1[ φ ( s i ) =
g ] (cid:105) is a surrogate function of J ( π ) .
We defer the proof and the justiﬁcation of J surr to Appendix B.1.
Theorem 1 reveals the connection between the goal-conditioned RL objective and the WGCSL/GCSL objective.
Meanwhile, it suggests that GCSL with the discount weight γ
i − t is a tighter bound for the surrogate function compared to the unweighted version.
We name this weight as discounted relabeling weight (DRW) as it intuitively assigns larger weights on shorter trajectories reaching the same relabeled goal.
More importantly, the result in Theorem 1 holds for any behavior policy even if it is from ofﬂine dataset, building a connection between goal-conditioned supervised learning and ofﬂine RL.
3. A M ORE G ENERAL W EIGHTING S CHEME
DRW can be viewed as a weight evaluating the importance of relabeled goals.
In this subsection, we consider a more general weighting scheme using a weight function measuring the importance of state-action-goal combination, revealed by the following corollary.
Corollary 1.
Suppose function h ( s, a, g ) ≥ 1 over the state-action-goal combination.
Given trajectory τ = ( s 0 , a 0 , s 1 , a 1 , · · · , s T ) , let w i,t = γ
i − t h
( s t , a t , φ ( s i )) , t ∈ [0 , T ] , i ∈ [ t, T ] , then the following bound holds:
[Equation]
The proof are provided in Appendix B.2.
Naturally, the Q-value in RL with some constant shift is a candidate of the function family h ( s, a, g ) .
Inspired by prior ofﬂine RL approaches (Wang et al., 2018; Peng et al., 2019; Nair et al., 2020), we choose h ( s, a, g ) =
exp( A ( s, a, g )+ C ) , an exponential function over the goal-conditioned advantage, which we refer to as goal-conditioned exponential advantage weight (GEAW).
The constant C is used to ensure that h ( s, a, g ) ≥ 1 .
The intuition is that GEAW assigns larger weight for samples with higher values.
In addition, the exponential advantage weighted formulation has been demonstrated to be a closed-form solution of an ofﬂine RL problem, where the learned policy is constrained to stay close to the behavior policy (Wang et al., 2018).
Compared to DRW, GEAW evaluates the importance of state-action-goal samples, taking advantage of the universal value function (Schaul et al., 2015).
To tackle the multi-modality challenge in goal-conditioned RL (Lynch et al., 2020), we further introduce the best-advantage weight (BAW) based on the learned value function.
BAW has the following form:
[Equation]
where ˆ A is a threshold, and (cid:15) min is a small positive value.
Implementation details of BAW can be found in Section 4.
In our implementation, ˆ A gradually increases.
Therefore, BAW gradually leads the policy toward the modal with the highest return rather than sticks to a position between multiple modals, especially when ﬁtting using a Gaussian policy.
Combining all three introduced weights, we have the weighting form: w i,t = γ
i − t exp( A ( s t , a t , φ ( s i ))
+ C ) · (cid:15)
( A ( s t , a t , φ ( s i ))) .
The constant
C only introduces a coefﬁcient independent of i, t for all transitions, therefore, we simply omit it for further analysis.
3. P OLICY I MPROVEMENT VIA WGCSL
We formally show that combining the three introduced weights, WGCSL can consistently improve the policy learned from the ofﬂine dataset.
First, we assume there exists a policy π relabel that can produce the relabeled experiences D τ .
GCSL is essentially equivalent to imitating π relabel :
[Equation]
Following (Wang et al., 2018), solving WGCSL is equivalent to imitating a new goal-conditioned policy ˜ π deﬁned as:
[Equation]
Then, we have the following proposition:
Proposition 1.
˜ π is uniformly as good as or better than π relabel .
That is, ∀ s t , φ ( s i ) ∈
D τ , i ≥ t
[Equation]
V ˜ π ( s t , φ ( s i )) ≥ V π relabel ( s t , φ ( s i )) , where D τ contains the experiences after goal relabeling.
The proof can be found in Appendix B.3.
Proposition 1 implies that WGCSL can generate a uniformly non-worse policy using the relabeled data D τ .
To be rigorous, we also need to discuss the relationship between the relabeled return and the original return.
In fact, there is a monotonic improvement for the relabeled return over the original one when relabeling with inverse RL (Eysenbach et al., 2020).
Below, we provide a simpler relabeling strategy which also offers monotonic value improvement.
Proposition 2.
Assume the state space can be perfectly mapped to the goal space, i.e., φ ( s t ) ∈
G, ∀ s t ∈ S , and deﬁne a relabeling strategy using mixed future states as
[Equation]
Then, the expected return after relabeling is uniformly better than the original return, i.e., ∀ s t , g, s i ∈
D τ , E τ ∼ π b ,s t ,s i ∼ τ
[ V π relabel ( s t , φ ( s i ))
] ≥ E τ ∼ π b ,s t ,g ∼ τ [ V π b ( s t , g )] , where π b is the behavior policy forming the dataset.
The proof is provided in Appendix B.4.
In Proposition 2, the relabeling strategy is slightly different from the strategy of relabeling with random future states (Andrychowicz et al., 2017).
When there is no successful future states, the relabeling strategy in Proposition 2 randomly samples from the future visited states for relabeling, otherwise it randomly samples from the successful future states.
Actually, in ofﬂine random dataset, there are rare successful states and the mixture relabeling strategy in
Proposition 2 acts similarly to relabeling with random future states.
Combining Propositions 1 and 2, we know that under certain assumptions, after performing relabeling and weighted supervised learning, the policy can be monotonically improved.
In fully ofﬂine settings, WGCSL is able to learn a better policy than the policy learned by GCSL and the behavior policy generating the ofﬂine dataset.
A LGORITHM
In this section, we summarize our proposed algorithm.
Denote the relabeled dataset as
[Equation]
≥ t, and we maximize the following WGCSL objective based on the relabeled data
[Equation]
where the weight w i,t is composed of three parts conforming to the general weighting scheme
[Equation]
(1) γ i − t is the discounted relabeling weight (DRW) as introduced in Section
(2) exp clip ( A ( s t , a t , φ ( s i )))
=
clip(exp( A ( s t , a t , φ ( s i ))) , 0 , M
) is the goal-conditioned exponential advantage weight (GEAW) with a clipped range (0 , M ] , where M is some positive number for numerical stability.
In our experiments, M is set as 10 for all the tasks.
TargetPoint (a)
TargetPointWall (b) (c) (d) (e) (f) (g) (h) (i) (j)
(3) (cid:15)
( A ( s t , a t , φ ( s i ))) is the best-advantage weight (BAW) with the form in Eq.
In our experiments, ˆ A is set as N percentile of recent advantage values and (cid:15) min is set as 0.05.
Motivated by the curriculum learning (Bengio et al., 2009), N gradually increases from 0 to 80 , i.e., we learn from all the experiences in the beginning stage and converge to learning from samples with the top 20% advantage values.
The advantage value in Eq.
4 can be estimated by A ( s t , a t , g (cid:48) )
=
r ( s t , a t , g (cid:48) )
+ γV ( s t +1 , g (cid:48) )
−
V ( s t , g (cid:48) ) , and we learn a Q-value function Q ( s t , a t , g (cid:48) ) by minimizing the TD error
[Equation]
where we relabel original goals g as g (cid:48)
=
φ ( s i ) with a probability of 80% and remain the original goal unchanged for the rest 20% time.
For simplicity, in our experiments, once it falls into the 80% probability of goal relabeling, we use purely relabeling with random future states, no matter whether there exist successful future states in the trajectory or not, to avoid parsing all the future states as discussed in Proposition 2.
ˆ Q refers to the target net which is slowly updated to stabilize training.
The same value function training method is adopted in (Andrychowicz et al., 2017; Fang et al., 2019).
The value function V ( s t , g (cid:48) ) is estimated using Q-value function as V ( s t , g (cid:48) )
=
Q ( s t , π ( s t +1 , g (cid:48) ) , g (cid:48) ) .
The entire algorithm is provided in Appendix C.
E XPERIMENTS
In our experiments, we demonstrate our proposed method on a diverse collection of continuous sparse-reward goal-conditioned tasks.
5. E NVIRONMENTS AND E XPERIMENTAL S ETTINGS
As shown in Figure 3, the introduced benchmark includes two point-based environments, and eight simulated robot environments.
In all tasks, the rewards are sparse and binary:
the agent receives a reward of +1 if it achieves a desired goal and a reward of 0 otherwise.
More details about those environments are provided in Appendix F. For the evaluation of ofﬂine goal-conditioned algorithms, we collect two types of ofﬂine dataset, namely ’random’ and ’expert’, following prior ofﬂine RL work (Fu et al., 2020).
Each dataset has the sample size of 2 × 10 6 for hard tasks (FetchPush,
FetchSlide, FetchPick and HandReach) and 1 × 10 5 for other relatively easy tasks.
The ’random’ dataset is collected by a uniform random policy, while the ’expert’ dataset is collected by the ﬁnal policy trained using online HER.
We also add Gaussian noise with zero mean and 0 .
2 standard deviation to increase the diversity of the ’expert’ dataset.
This may lead to the consequence that the deterministic policy learned by behavior cloning achieves a higher average return than that in the dataset, which has also been observed in previous works (Kumar et al., 2019; Fujimoto et al., 2019).
The average returns of the two datasets are provided in Appendix D. Beyond the ofﬂine setting, we also evaluate our method in online setting and the results are reported in Appendix E.8.
As for the implementation of GCSL and WGCSL, we utilize the Diagonal Gaussian policy with a mean vector π ( s t , g ) and a constant variance σ 2 for continuous control, which meets the requirements of a random policy.
In the ofﬂine setting, σ does not need to be deﬁned explicitly because we don’t need to collect samples.
The loss function of GCSL can be rewritten as:
[Equation]
Our implemented WGCSL learns a policy to minimize the following loss
[Equation]
where w i,t is deﬁned in Eq 5.
The policy networks of GCSL and WGCSL are 3-layer MLP with 256 units each layer and relu activation.
Besides, WGCSL learns a value network with the same structure except for the input and output layers.
The batch size is set as 128 for the ﬁrst 6 tasks, and 512 for harder tasks from FetchPush to HandReach.
We use Adam optimizer with a learning rate of 5 × 10 − 4 .
When evaluation, we use the deterministic policy
π ( s t , g ) without randomness.
For all the experiments we repeat for 5 different random seeds and report the mean with standard deviation.
[Table]
5. E XPERIMENTAL R ESULTS
In the ofﬂine goal-conditioned setting, we compare WGCSL with GCSL , HER and modiﬁed versions of most related ofﬂine works such as goal-conditioned MARWIL (Wang et al.,
( Goal MARWIL ), goal-conditioned behavior cloning (Bain & Sammut, 1995) ( Goal BC ), goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019)
( Goal BCQ ), and goalconditioned Conservative Q-learning (Kumar et al., 2020)
( Goal CQL ).
The results are presented in
Table 1 and Figure 4.
Expert Datasets The top row of Figure 4 reports the performance of different algorithms using the expert dataset of four hard tasks.
In all these tasks, WGCSL outperforms other baselines in terms of both learning efﬁciency and ﬁnal performance.
GCSL converges very slowly but can ﬁnally reach beyond the average return of expert dataset in FetchSlide and FetchPick.
Goal MARWIL is more efﬁcient than GCSL and achieves the second best performance.
However, none of the baseline algorithms succeeds in the HandReach task, which might because of high-dimensional states, goals, actions and the noise used to collect the dataset.
In addition, HER itself has the ability to handle ofﬂine dataset as the relabeled data is far off-policy (Plappert et al., 2018).
But we can conclude from Table 1 that the performance of HER is unstable and inconsistent across different tasks.
In contrast, learning curves of WGCSL are consistently stable.
It is also veriﬁed that techniques used in WGCSL are effective to generate general policy in the ofﬂine goal-conditioned RL settings.
Random Datasets
The bottom row of Figure 4 shows the training results of different methods on the random dataset.
We can observe that WGCSL outperforms other baselines by a large margin.
Baselines such as Goal MARWIL, Goal BC, Goal BCQ and Goal CQL can hardly learn to reach goals with the random dataset.
GCSL and HER can learn a relatively good policy in some tasks, which proves the importance of relabeling goals.
Interestingly, in tasks such as Reacher, SawyerReach and FetchReach, results of WGCSL and HER on the random dataset exceeds that of the expert dataset.
The possible reason is that the data coverage of random dataset could be larger than the expert dataset, and the agent can learn to reach more goals via goal relabeling
Ablation Studies We also include ablation studies for ofﬂine experiments in Figure 5.
In the ablation studies, we investigate the effect of the three proposed weights, i.e., DRW, GEAW and BAW, by applying them to GCSL respectively.
The results demonstrate that all three weights are effective compared to the plain GCSL, while GEAW and
BAW shows larger beneﬁts.
This is reasonable since
GEAW and BAW utilizes an additional value network while DRW does not.
Moreover, the results suggest that the learned policy can be improved by combining three weights.
Value Estimation Off-policy RL algorithms are prone to distributional shift and suffer from value overestimation problem on out-of-distribution actions.
As shown in Figure 6, DDPG and HER exhibit large estimated values during the training process.
In contrast, WGCSL has a more robust value approximation through weighted supervised learning, thus achieving higher returns in the HandReach task.
R ELATED W ORK
Solving Goal-Conditioned RL task is an important yet challenging domain in reinforcement learning.
In those tasks, the agent are required to achieve multiple goals rather than a single task (Schaul et al., 2015).
Such setting puts much burden on the generalization ability of the learned policy when dealing with unseen goals.
Another challenge arises in goal-conditioned RL is the sparse reward (Plappert et al., 2018).
HER (Andrychowicz et al., 2017) tackles the sparse reward issue via relabeling the failed rollouts as successful ones to increase the proportion of successful trails during learning.
HER is then extended to deal with demonstration data (Nair et al., 2018;
Ding et al., 2019) and combined with curriculum learning for exploration-exploitation trade-off (Fang et al., 2019).
Eysenbach et al.
(2020) further improves the goal sampling strategy from the perspective of inverse RL.
Zhao et al.
(2020) encourages agents to control their environment without external reward by optimizing a mutual information objective.
Different from prior works, we consider the ofﬂine setting, where the agent cannot access the environment to collect data in the training phase.
Our work has strong connections to imitation learning (IL).
Behavior cloning is one of the simplest
IL methods that learns a policy predicting the expert actions (Bain & Sammut, 1995; Bojarski et al., 2016).
Inverse RL (Ng et al., 2000) extracts a reward function from demonstrations and then uses the reward function to generate policies.
Several works also put forward to leverage self-imitation learning to improve the stability and efﬁciency of RL without expert experience.
Self-imitation learning (Oh et al., 2018) imitates the agent’s past advantageous decisions and indirectly drives deep exploration.
Policy Continuation with Hindsight Inverse Dynamics (Sun et al., 2019) introduces a dynamic programming method to learn policies progressively via self-imitation.
Play-LMP (Lynch et al., 2020) employs VAE to jointly learn a latent plan representation and goal-conditioned policy from replay without the need for rewards.
Our work is most relevant to GCSL (Ghosh et al., 2021), which iteratively relabels and imitates its own collected experiences.
The difference is that we further generalize GCSL to the ofﬂine goal-conditioned RL setting via weighted supervised learning.
Our work belongs to the scope of ofﬂine RL.
Learning policies from previously collected ofﬂine data is challenging for general off-policy RL algorithms due to error accumulation with distributional shift (Fujimoto et al., 2019; Kumar et al., 2019).
To address such issue, a series of algorithms are proposed to constrain the policy updates to avoid excessive deviation from the data distribution (Wang et al., 2018; Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Nair et al., 2020).
MARWIL (Wang et al., 2018) can learn a better policy than the behavior policy of the ofﬂine dataset with theoretical guarantees.
In addition to policy regularization, other works leverages techniques such as value underestimation (Kumar et al., 2020; Yu et al., 2020), robust value estimation via random ensemble (Agarwal et al., 2020) or data augmentation (Sinha & Garg, 2021) to handle distribution shift.
As ofﬂine RL can easily overﬁt to one task, generalizing ofﬂine RL to unseen tasks can be even more challenging.
Li et al.
(2019) tackles the challenge by leveraging the triplet loss for robust task inference.
Our method has similar exponential advantage weighted form as MARWIL, however, we tackle goal-conditioned tasks and introduces a general weighting scheme for WGCSL.
A most recent work (Chebotar et al., 2021) also addresses the ofﬂine goal-conditioned problem through underestimating the value of unseen actions.
The difference is that WGCSL is a weighted supervised learning method without explicitly value underestimation.
Therefore, WGCSL is substantially simpler, more stable, and more applicable for real world tasks.
C ONCLUSION
In this paper, we have proposed a novel algorithm, Weighted Goal-Conditioned Supervised Learning (WGCSL), to solve ofﬂine goal-conditioned RL problems with sparse rewards, by ﬁrst revisiting the Goal-Condition Supervised Learning (GCSL) and then generalizing GCSL to ofﬂine goalconditioned RL.
We further derive theoretical guarantees to show that WGCSL can consistently generate monotonically improving policies under certain conditions.
Experiments on an introduced benchmark demonstrate that WGCSL outperforms state-of-the-art ofﬂine approaches by a great margin in terms of learning efﬁciency and convergent performance.
For future directions, we are interested in building more stable and efﬁcient goal-conditioned RL algorithms based on WGCSL for real-world applications.
E THICS AND R EPRODUCIBILITY S TATEMENTS
Our work provides a way to learn general policy from ofﬂine data.
Learning policy for general purpose is one of the fundamental challenges in artiﬁcial general intelligence (AGI).
The proposed ofﬂine goal-conditioned algorithm can provide new insights into training agents with diverse skills, but currently we cannot foresee any negative social impact of our work.
The algorithm is evaluated on simulated robot environments, thus our experiment should not suffer from discrimination/bias/fairness concerns.
Appendix B includes the proofs of theories and propositions in the paper.
The details of experimental settings and additional results are also provided in the Appendix.
All the implementation code and ofﬂine dataset are available in our anonymous code link 2 .
A A C OMPREHENSIVE S UMMARY FOR WGCSL
WGCSL is strongly efﬁcient in the ofﬂine goal-conditioned RL due to three advantages:
(1) WGCSL learns a universal value function (Schaul et al., 2015), which is more general and robust than the value function learned in a single task.
(2) WGCSL utilizes goal relabeling and it contributes to augment huge amount of data ( (cid:0) T 2 (cid:1) times) for ofﬂine training, meanwhile goal relabeling alleviates the sparse reward issue.
(3) WGCSL leverages the goal-conditioned exponential advantage weight (GEAW), keeping the learned policy close to the ofﬂine dataset implicitly.
In addition to the GEAW, WGCSL utilizes the discounted relabeling weight and the bestadvantage weight to learn a better policy with theoretical guarantees.
B T HEORETICAL P ROOFS
In our proofs, we will assume discrete MPDs with discrete state and goal spaces.
B. P ROOF OF T HEOREM
Theorem.
Assume a discrete MDP with discrete state/goal spaces, a random discrete policy π which selects actions with non-zero probability and a sparse goal-conditioned reward function r ( s t , a t , g ) =
1[ φ ( s t )
= g ] , where φ is the state-to-goal mapping and 1[ φ ( s t ) =
g ] is an indicator function.
Given trajectory τ = ( s 0 , a 0 , s 1 , a 1 , · · · , s T ) and reward discount γ ∈ (0 , 1] , let the weight w i,t = γ
i − t , t ∈ [0 , T ] , i ∈
[ t, T ] , then the following bounds hold:
[Equation]
where
[Equation]
s t
[Equation]
g
[Equation]
g ]
[Equation]
is a surrogate function of J ( π )
[Equation]
Proof.
[Equation]
≥
[Equation]
Eq. 6 holds because of the non-positive logarithmic function over π .
Compared to the surrogate function used in Appendix B.1 of (Ghosh et al., 2021), we further consider the discounted cumulative return.
The surrogate function J surr ( π ) shares the same gradient with J ( π ) after scaling with 1 T under certain conditions.
The theorem shows that WGCSL optimizes over a tighter lower bound of
J surr ( π ) compared to GCSL.
Remark.
While Theorem 1 bounds J WGCSL and J GCSL with the surrogate function J surr , here we bridge the gap between J surr and the goal-conditioned RL objective J . We ﬁrst show that log J is an upper bound of J surr in addition to a constraint and KL divergence. Let Rτ =
(cid:80) Ti =0 γ i · 1[ φ ( s i ) =
g ] and C ( g ) = (cid:82) π b ( τ | g ) R ( τ ) d τ , and assume C ( g ) > 0 . Since practically g is sampled from ofﬂine dataset, we can ﬁnd C m = min g C ( g ) . Similarly to (Kober & Peters, 2014)
[Equation]
we have following results using Jensen’s inequality and absorbing constants into ’constant’.
[Equation] Naturally, in ofﬂine setting, we explicitly or implicitly bound the KL divergence between π b and π . Therefore, we can regard J WGCSL as a tighter lower bound of log J than J GCSL . Furthermore, We show that J surr ( π ) shares the same gradient with J ( π ) when π = π b after scaling with 1 T .
[Equation]
τ π b ( τ | g ) ∇ log π b ( τ | g )
T (cid:88) i
= γ i · 1[ φ ( s i ) =
g ] dτ (cid:35) = E g ∼ p ( g ) ,τ ∼ π b ( ·| g ) (cid:34) T (cid:88) t = ∇ log π b ( a t | s t , g ) T (cid:88) i
= γ i · 1[ φ ( s i ) =
g ] (cid:35) =
E g ∼ p
( g ) ,τ ∼ π b ( ·| g ) (cid:34) T (cid:88) t = ∇ log π b ( a t | s t , g )
T (cid:88) i =
t γ i · 1[ φ ( s i ) =
g ] (cid:35) =
T · ∇ J surr ( π b ) .
Therefore, taking a gradient update with a proper step size on J surr can also improve the policy on the objective J .
In our algorithm, the exponential weight poses an implicit constraint on the policy update similar to Nair et al.
With a proper constraint, optimizing on the J surr is equivalent as optimizing on the goal-conditioned objective J .
Intuitively, J surr maximizes the likelihood for state-action pairs with large returns, and therefore it can lead to policy improvement based on the ofﬂine dataset.
B. P ROOF OF C OROLLARY
Corollary.
Suppose function h ( s, a, g ) ≥ 1 over the state-action-goal combination.
Given trajectory τ = ( s 0 , a 0 , s 1 , a 1 , · · · , s T ) , let w i,t = γ
i − t h
( s t , a t , φ ( s i )) , t ∈ [0 , T ] , i ∈ [ t, T ] , then the following bounds hold:
[Equation]
Proof.
Similarly to the proof of Theorem 1, we have
[Equation]
, a t , φ ( s i )) log π ( a t | s t , φ ( s i ))
(cid:3) (cid:44) T · J WGCSL ( π ) .
B. P ROOF OF P ROPOSITION
Proposition.
˜ π is uniformly as good as or better than π relabel .
That is, ∀ s t , φ ( s i )
∈ D τ
and i ≥ t ,
V ˜ π ( s t , φ ( s i )) ≥ V π relabel ( s t , φ ( s i )) , where D τ contains the experiences after goal relabeling.
Proof.
We consider the compound state ˆ s =
( s t , φ ( s i )) from the relabeled distribution D τ .
When ˆ s is ﬁxed (the index of i, t are ﬁxed), γ i − t is a constant, and f (ˆ s, A (ˆ s, a ))
=
clip ( A (ˆ s, a t ) , −∞ , log M ) + log( γ i − t )
+ log( (cid:15)
( A (ˆ s, a t )))
+ C (ˆ s ) is a monotonically non-decreasing function for ﬁxed ˆ s , where C (ˆ s ) is a normalizing factor, M is the positive maximum exponential advantage value, and (cid:15)
( A (ˆ s, a t )) is a non-decreasing function introduced in Section 3.2 with minimum (cid:15) min > 0 .
Leveraging the Proposition 1 in MARWIL (Wang et al., 2018), we can conclude that V ˜ π
(ˆ s ) ≥ V π relabel (ˆ s ) .
B. P ROOF OF P ROPOSITION
Proposition.
Assume the state space can be perfectly mapped to the goal space, i.e., φ ( s t ) ∈
G, ∀ s t ∈ S , and deﬁne a relabeling strategy using mixed future states as g (cid:48) t = (cid:26) φ ( s i ) , ∀ i ≥ t, if φ ( s i ) (cid:54) = g, ∀ i ≥ t, φ ( s i ) , ∀ i ∈ { i | φ ( s i )
= g, i ≥ t } , otherwise.
Then, the expected return after relabeling is uniformly better than the original return, i.e., ∀ s t , g, s i ∈
D τ , E τ ∼ π b ,s t ,s i ∼ τ
[ V π relabel ( s t , φ ( s i ))
] ≥ E τ ∼ π b ,s t ,g ∼ τ [ V π b ( s t , g )] , where π b is the behavior policy forming the dataset.
Proof.
Denote the trajectory starting from t : τ t : T = s t , a t , s t +1 , a t +1 , . . .
, s T , and we have
[Equation]
[Equation]
We consider two trajectory sets collected by π b : A 1 = { τ t : T |∃ i ≥ t, φ ( s i ) =
g } and
A= { τ t : T |∀ i ≥ t, φ ( s i ) (cid:54) = g } .
We use subscripts to denote values of different sets
[Equation]
γ
[Equation]
t · [ φ
[Equation]
s j ) =
g ] (cid:3) .
When there exists a state s i that achieves the desired goal φ ( s i )
=
g, i ≥ t , using the relabeling method in Proposition 2 we have:
[Equation]
If there is no successful future state, the relabeled trajectories contain at least a positive reward at step i , then V π relabel A
[-≥ γ i − t > 
-][Equation] g ) .
By combining the two trajectory sets, we can prove that: ∀ τ ∼ π b , ( s t , g, s i ) ∼ τ , using the mixed relabeling strategy in Proposition 2, the relabeled return is uniformly better than the original return:
[Equation]
where p ( A 1 ) , p ( A 2 ) > 0 are the probabilities that trajectories collected by π b belong to set A 1 and
A 2 .
Only when p ( A 1 ) = 1 , p ( A 2 ) = 0 , the equal sign is established.
Remark.
The relabeling strategy of Proposition 2 requires a complete scan of all future states of the trajectory, which is costly.
Empirically we ﬁnd that relabeling with random future states has comparable performance, and is very simple and efﬁcient to implement.
We provide the comparison between the relabeling strategy of Proposition 2 (called WGCSL-slow-relabel) and the relabeling strategy that we used in practice in Table 2.
[Table]
-expert 10.73 ( ± 1.09) 9.34 ( ± 1.71)
FetchPick-expert 34.37 ( ± 0.51) 33.99 ( ± 1.81)
HandReach-expert 26.73 ( ± 1.20) 28.61 ( ± 1.80)
PointRooms-random 35.52 ( ± 0.80) 36.56 ( ± 1.28)
FetchReach-random 46.50 ( ± 0.09) 46.61 (
[Table]
FetchPush-random 11.48 ( ± 1.03) 11.64 ( ± 2.07)
FetchSlide-random 2.34 ( ± 0.28) 1.81 (
[Table]
FetchPick-random
[Table]
[Table]
[Table]
[Table]
[Table]
HandReach-random
[Table]
[Table]
[Table]
[Table]
C A LGORITHM
Here we summarize our approach WGCSL into Algorithm 1.
Given ofﬂine dataset D , the best advantage percentile threshold C p ;
Initialize policy π θ and value function Q ;
Initialize a small buffer B = {} to save recent advantage values; for training step = 1 , 2 , . .
. , N do
Sample a mini-batch from the ofﬂine dataset: { ( s t a t g r t ) D ;
Relabel the mini-batch using future goals: { ( s t , a t , φ ( s i ) , r ( s t , a t , φ ( s i )))
i ≥ t } ∼ D τ ;
Update value function Q to minimize TD error L TD on the relabeled mini-batch;
Estimate advantage value A ( s t , a t , φ ( s i )) using value function Q ; Update buffer B with the estimated advantage values; Get the C p percentile advantage value ˆ A from B ; Update policy π θ to maximize the weighted imitation learning objective in Eq.
4 with the relabeled mini-batch:
[Table]
i )
[Table]
exp clip ( A ( s t , a t , φ ( s i ))) · (cid:15) ( A ( s t , a t , φ ( s i )))] end for
[Table]
D O FFLINE S ETTINGS
In the ofﬂine setting, we only use the collected data set to train the agent, without additional interaction with the environment.
The difference between ofﬂine goal-conditioned dataset and the general ofﬂine dataset is that it also needs to save the original desired goals and the achieved goals.
We follow previous work (Fujimoto et al., 2019; Fu et al., 2020) to introduce two different dataset settings:
• Random dataset: using a uniformly random policy to collect trajectories.
• Expert dataset: using the ﬁnal online policy trained by HER to collect data, additional Gaussian noise with zero mean and 0 .
2 standard deviation is used for diversity.
For the dataset of six easy tasks (PointReach, PointRooms, Reacher, SawyerReach, SawyerDoor,
FetchReach), we save a dict of 1 × 10 5 transitions (2000 trajectories) containing observations, actions, desired goals and achieved goals.
Regarding the other four hard tasks (FetchPush, FetchSlide,
FetchPick, HandReach), we store 2 × 10 6 transitions in the ofﬂine dataset.
The rewards can be computed by leveraging the corresponding interface (namely env.compute reward ) of the environment.
The average returns of the two datasets are shown in Table 3.
Average returns of the expert dataset are close to that of the optimal policy, while average returns of the random dataset are close to 0.
In the ofﬂine goal-conditioned setting, we denote the ofﬂine dataset as D and consider the following methods and baselines
• WGCSL : using ofﬂine samples after relabeling (denoted as D τ ) to maximize
[Equation]
• GCSL : GCSL (Ghosh et al., 2021) without online interaction, only using ofﬂine samples after relabeling (denoted as D τ ) to maximize
[Equation]
• Goal MARWIL : goal-conditioned MARWIL (Wang et al., 2018), using ofﬂine samples to maximize
[Equation]
To fairly compare with our method, Goal MARWIL is implemented with actor-critic style similar to WGCSL, therefore Goal MARWIL can also be treated as ’Goal AWAC’ (Nair et al., 2020).
• Goal AWR : goal-conditioned AWR (Peng et al., 2019), using ofﬂine samples to maximize
[Equation]
exp( R t − V ( s t , g ))]
[Equation]
where R t = (cid:80) T − t i =
γ i r t
+ i .
Another difference between our Goal AWR and Goal MARWIL is that we use Monte Carlo return R t to train the value function.
• Goal BC : goal-conditioned behavior cloning (Bain & Sammut, 1995), using ofﬂine samples to maximize
[Equation]
g ) [Equation] • Goal BCQ : goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019), which we use the ofﬁcial code base and concatenate observations, desired goals and achieved goals as states.
The objective function of Goal BCQ is as follows:
[Equation]
s t , a, g, Φ) , g )] , where ξ ( s t , a, g, Φ) is the goal-conditioned perturbation model, which outputs an adjustment to an action a within the range [ − Φ , Φ] .
G w is the VAE ﬁtted on the behavior policy of the ofﬂine dataset.
The policy is optimized through optimizing ξ .
The Q function is learned using the Clipped Double Q-learning in (Fujimoto et al., 2019), and the only difference is that the Q function also includes goals g as input.
• Goal CQL : goal-conditioned Conservative Q-Learning
(Kumar et al., 2020), which we also use the ofﬁcial code base and concatenate observations, desired goals and achieved goals as states.
The objective function of Goal CQL is as follows:
[Equation]
which jointly optimizes estimated return and policy entropy.
The Q function of CQL is learned by minimizing the following loss:
[Equation]
where π b is the behavior policy for the ofﬂine dataset, which can be learned through imitation learning.
B π is the Bellman operator.
• HER : Hindsight Experience Replay (Andrychowicz et al., 2017), using ofﬂine samples after relabeling to maximize
[Equation]
where g (cid:48) =
φ ( s i ) , i ≥ t and the Q function is learned by minimizing TD error:
[Equation]
For fair comparison, all the implementation of WGCSL, GCSL, Goal MARWIL, Goal AWR and
Goal BC use the Diagnal Gaussian policy with a mean vector π ( s, g ) and a non-zero constant variance σ 2 , where σ is set as 0.2.
WGCSL, Goal MARWIL and Goal AWR share the same network structure, i.e., the value function and the policy network (along with their target networks) are both 3-layer MLPs with 256 units each layer and relu activation.
Besides, GCSL and Goal BC both keep a single policy network without target network and value function.
We also normalize the input with estimated mean and standard deviation for WGCSL and other baselines.
All the hyper-parameters of different algorithms are kept the same for each task, such as batch size, and Adam optimizer with learning rate 5 × 10 − 4 .
As regards to Goal BCQ and Goal CQL, we use batch size of 128 and other parameters are set as default (we ﬁnd that default parameters perform equally well, if not better than others, in the grid search of critical parameters).
To obtain higher performance of BCQ and CQL, we train them with 4 ∼ 8 × training steps.
All the implementation code is available in our anonymous code link. [Figure] r [Figure]
t
[Figure] r n [Figure] E A DDITIONAL E XPERIMENTAL R ESULTS
In this section [Figure] we provide more empirical results in both ofﬂine and online goal-conditioned RL setting, including average return, ﬁnal distance of the ofﬂine setting, results on small ( 10% ) dataset, and the results of online experiments.
E. F ULL O FFLINE R ESULTS
We provide the learning curves of average return of all tasks in Figure 7.
WGCSL performs consistently well across ten environments, especially when handling hard tasks (FetchPush, FetchSlide,
FetchPick, HandReach) and learning from the random ofﬂine dataset.
Other conclusions have been discussed in Section 5.2.
Comparing AWR with MARWIL, the major difference is using Monte
Carlo return to replace the TD estimation for learning both value function and policy.
Hence, we can ﬁnd that TD bootstrap learns more efﬁciently in weighted supervised learning, which is similar to the conclusion of AWAC (Nair et al., 2020).
In addition, we present all ﬁnal performance of different algorithms in Table E. [Figure] e r age R e t u r n
[Figure]
v e r age R e t u r n [Figure] E. C OMPARISON R ESULTS WITH A CTIONABLE M ODELS , HER AND DDPG
In this subsection [Figure] we make comparison with Actionable Models (AM) (Chebotar et al., 2021), which applies the conservation estimation idea similar to (Kumar et al., 2020) into ofﬂine goalconditioned RL.
There are some setting difference between AM and ours.
For example, we consider cumulative return of the maximum horizon, i
[Figure]
e., the agent doesn’t terminate interaction after reaching the goal.
To fairly compare with AM, we implement AM based on HER.
Speciﬁcally, we revise the value update as:
[Equation]
In the formula, AM tries to minimize TD error and minimize unseen Q value at the same time.
The action a is sampled according to exponential Q value.
In our implementation, we sample 20 actions from the action space and compute their exponential Q value in order to sample a .
The results are shown in Figure 8.
We can observe that DDPG and HER are unstable during training, with large variance and performance drops at the end.
In both expert and random dataset, HER outperforms DDPG, which demonstrates the beneﬁts of goal relabeling for ofﬂine goal-conditioned RL.
Besides, Actionable Model (AM) achieves higher ﬁnal performance than HER and DDPG especially in expert dataset, which indicates the conservation technique is useful for ofﬂine goal-conditioned
RL.
Moreover, WGCSL is more efﬁcient than AM, especially in harder tasks with larger action space.
Intuitively, WGCSL promotes the probability of visited actions, while AM restrains the probability of unvisited actions.
For tasks with large action space, conservation strategy can be ineffective compared to WGCSL.
[Figure]
E. N ORMALIZED S CORE A CROSS 10 T ASKS
For clarity, we report the normalized score across 10 tasks to compare different algorithms in Figure 9.
We can observe the signiﬁcant improvement of WGCSL over other baselines.
We take an insight into the impact of the goal relabeling technique.
For expert datasets, goal relabeling is not a dominant factor, e.g., the learning curve of Goal BC and GCSL are very close.
In random datasets, goal relabeling is essential to reach a high performance.
In addition to goal relabeling, WGCSL contains other techniques to jointly promote the performance in ofﬂine goal-conditioned RL tasks.
Other conclusions are the same as prior subsections (Appendix E.1 and Appendix E.2).
E. C OMPARISON OF T RAINING T IME
We further analyze the computation cost of the four most effective algorithms in the random ofﬂine setting, i.e., WGCSL, GCSL, HER and Actionable Models (AM).
The results are shown in Figure 10, from which we can conclude that GCSL needs the minimal training , while WGCSL and HER require similar cost.
It needs to be emphasized that WGCSL has only a small amount of additional training time over GCSL, but it achieves signiﬁcantly higher performance.
Therefore, we believe that the cost of learning the value function on top of GCSL is worthwhile.
Moreover, AM requires [Figure] the largest computation among these algorithms, which is mainly because it needs to sample several actions and compute their exponential Q value for constraining the large Q value.
These procedures make AM the most time-consuming algorithm.
In contrast, WGCSL can achieve higher performance with little additional trainin time [Figure]
E. A VERAGE P ROBABILITY OF I MPROVEMENT
In this subsection [Figure] we adopt the average probability of improvement (Agarwal et al., 2021), a robust metric to measure how likely it is for one algorithm to outperform another on a randomly selected task.
The results are reported in Figure 11.
As shown in the results, WGCSL robustly outperforms many baselines in both expert and random datasets.
For instance, WGCSL is 100% better than 4 baselines on the expert dataset, and 6 baselines on the random dataset.
Note that the most two effective and robust algorithms on both random and expert datasets are WGCSL and Actionable
Models (AM), which are speciﬁcally designed for ofﬂine goal-conditioned RL.
Comparing the two algorithms, WGCSL outperforms AM with a probability of 75% on the expert dataset and 62 .
5% on the random dataset.
E. F INAL D ISTANCES IN THE O FFLINE S ETTING
GCSL (Ghosh et al., 2021) uses the ﬁnal distance to desired goal as a measure for goal-reaching problems.
We also provide average ﬁnal distances in the ofﬂine setting to evaluate the learned goal-conditioned policy in Figure 12.
From the results we can conclude that WGCSL substantially outperforms other baselines in both random and expert dataset.
Although GCSL learns a suboptimal
[Figure]
policy, it also achieves relatively consistent ﬁnal distance across different environments and ofﬂine datasets.
Besides, Goal MARWIL and Goal BC are two effective baselines when learning from the expert dataset, but they do not perform well on the random dataset.
In addition, HER and
DDPG learns an unstable results mainly because of bootstrapping from out-of-distribution actions
[Figure] Interestingly, in tasks such as SawyerDoor-expert, PointReach-random, HER learns with a slightly higher average return [Figure] but WGCSL achieves a smaller ﬁnal distance.
E. R ESULTS ON 10% OF O FFLINE D ATASET
To evaluate the impact of dataset size on the learning performance
[Figure] we select 10% of samples (i [Figure] e., 2 × 10 5 transitions) from the ofﬂine dataset of hard tasks (FetchPush, FetchSlide, FetchPick, HandReach).
The results are reported in Figure 13 and Figure 14.
From the results, we can conclude that with only 10% ofﬂine data WGCSL consistently achieves the best performance out of these algorithms.
In Figure 14, we can see that WGCSL is very robust to the data size in the expert dataset, while the performance is affected more by the small data size in the random dataset.
The probable reason is that in the ofﬂine setting, the number of samples inﬂuence the data coverage greatly and learning generalizable value function from a small set of data is very difﬁcult. [Figure] [Figure] v
[Figure]
r age
R e t u r n
[Figure]
E. O NLINE E XPERIMENTS
We introduce implementation details and hyper-parameters of the online setting in this section.
As
GCSL is originally proposed for online goal-conditioned RL, weighted GCSL can also be applied to online goal-conditioned setting.
In the online training setting, the agent iterates between collecting trajectories from the environment and training with samples from the replay buffer.
For easier tasks, i.e., Point2DLarge, Point2D-FourRooms, SawyerReach, FetchReach, the buffer size is set as 1 × 10 4 , and for other tasks, i.e., Reacher and SawyerDoor, the buffer size is set as 5 × 10 4 .
Before training starts, we ﬁrst randomly sample 20 trajectories to initialize the buffer.
The policies of WGCSL and
GCSL are both Diagonal Gaussian policy with a mean vector π θ ( s, g ) parameterized as θ and a non- zero constant variance σ 2 .
The standard deviation σ is set as 0.2.
All the policy networks and value networks are the same as that used in the ofﬂine setting.
When interacting with the environments, we use additional random actions for exploration with a probability of 0 . 3 .
After data collection, we sample a mini-batch from the replay buffer and perform goal relabeling for WGCSL and GCSL.
Then we train the agent for 1 mini-batch (Point2DLarge, Point2D-FourRooms) or 4 mini-batches for other tasks.
The batch size, optimizer, learning rate keeps the same as the ofﬂine setting.
The maximum clipping upper bound M for WGCSL is set as 10.
We also utilize target nets for WGCSL to stabilize training, and the polyak average coefﬁcient for target nets are 0.9.
With respect to the evaluation phase, we use the deterministic policy π θ ( s t , g ) without randomness.
After each epoch, we evaluate the policy for 100 episodes and compute the average success rate.
For all the experiments we repeat for 10 different random seeds and report the mean performance with standard deviation.
We report the results of WGCSL with the discounted relabeling weight (DRW) and the goal-conditioned exponential advantage weight (GEAW).
We provide empirical results regarding the average success rate of the online setting in Figure 15.
From Figure 15 we can conclude that WGCSL achieves better success rate and converges faster than other baselines in most of the environments except SawyerReach.
Therefore, WGCSL is proved to be highly efﬁcient in both online and ofﬂine goal-conditioend RL, and we hope it could be a stepping stone for future real-world RL applications.
[Figure]
[Figure]
E.
T HE I MPACT OF C LIPPING IN E XPONENTIAL A DVANTAGE W EIGHTS
We study the impact of clipping in WGCSL’s exponential advantage weights by comparing with the following variants:
• WGCSL w/o clip: no clipping weights performed.
• WGCSL clip 5: clipping the exponential advantage weights to (0 , 5] .
• WGCSL clip 10: clipping the exponential advantage weights to (0 , 10] , which is used in our paper.
From the results in Figure 16 we can observe that without clipping the goal-conditioned advantage weights can harm the performance in Reacher and SawyerDoor.
Therefore, we perform clipping weights by default in our paper.
F D ETAILS OF E NVIRONMENTS
All of the ﬁve tasks have continuous state space, action space and goal space.
The maximum episode horizon is set as 50.
In this section, we introduce these tasks in detail.
PointReach PointReach is taken from the open-source environment multi-world
and it requires the blue point to reach the green circle.
The state space [ − 5 , 5] × [ − 5 , 5] has two dimensions representing Cartesian coordinates of the blue point, and the action space [ − 1 , 1] × [ − 1 , 1] also has two dimensions meaning the horizontal and vertical displacement.
The goal space is the same as state space, which means φ ( s )
= s .
The bule point and the green circle are randomly initialized in the state space.
The allowable error (cid:15) of reaching goal is the radius of the target circle and is set as 1 .
The reward function is deﬁned as:
[Equation]
PointRooms
The PointRooms environment is built on multi-world .
The state space, the action space, the goal space, and the reward function are same as PointReach.
The difference is that there are four rooms separated by gray walls.
Reacher The Reacher environment is revised from OpenAI Gym (Brockman et al., 2016).
States are 11-dimensional, which indicates the angles, the positions, and the velocity of the joints.
Actions are 2 -dimensional and control the movement of two joints.
The goals are 2 -dimension representing the expected XY position.
And the state-to-goal mapping is φ ( s ) =
s [ − 3 : − 1] , where the last three dimensions are the XYZ position of the end-effect.
The reward function is deﬁned as:
[Equation]
where the allowable error (cid:15) is set as 0 .
SawyerReach The SawyerReach environment is taken from multi-world .
The Sawyer robot aims to reach a target position with its end-effector.
The observation space is 3-dimensional, representing the 3D Cartesian position of the end-effector.
Correspondingly, the goal space is 3-dimensional and describes the expected position, and the state-to-goal mapping is φ ( s )
= s .
Besides, the action space has 3 dimensions describing the next position of the end-effector.
The reward function is deﬁned as:
[Equation]
where the allowable error (cid:15)
= 0 .
SawyerDoor
The SawyerDoor environment is revised from multi-world .
The Sawyer robot is required to open the door to a desired angle.
The state space (4-dimensional) consists of the Cartesian coordinates of the Sawyer end-effector and the door’s angle.
As in the SawyerReach task, the action space is 3 -dimensional and controls the position of the end-effector.
The desired goals are uniform from 0 to 0.83 radians.
And the reward function is deﬁned as:
[Equation]
where the allowable error (cid:15) is set as 0 .
FetchReach The FetchReach environment is taken from OpenAI Gym (Brockman et al., 2016;
Plappert et al., 2018).
In this environment, a 7-DoF robotic arm is expected to touch a desired location with its two-ﬁnger gripper.
The state space is 10-dimensional, including the gripper’s position and linear velocities.
The action space is 4-dimensional, which represents the gripper’s movements and its status about the opening and closing.
Moreover, the goals are 3-dimensional vectors representing the target place of the gripper.
The state-to-goal mapping is φ ( s ) =
s [0 : 3] , because the ﬁrst 3 dimensions of state describe the position of the gripper.
The allowable error in FetchReach is (cid:15) = 0 .
The reward function is deﬁned as:
[Equation]
FetchPush Similar to FetchReach, the FetchPush environment is taken from OpenAI Gym where a 7-DoF robotic arm is controlled to move a box to the desired location.
The state space is 25dimensional, including the gripper’s position, linear velocities, and the box’s position, rotation, linear and angular velocities.
The action space is also 4-dimensional as in FetchReach.
Different from
FetchReach, the 3-dimensional goals and achieved goals represent the target place of the box.
The state-to-goal mapping is φ ( s ) =
s [3 : 6] , because the 3 dimensions of state describe the position of the box.
The allowable error in FetchPush is (cid:15) = 0 . 05 .
The reward function is deﬁned as:
[Equation]
FetchSlide The FetchSlide environment is like FetchPush, and controls a 7-DoF robotic arm to slide a box to the desired location (far away from the starting position).
The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as
FetchPush.
The difference is that the target position is outside of the robot’s reach thus it has to hit the box and makes the box slides and then stops at the target position.
FetchPick Like FetchPush and FetchSlide environments, the FetchPick environment controls a 7-DoF robotic arm to pick a box up to the desired location (far away from the starting position).
The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush.
The difference is that the target position may be on the table or in the air, thus the agent needs to pick the box with its gripper.
HandReach In HandReach environment, the agent is required to controls a 24-DoF anthropomorphic hand to use its ﬁngers to reach the target place or manipulate a speciﬁc object (e.g., a block) to the desired position.
Observations is 63-dimensional containing two 24-dimensional vectors about positions and velocities of the joints, and additional 15 dimensions indicating current state of ﬁngertips.
Actions are 20-dimensional vectors and control the non-coupled joints of the hand.
Goals have 15 dimensions for HandReach representing the target Cartesian position of each ﬁngertip.
The reward function is the same as Fetch tasks except the state-to-goal-mapping ( φ = s [ − 15 :] ) and the allowable threshold ( (cid:15) = 0 . 01 ).
[Table]