S KEW -F IT : S TATE -C OVERING S ELF -S UPERVISED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Autonomous agents that must exhibit ﬂexible and broad capabilities will need to be equipped with large repertoires of skills.
Deﬁning each skill with a manuallydesigned reward function limits this repertoire and imposes a manual engineering burden.
Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difﬁcult, and often involves heuristic design decisions.
In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage.
We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations.
To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions.
Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals.
We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand.
Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.
I NTRODUCTION
Reinforcement learning (RL) provides an appealing formalism for automated learning of behavioral skills, but separately learning every potentially useful skill becomes prohibitively time consuming, both in terms of the experience required for the agent and the effort required for the user to design reward functions for each behavior.
What if we could instead design an unsupervised RL algorithm that automatically explores the environment and iteratively
distills this experience into general-purpose policies that can accomplish new user-speciﬁed tasks at test time?
For an agent to learn autonomously, it needs an exploration objective.
In the absence of any prior knowledge about which states are more useful, an effective exploration scheme is one that visits as many states as possible, allowing a policy to autonomously prepare for user-speciﬁed task that it might see at test time.
This objective has been formalized as maximizing the entropy of the learned policy’s visited state distribution 1 H ( S ) (Hazan et al., 2018a), since a policy that maximizes this objective should approach a uniform distribution over valid states.
Unfortunately, directly optimizing H ( S ) requires an accurate model of the policy and environment (Hazan et al., 2018a).
Moreover, even if this optimization were tractable, another short-coming of this objective is that the resulting policy cannot be used to solve new tasks: it only knows how to maximize state entropy.
In other words, to develop principled unsupervised RL algorithms that result in useful policies, maximizing H ( S ) is not enough.
We need a mechanism that allows us to control the resulting policy to achieve new tasks at test-time.
We argue that this can be accomplished by performing goal-directed exploration .
In addition to maximizing the state entropy, we should be able to control where the policy goes by giving it a goal
G that corresponds to a state that it must reach.
Mathematically, a goal-conditioned policy should minimize the conditional entropy over the states given a goal, H ( S | G ) .
This objective provides us with a principled way for training a policy to explore all states, by maximizing H ( S ) , such that the state that is reached can be controlled by commanding goals, which means minimizing H ( S | G ) .
Directly using this objective is often intractable, since it requires optimizing the entropy of the marginal state distribution of the policy, H ( S ) .
However, we can sidestep this issue by noting that the objective is the mutual information between the state and the goal, I ( S ; G ) , which can be written as:
[Equation]
Equation 1 thus gives an equivalent objective for an unsupervised RL algorithm: the agent should set diverse goals, maximizing H ( G ) , and learn how to reach them, minimizing H ( G | S ) .
While the second term is the typical objective studied in goal-conditioned RL (Kaelbling, 1993;
Andrychowicz et al., 2017), maximizing the diversity of goals is crucial for effectively learning to reach all possible states.
In a new environment, acquiring such a maximum-entropy goal distribution is challenging: how can an agent set diverse goals when it does not even know what states exist?
In this paper, we address this question via a new algorithm, Skew-Fit, which learns to model the uniform distribution over states, given only access to data collected by an autonomous goalconditioned policy.
Our paper makes the following contributions.
First, we propose a principled objective for unsupervised RL, based on Equation 1.
While a number of prior works ignore the H ( G ) term, we argue that jointly optimizing the entire quantity is needed to develop effective and useful exploration.
Second, we propose a method called Skew-Fit and prove that, under some regularity conditions, it learns a generative model that converges to a uniform distribution over the goal space, even when the set of valid states is unknown (e.g., as in the case of images).
Third, we empirically demonstrate that, when combined with goal-conditioned RL, Skew-Fit allows us to autonomously train goal-conditioned policies that reach diverse states.
We test this method on a variety of simulated vision-based robot tasks without any task-speciﬁc reward function.
In these experiments, Skew-Fit reaches substantially better ﬁnal performance than prior methods, and learns much more quickly.
We also demonstrate that our approach solves a real-world manipulation task, which requires a robot to learn to open a door from scratch in about ﬁve hours, directly from images, and without any manually-designed reward function.
P ROBLEM F ORMULATION
To ensure that an unsupervised reinforcement learning agent learns to reach all possible states in a controllable way, we maximize the mutual information between the state S and the goal G ,
I ( S ; G ) , as stated in Equation 1.
This section discusses how to optimize Equation 1 by splitting the optimization into two parts: minimizing H ( G | S ) and maximizing H ( G ) .
2. M INIMIZING H ( G | S ) : G OAL -C ONDITIONED R EINFORCEMENT L EARNING
Standard RL considers a Markov decision process (MDP), which has a state space S , action space A , and unknown dynamics ρ ( s t +1 | s t , a t ) : S ×S ×A (cid:55)→ [0 , + ∞ ) .
Goal-conditioned RL also includes a goal space G .
For simplicity, we will assume in our derivation that the goal space matches the state space, such that G = S , though the approach extends trivially to the case where G is a hand-speciﬁed subset of S , such as the global x-y position of a robot.
A goal-conditioned policy π ( a | s , g ) maps a state s ∈ S and goal g ∈ S to a distribution over actions a ∈ A , and its objective is to reach the goal, i.e., to make the current state equal to the goal.
Goal-reaching can be formulated as minimizing H ( G | S ) , and many practical goal-reaching algorithms (Kaelbling, 1993; Lillicrap et al., 2016; Schaul et al., 2015; Andrychowicz et al., 2017;
Nair et al., 2018;
Pong et al., 2018; Florensa et al., 2018a) can be viewed as approximations to this objective by observing that the optimal goal-conditioned policy will deterministically reach the goal, resulting in a conditional entropy of zero: H ( G | S )
= 0 .
See Appendix E for more details.
Our method may thus be used in conjunction with any of these prior goal-conditioned RL methods in order to jointly minimize H ( G | S ) and maximize H ( G ) .
2. M AXIMIZING H ( G ) : S ETTING D IVERSE G OALS
We now turn to the problem of setting diverse goals or, mathematically, maximizing the entropy of the goal distribution H ( G ) .
Let U S be the uniform distribution over S , where we assume S has ﬁnite volume so that the uniform distribution is well-deﬁned.
Let p φ be the goal distribution from which goals G are sampled.
Our goal is to maximize the entropy of p φ , which we write as H ( G ) .
Since the maximum entropy distribution over S is the uniform distribution U S , maximizing H ( G ) may seem as simple as choosing the uniform distribution to be our goal distribution: p φ = U S .
However, this requires knowing the uniform distribution over valid states, which may be difﬁcult to obtain when
S is a subset of R n , for some n .
For example, if the states correspond to images viewed through a robot’s camera, S corresponds to the (unknown) set of valid images of the robot’s environment, while R n corresponds to all possible arrays of pixel values of a particular size.
In such environments, sampling from the uniform distribution R n is unlikely to correspond to a valid image of the real world.
Sampling uniformly from S would require knowing the set of all possible valid images, which we assume the agent does not know when starting to explore the environment.
While we cannot sample arbitrary states from S , we can sample states by performing goal-directed exploration.
To derive and analyze our method, we introduce a simple model of this process: a goal
G ∼ p φ is sampled from the goal distribution p φ , and then the agent attempts to achieve this goal, which results in a distribution of states S ∈ S seen along the trajectory.
We abstract this entire process by writing the resulting marginal distribution over S as p ( S | p φ ) .
We assume that p ( S | p φ ) has full support, which can be accomplished with an epsilon-greedy goal reaching policy in a communicating
MDP.
We also assume that the entropy of the resulting state distribution H ( p ( S | p φ )) is no less than the entropy of the goal distribution H ( p φ ( S )) .
Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are.
Note that this assumption does not require that the entropy of p ( S | p φ ) is strictly larger than the entropy of the goal distribution, p φ .
This simpliﬁed model allows us to analyze the behavior of our goal-setting scheme separately from any speciﬁc goal-reaching algorithm.
We will however show in Sectionthat we can instantiate this approach into a practical algorithm that jointly learns the goal-reaching policy.
In summary, our goal is to acquire a maximum-entropy goal distribution p φ over valid states
S , while only having access to state samples from p ( S | p φ ) .
S KEW -F IT : L EARNING A M AXIMUM E NTROPY G OAL D ISTRIBUTION
Our method, Skew-Fit, learns a maximum entropy goal distribution p φ using samples collected from a goal-conditioned policy.
We analyze the algorithm and show that Skew-Fit maximizes the entropy of the goal distribution, and present a practical instantiation for unsupervised deep RL.
3. S KEW -F IT A LGORITHM
To learn a uniform distribution over valid goal states, we present a method that iteratively increases the entropy of a generative model p φ .
In particular, given a generative model p φ t at iteration t , we would like to train a new generative model p φ t +1 such that p φ t +1 has higher entropy than p φ t over the set of valid states.
While we do not know the set of valid states S , we can sample states from p ( S | p φ t ) , resulting in an empirical distribution p emp t over the states
[Equation]
and use this empirical distribution to train the next generative model p φ t +1 .
However, if we simply train p φ t +1 to model this empirical distribution, it may not necessarily have higher entropy than p φ t .
The intuition behind our method is quite simple: rather than ﬁtting a generative model to our empirical distribution, we skew the empirical distribution so that rarely visited states are given more weight.
See Figure 2 for a visualization of this process.
How should we skew the empirical distribution if we want to maximize the entropy of p φ t
+1 ?
If we had access to the density of each state, p emp t ( S ) , then we could simply weight each state by 1 /p emp t ( S ) .
We could then perform maximum likelihood estimation (MLE) for the uniform distribution by using the following loss to train φ t +1 :
[Equation]
where we use the fact that the uniform distribution U S ( S ) has constant density for all states in S .
However, computing this density p emp t ( S ) requires marginalizing out the MDP dynamics, which requires an accurate model of both the dynamics and the goal-conditioned policy.
We avoid needing to model the entire MDP process by approximating p emp t ( S ) with our previous learned generative model: p emp t ( S )
≈ p ( S | p φ t )
≈ p φ t ( S ) .
We therefore weight each state by the following weight function
[Equation] [Equation]
) where α is a hyperparameter that controls how heavily we weight each state.
If our approximation p φ t was exact, we could choose α = − 1 and recover the exact importance sampling procedure described above.
If α = 0 , then this skew step has no effect.
By choosing intermediate values of α , we can trade off the reliability of our estimate p φ t ( S ) with the speed at which we want to increase the entropy of the goal distribution.
Variance Reduction As described, this procedure relies on importance sampling (IS), which can have high variance, particularly if p φ t ( S ) ≈ 0 .
We therefore choose a class of generative models where the probabilities are prevented from collapsing to zero, as we will describe in Section 4.
To further reduce the variance, we train p φ t +1 with sampling importance resampling (SIR) (Rubin, 1988).
Rather than sampling from p emp t and weighting the update from each sample by w t,α , SIR explicitly deﬁnes a skewed distribution as
[Equation]
where Z α is the normalizing coefﬁcient and p emp t is given by Equation 2.
We note that computing
Z α adds little computational overhead, since all of the weights already need to be computed.
We then ﬁt the generative model at the next iteration p φ t +1 to p skewed t using standard MLE.
We found that using SIR resulted in signiﬁcantly lower variance than IS.
See Appendix B.2 for this comparision.
Goal Sampling Alternative
Because p φ t +1 ≈ p skewed t , at iteration t + 1 , one can sample goals from either p φ t +1 or p skewed t .
Sampling goals from p skewed t may be preferred if sampling from the learned generative model p φ t +1 is computationally or otherwise challenging.
In either case, one still needs to train the generative model p φ t to create p skewed t .
In our experiments, we found that both methods perform well.
Summary Overall, Skew-Fit samples data from the environment and weights different samples by their density under the generative model p φt .
We prove in the next section conditions under which this weighting makes the generative model at the next iteration p φt +1 have higher entropy.
With higher entropy, the p φ t +1 is more likely to generate goals at the frontier of unseen states, which results in more uniform state coverage.
Skew-Fit is shown in Figure 2 and summarized in Algorithm 1.
1: for Iteration t = 1 , 2 , ... do 2:
Collect N states { S i } Ni =1 by sampling goals from p φ t (or p skewed t ) and running goalconditioned policy.
Construct skewed distribution p skewed t (Equation 3 and Equation 4).
Fit p φ t +1 to skewed distribution p skewed t using MLE.
5: end for
3. S KEW -F IT A NALYSIS
In this section, we provide conditions under which p φ t converges in distribution to the uniform distribution over the state space S .
To make this analysis possible, we consider the case where
N
→ ∞ , which allows us to study the limit behavior of the goal distribution p skewed t .
Our most general result is stated as follows:
Lemma 3.1.
Let S be a compact set.
Deﬁne the set of distributions Q = { p : support of p is S} .
Let
F : Q (cid:55)→ Q be a continuous function and such that H ( F ( p )) ≥ H ( p ) with equality if and only if p is the uniform probability distribution on S , U S .
Deﬁne the sequence of distributions P =
( p 1 , p 2 , . . . )
by starting with any p 1 ∈ Q and recursively deﬁning p t +1 = F ( p t ) .
The sequence P converges to U S .
Proof.
See Appendix Section E.
We will apply Lemma 3.1 to be the map from p skewed t to p skewed t +1 to show that p skewed t converges to U S .
If we assume that the goal-conditioned policy and generative model learning procedure are well behaved ( i.e., the maps from p φ t ( S ) to p emp t and from p skewed t to p φ t
+1 are continuous ), then to apply Lemma 3.1, we only need to show that H ( p skewed t ) ≥ H ( p emp t ) with equality if and only if p emp t = U S .
For the simple case when p φ t = p emp t identically at each iteration, we prove the convergence of Skew-Fit true for any value of α ∈ [ − 1 , 0) in Appendix A.3.
However, in practice, p φ t only approximates p emp t .
To address this more realistic situation, we prove the following result:
Lemma 3.2.
Given two distribution p emp t and p φ t where p emp t (cid:28) p φ t 2 and
[Equation]
deﬁne the distribution p skewed t as in Equation 4.
Let H α ( α ) be the entropy of p skewed t for a ﬁxed α .
Then there exists a constant a < 0 such that for all α ∈ [ a, 0) ,
[Equation]
Proof.
See Appendix Section E.
Thus, our generative model p φ t does not need to exactly ﬁt the empirical distribution.
We merely need for the log densities of p φ t and p emp t to be correlated, which we expect to happen frequently with an accurate goal-conditioned policy, since p emp t is the set of states seen when trying to reach goals from p φ t .
In this case, if we choose negative values of α that are small enough, then the entropy of p skewed t will be higher than that of p emp t .
Empirically, we found that α values as low as α = −performed well.
In summary, we see that under certain assumptions, p skewed t converges to U S .
Since we train each generative model p φ t +1 by ﬁtting it to p skewed t , we expect p φ t to also converge to U S .
T RAINING G OAL -C ONDITIONED P OLICIES WITH S KEW -F IT
Thus far, we have presented and derived Skew-Fit
assuming that we have access to a goal-reaching policy, allowing us to separately analyze how we can maximize H ( G ) .
However, in practice we do not have access to such a policy, and in this section we discuss how we concurrently train a goal-reaching policy.
Maximizing I ( S ; G ) can be done by simultaneously performing Skew-Fit and training a goal conditioned policy to minimize H ( G | S ) , or, equivalently, maximize −H ( G | S ) .
Maximizing −H ( G | S ) requires computing the density log p ( G | S ) , which may be difﬁcult to compute without strong modeling assumptions.
However, for any distribution q , the following lower bound for −H ( G | S ) holds:
[Equation]
where D KL denotes Kullback–Leibler divergence as discussed by Barber & Agakov (2004).
Thus, to minimize H ( G | S ) , we train a policy to maximize the following reward:
[Equation]
For the RL algorithm, we use reinforcement learning with imagined goals (RIG) (Nair et al., 2018), though in principle any goal-conditioned method could be used.
RIG is an efﬁcient off-policy goalconditioned method that solves the vision-based RL problem in a learned latent space.
In particular,
RIG ﬁts a β -VAE and uses it to encode all observations and goals into a latent space, which it uses as the state representation.
RIG also uses the β -VAE to compute rewards, log q ( G | S ) .
Unlike RIG, we use the goal distribution from Skew-Fit to sample goals, both for exploration and for relabeling goals during training (Andrychowicz et al., 2017).
Since RIG already trains a generative model over states, we reuse this β -VAE for the generative model p φ of Skew-Fit.
To make the most use of the data, p φ is trained on all visited state rather than only the terminal states, which we found to work well in practice.
In other words, our method uses the likelihood estimates from the β -VAE to choose the probability of sampling each state in Equation 3.
To prevent these probabilities from collapsing to zero, we model the posterior of the β -VAE as a multivariate Gaussian distribution with a ﬁxed variance and only learn the mean.
We include a detailed summary of RIG and description our how we combine Skew-Fit and RIG in Appendix C.1.
R ELATED W ORK
Many prior methods for training goal-conditioned policies assume that a goal distribution is available to sample from during exploration (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017;
Pong et al., 2018).
Other methods use data collected from a randomly initialized policy or heuristics based on data collected online to design a non-parametric (Colas et al., 2018b; Warde-Farley et al., 2018; Florensa et al., 2018a; Zhao & Tresp, 2019) or parametric (Péré et al., 2018; Nair et al., 2018) goal distribution.
We remark that Warde-Farley et al.
(2018) also motivate their work in terms of minimizing a lower bound for H ( G | S ) .
Our work is complementary to these goal-reaching methods: rather than focusing on how to train goal-reaching policies, we propose a principled method for maximizing the entropy of a goal sampling distribution, H ( G ) .
Our method learns without any task rewards, directly acquiring a policy that can be reused to reach user-speciﬁed goals.
This stands in contrast to exploration methods that give bonus rewards based on state visitation frequency (Bellemare et al., 2016;
Ostrovski et al., 2017; Tang et al., 2017;
Savinov et al., 2018; Chentanez et al., 2005; Lopes et al., 2012; Stadie et al., 2016; Pathak et al., 2017;
Burda et al., 2018; 2019; Mohamed & Rezende, 2015; Tang et al., 2017; Fu et al., 2017).
While these methods can also be used without a task reward, they provide no mechanism for distilling the knowledge gained from visiting diverse states into ﬂexible policies that can be applied to accomplish new goals at test-time: their policies visit novel states, and they quickly forget about them as other states become more novel.
Other prior methods extract reusable skills in the form of latent-variable-conditioned policies, where latent variables can be interpreted as options (Sutton et al., 1999) or abstract skills (Hausman et al., 2018; Gupta et al., 2018b;
Eysenbach et al., 2019; Gupta et al., 2018a;
Florensa et al., 2017).
The resulting skills may be diverse, but they have no grounded interpretation, while our method can be used immediately after unsupervised training to reach diverse user-speciﬁed goals.
Some prior methods propose to choose goals based on heuristics such as learning progress (Baranes & Oudeyer, 2012; Veeriah et al., 2018; Colas et al., 2018a), how off-policy the goal is (Nachum et al., 2018), level of difﬁculty (Florensa et al., 2018b) or likelihood ranking (Zhao & Tresp, 2019).
In contrast, our approach provides a principled framework for optimizing a concrete and well-motivated exploration objective, and can be shown to maximize this objective under regularity assumptions.
The work of Hazan et al.
(2018b) also provably optimizes a well-motivated exploration objective, but is limited to tabular MDPs, while Skew-Fit is able to handle high dimensional settings such as vision-based continuous control.
E XPERIMENTS
Our experiments study the following questions: (1) Does Skew-Fit empirically result in a goal distribution with increasing entropy?
In image-based domains, how does Skew-Fit compare to prior work on choosing goals for goal-conditioned RL?
(3) Can Skew-Fit be applied to a real-world, vision-based robot task?
Does Skew-Fit Maximize Entropy?
To see the effects of Skew-Fit on goal distribution entropy in isolation of learning a goal-reaching policy, we begin by studying an idealized example where the policy is a near-perfect goal-reaching policy.
The MDP is deﬁned on a 2-by-2 unit square-shaped corridor (see Figure 3).
At the beginning of an episode, the agent begins in the bottom-left corner and samples a goal from the goal distribution p φ t .
To simulate the stochasticity of the policy and environment, we add a Gaussian noise with standard deviation of 0 .
05 to this goal.
The policy reaches the state that is closest to this noisy goal and inside the corridor, giving us a state S to add to our empirical distribution.
We compare Skew-Fit to sampling uniformly from the replay buffer (labeled MLE).
The β -VAE hyperparameters used to train p φ t are given in Appendix C.5.
As seen in Figure 3, naively using previous experience to set goals results in a policy that primarily sets goal near the initial state distribution and only relies on the stochasticity of the policy and environment to explore. In contrast, Skew-Fit results in quickly learning a high entropy, near-uniform distribution over the state space.
Vision-Based Continuous Control Tasks
We now evaluate Skew-Fit on a variety of continuous control tasks, where the policy must control a robot arm using only image observations, without access to any ground truth reward signal.
We test our method on three different simulated continuous control tasks released by the authors of RIG (Nair et al., 2018): Visual Door , Visual Pusher , and
Visual Pickup .
To our knowledge, these are the only goal-conditioned, vision-based continuous control environments that are publicly available and used in experimental evaluations in prior work, making them a good point of comparison.
See Figure 4 for visuals and Appendix C for details of these environments.
The policies are trained in a completely unsupervised manner, without access to any prior information about the state-space or any pre-deﬁned goal-sampling distribution.
To evaluate their performance, we sample goal images from a uniform distribution over valid states and report the agent’s ﬁnal distance to the corresponding simulator states (e.g., distance of the object to the target object location), but the agent never has access to this true uniform distribution nor the ground-truth state information during training.
While this evaluation method and metric is only practical in simulation, it provides us with a quantitative measure of a policy’s ability to reach a broad coverage of goals in a vision-based setting.
We use these domains to compare Skew-Fit to a number of existing methods on goal-sampling.
We compare to Warde-Farley et al.
(2018), a vision-based method which uses a non-parametric approach based on clustering to sample goals and an image discriminator to compute rewards.
We denote this method as DISCERN .
The other methods that we compare to were developed in non-vision, statebased environments.
To ensure a fair comparison across methods, we combine these prior methods with a policy trained using RIG.
First, we compare to RIG without Skew-Fit.
We also compared to
RIG using the relabeling scheme described in the hindsight experience replay (labeled HER ).
We compare to curiosity-driven prioritization ( Ranked-Based Priority ) (Zhao & Tresp, 2019), a variant of HER that samples goals for relabeling based on their ranked likelihoods.
Florensa et al.
(2018b)
samples goals from a GAN based on the difﬁculty of reaching the goal.
We compare against this method by replacing p φ with the GAN and label it AutoGoal GAN .
We also separately compare to the goal proposal mechanism proposed by Warde-Farley et al.
and otherwise train the policy with RIG, which we label DISCERN-g .
Lastly, to demonstrate the difﬁculty of the exploration challenge in these domains, we compare to # Exploration (Tang et al., 2017), an exploration method that assigns bonus rewards based on the novelty of new states.
Implementation details of the prior methods is given in Appendix C.3.
We see in Figure 5 that Skew-Fit signiﬁcantly outperforms prior methods both in terms of task performance and sample complexity.
The most common failure mode for prior methods is that the goal distributions collapse, resulting in the agent learning to reach only a fraction of the state space, as shown in Figure 1.
For comparison, additional samples of p φ when trained with and without
Skew-Fit are shown in Appendix B.3.
Those images show that without Skew − Fit , p φ produces a small, non-diverse distribution for each environment: the object is in the same place for pickup, the puck is often in the starting position for pushing, and the door is always closed.
In contrast, Skew-Fit proposes goals where the object is in the air and on the ground, where the puck positions are varied, and the door angle changes.
The direct effect of these goal choices can be seen by visualizing more example rollouts for RIG and Skew-Fit .
Due to space constraints, these visuals are in Figure 13 in Appendix B.3.
The ﬁgure shows that standard RIG only learns to reach states close to the initial position, while Skew-Fit learns to reach the entire state space.
A quantitative comparison of the various methods on the pickup task can be seen in Figure 6, which gives the cumulative total exploration pickups for each method.
From the graph, we can see that only Skew-Fit learns to pay attention to the object and therefore consistently increases the rate at which the policy picks up the object during exploration.
In contrast, the other methods have near constant slopes past 40 k steps, meaning that they do not continue to learning, and many methods have a near-constant rate of object lifts throughout all of training.
[Figure]
Visual Object Pickup
[Figure]
pick it up at the same rate as the initial policy. In contrast, after seeing the object picked up a few times, Skew-Fit practices picking up the object more often by sampling the appriopriate exploration goals. [Figure] u l a ti v e S u cce ss e s
[Figure] Real-World Vision-Based Robotic Manipulation We also demonstrate that Skew-Fit scales well to the real world with a door opening task, Real World Visual Door .
See
Figure 4 for a picture of this environment.
While a number of prior works have studied RL-based learning of door opening Kalakrishnan et al.
(2011); Chebotar et al.
, we demonstrate the ﬁrst method for autonomous learning of door opening without a user-provided, task-speciﬁc reward function.
As in simulation, we do not provide any goals to the agent and simply let it interact with the door to solve the door opening task from scratch, without any human guidance or reward signal.
We train two agents using Skew-Fit with RIG and RIG alone.
Unlike in simulation, we cannot measure the difference between the policy’s achieved and desired door angle since we do not have access to the true state of the world.
Instead, we simply visually denote a binary success/failure for each goal based on whether the last state in the trajectory achieves the target angle.
Every seven and a half minutes of interaction time we evaluate on 5 goals and plot the cumulative successes for each method.
As Figure 7 shows, standard RIG only starts to open the door after ﬁve hours of training.
In contrast, Skew-Fit learns to occasionally open the door after three hours of training and achieves a near-perfect success rate after ﬁve and a half hours of interaction time, demonstrating that Skew-Fit is a promising technique for solving real world tasks without any human-provided reward function.
Videos of Skew-Fit solving this task and the simulated tasks can be viewed on our website.
Additional Experiments To study the sensitivty of our method to the hyperparameter α , we sweep α across the values [ − 1 , − 0 .
75 , − 0 .
5 , − 0 .
25 , 0] on the simulated image-based tasks.
Due to space constraints, the sensitivity analysis over the hyperparameter α is in Appendix B, and the results demonstrate that Skew-Fit works across a large range of values for α , and α = − 1 consistently outperform α = 0 , where the empirical distribution is not skewed.
Additionally, Appendix C provides a complete description our method hyper-parameters, including network architecture and
RL algorithm hyperparameters.
C ONCLUSION
We presented a formal objective for self-supervised goal-directed exploration, allowing researchers to quantify progress and compare progress when designing algorithms that enable agents to autonomously learn.
We also presented Skew-Fit, an algorithm for training a generative model to approximate a uniform distribution over valid states, using data obtained via goal-conditioned reinforcement learning, and our theoretical analysis gives conditions under which Skew-Fit converges to the uniform distribution.
When such a model is used to choose goals for exploration and to relabeling goals for training, the resulting method results in much better coverage of the state space, enabling our method to explore effectively.
Our experiments show that when we concurrently train a goal-reaching policy using self-generated goals, Skew-Fit produces quantiﬁable improvements on simulated robotic manipulation tasks, and can be used to learn a door opening skill to reach a 95% success rate directly on a real-world robot, without any human-provided reward supervision.
A P ROOFS A. P ROOF OF L EMMA 3.
Lemma A.1.
Let S be a compact set.
Deﬁne the set of distributions Q = { p : support of p is S} .
Let F : Q (cid:55)→ Q be a continuous function and such that H ( F ( p )) ≥ H ( p ) with equality if and only if p is the uniform probability distribution on S , U S .
Deﬁne the sequence of distributions
P = ( p 1 , p 2 , . . . )
by starting with any p 1 ∈ Q and recursively deﬁning p t +1 = F ( p t ) .
The sequence P converges to U S .
Proof.
The uniform distribution U S is well deﬁned since S is compact.
Because S is a compact set, by Prokhorov’s Theorem Billingsley (2013), the set Q is sequentially compact.
Thus, P has a convergent subsequence P (cid:48) =
( p k 1 , p k 2 , . . . )
⊂ P for k 1 < k 2 < . . .
that converges to a distribution p ∗ ∈ Q .
Because F is continuous, p ∗ must be a ﬁxed point of F since by the convergence mapping theorem, we have that
[Equation]
and so
[Equation]
The only ﬁxed point of F is U S since for any distribution p that is not the uniform distribution,
U S , we have that H ( F ( p )) >
H ( p ) which implies that F ( p ) (cid:54) = p .
Thus, P (cid:48) converges to the only ﬁxed point, U S .
Since the entropy cannot decrease, then entropy of the distributions in P must also converge to the entropy of U S .
Lastly, since entropy is a continuous function of distribution, P must converge to U S .
A. P ROOF OF L EMMA 3.
Lemma A.2.
Given two distribution p ( x ) and q ( x ) where p (cid:28) q and
[Equation]
deﬁne the distribution p α as
[Equation]
where α ∈ R and Z α is the normalizing factor.
Let H α ( α ) be the entropy of p α .
Then there exists a constant a > 0 such that for all α ∈ [ − a, 0) ,
[Equation]
Proof.
Observe that { p α : α ∈ [ − 1 , 0] } is a one-dimensional exponential family
[Equation]
with log carrier density k ( x ) = log p ( x ) , natural parameter α , sufﬁcient statistic T ( x ) = log q ( x ) , and log-normalizer A ( α )
= (cid:82) X e αT ( x )+ k ( x ) dx .
As shown in Nielsen & Nock (2010), the entropy of a distribution from a one-dimensional exponential family with parameter α is given by:
[Equation]
The derivative with respect to α is then
[Equation]
where we use the fact that the n th derivative of A ( α ) give the n central moment, i.e. A (cid:48) ( α )
=
E p α
[ T ( x )] and A (cid:48)(cid:48) ( α ) =
Var p
α [ T ( x )] .
The derivative of α = 0 is
[Equation]
which is negative by assumption.
Because the derivative at α = 0 is negative, then there exists a constant a > 0 such that for all α ∈ [ − a, 0] , H α ( α ) >
H α (0) =
H ( p ) .
A. S IMPLE C ASE P ROOF
We prove the convergence directly for the (even more) simpliﬁed case when p θ = p ( S | p φ t ) using a similar technique:
Lemma A.3.
Assume the set S has ﬁnite volume so that its uniform distribution U S is well deﬁned and has ﬁnite entropy.
Given any distribution p ( s ) whose support is S , recursively deﬁne p t with p 1 = p and
[Equation]
where Z tα is the normalizing constant and α ∈ [0 , 1) .
The sequence ( p 1 , p 2 , . . . ) converges to U S , the uniform distribution S .
Proof.
If α = 0 , then p 2 (and all subsequent distributions) will clearly be the uniform distribution.
We now study the case where α ∈ (0 , 1) .
At each iteration t , deﬁne the one-dimensional exponential family { p tθ : θ ∈ [0 , 1] } where p tθ is
[Equation]
with log carrier density k ( s ) = 0 , natural parameter θ , sufﬁcient statistic T ( s ) = log p t ( s ) , and lognormalizer A ( θ )
= (cid:82) S e θT ( s ) d s .
As shown in Nielsen & Nock (2010), the entropy of a distribution from a one-dimensional exponential family with parameter θ is given by:
[Equation]
The derivative with respect to θ is then
[Equation]
where we use the fact that the n th derivative of A ( θ ) is the n central moment, i.e. A (cid:48)(cid:48)
( θ ) =
Var s ∼ p tθ [ T ( s )] .
Since variance is always non-negative, this means the entropy is monotonically decreasing with θ .
Note that p t +1 is a member of this exponential family, with parameter θ = α ∈ (0 , 1) .
So
[Equation]
which implies
[Equation]
This monotonically increasing sequence is upper bounded by the entropy of the uniform distribution, and so this sequence must converge.
The sequence can only converge if ddθ H tθ ( θ ) converges to zero.
However, because α is bounded away from 0 , Equation 8 states that this can only happen if
[Equation]
Because p t has full support, then so does p tθ .
Thus, Equation 9 is only true if log p t ( s ) converges to a constant, i.e. p t converges to the uniform distribution.
B A DDITIONAL R EINFORCEMENT L EARNING E XPERIMENTS B. S KEW -F IT WITH L EARNED P OLICIES
Sensitivity Analysis We study the sensitvity of the α hyperparameter by testing values of α ∈ [ − 1 , − 0 .
75 , − 0 .
5 , − 0 .
25 , 0] on the Visual Door and Visual Pusher task.
The results are included in Figure 8 and shows that our method is robust to different parameters of α , particularly for the more challenging Visual Pusher task.
Also, the method consistently outperform α = 0 , which is equivalent to sampling uniformly from the replay buffer.
2D navigation We reproduce the 2D navigation environment experiment from Section 6, and replace the oracle goal-reacher with a goal-reaching policy that is simultaneously trained with the goal setter.
The policy outputs velocities with maximum speed of one.
Evaluation goals are chosen uniformly over the valid states.
In Figure 9a, we can see that a policy trained with a goal distribution trained by Skew-Fit consistently learns to reach all goals, whereas a goal distribution trained with uniform sampling (labeled MLE) results in a policy that fails to reach states far from the starting position.
[Figure]
[Table]
[Figure]
V ARIANCE A BLATION
We measure the gradient variance of training a VAE on an unbalanced Visual Door image dataset with
Skew-Fit vs Skew-Fit with importance sampling (IS) vs no Skew-Fit (labeled MLE).
We construct the imbalanced dataset by rolling out a random policy in the environment and collecting the visual observations.
Most of the images contained the door in a closed position; in a few, the door was opened.
In Figure 10, we see that the gradient variance for Skew-Fit with IS is catastrophically large for large values of α .
In contrast, for Skew-Fit with SIR, which is what we use in practice, the variance is relatively similar to that of MLE.
Additionally we trained three VAE’s, one with MLE on a uniform dataset of valid door opening images, one with Skew-Fit on the unbalanced dataset from above, and one with MLE on the same unbalanced dataset.
As expected, the VAE that has access to the uniform dataset gets the lowest negative log likelihood score.
This is the oracle method, since in practice we would only have access to imbalanced data.
As shown in Table 1, Skew-Fit considerably outperforms MLE, getting a much closer to oracle log likelihood score.
B. G OAL AND P ERFORMANCE V ISUALIZATION
We visualize the goals sampled from Skew-Fit as well as those sampled when using the prior method,
RIG (Nair et al., 2018).
As shown in Figure 11 and Figure 12, the generative model p φ results in much more diverse samples when trained with Skew-Fit.
We we see in Figure 13, this results in a policy that more consistently reaches the goal image.
C I MPLEMENTATION D ETAILS C. RIG WITH S KEW -F IT S UMMARY
Algorithm 2 provides detailed pseudo-code for how we combined our method with RIG.
Steps that were removed from the base RIG algorithm are highlighted in blue and steps that were added are highlighted in red.
The main differences between the two are (1) sampling exploration goals from the buffer using p skewed instead of the VAE prior, (2) relabeling with replay buffer goals sampled using p skewed instead of from the VAE prior, and (3) training the VAE on replay buffer data data sampled using p skewed instead of uniformly.
C. L IKELIHOOD E STIMATION USING β -VAE
We estimate the density under the VAE by using a sample-wise approximation to the marginal over x estimated using importance sampling:
[Equation]
where q θ is the encoder, p ψ is the decoder, and p ( z ) is the prior, which in this case is unit Gaussian.
We found that sampling N = 10 latents for estimating the density worked well in practice.
C. I MPLEMENTATION OF P RIOR W ORK
We replaced TD3 (Fujimoto et al., 2018) with soft actor critic (SAC) from Haarnoja et al.
(2018) for all the methods that use RIG, including Skew-Fit
This is in contrast to the original RIG Nair et al.
(2018) paper which used TD3 Fujimoto et al.
We found that maximum entropy policies in general improved the performance of RIG, and that we did not need to add noise on top of the stochastic policy’s noise.
For our RL network architectures and training scheme, we use fully connected networks for the policy, Q-function and value networks with two hidden layers of size 400 and 300 each.
We also delay training any of these networks for 10000 time steps in order to collect sufﬁcient data for the replay buffer as well as to ensure the latent space of the VAE is relatively stable (since we train the VAE online in this setting).
As in RIG, we train a goal-conditioned value functions Schaul et al.
(2015) using hindsight experience replay Andrychowicz et al.
(2017), relabelling 50% of exploration goals as goals sampled from the VAE prior N (0 , 1) and 30% from future goals in the trajectory.
In the prior RIG method, the VAE was pre-trained on a uniform sampling of images from the state space of each environment.
In order to ensure a fair comparison to
Skew-Fit, we forego pre-training and instead train the VAE alongside RL, using the variant described in the RIG paper.
C. V ISION -B ASED C ONTINUOUS C ONTROL E XPERIMENTS
In our experiments, we use an image size of 48x48.
For our VAE architecture, we use a modiﬁed version of the architecture used in the original RIG paper Nair et al.
Our VAE has three convolutional layers with kernel sizes: 5x5, 3x3, and 3x3, number of output ﬁlters: 16, 32, andand strides: 3, 2, and 2.
We then have a fully connected layer with the latent dimension number of units, and then reverse the architecture with de-convolution layers.
We vary the latent dimension of the VAE, the β term of the VAE and the α term for Skew-Fit based on the environment.
Additionally, we vary the training schedule of the VAE based on the environment.
See the table at the end of the appendix for more details.
Our VAE has a Gaussian decoder with identity variance, meaning that we train the decoder with a mean-squared error loss.
When training the VAE alongside RL, we found the following two schedules to be effective for different environments:
For ﬁrst 5 K steps: Train VAE using standard MLE training every 500 time steps forbatches.
After that, train VAE using Skew-Fit every 500 time steps for 200 batches.
For ﬁrst 5 K steps: Train VAE using standard MLE training every 500 time steps forbatches.
For the next 45 K steps, train VAE using Skew-Fit every 500 steps for 200 batches.
After that, train VAE using Skew-Fit every 1000 time steps for 200 batches.
We found that initially training the VAE without Skew-Fit improved the stability of the algorithm.
This is due to the fact that density estimates under the VAE are constantly changing and inaccurate during the early phases of training.
Therefore, it made little sense to use those estimates to prioritize goals early on in training.
Instead, we simply train using MLE training for the ﬁrst 5 K timesteps, and after that we perform Skew-Fit according to the VAE schedules above.
Table 3 lists the hyperparameters that were shared across the continuous control experiments.
Table 4 lists hyper-parameters speciﬁc to each environment.
Additionally, Appendix C.1 shows the combined RIG + Skew-Fit algorithm.
[Table] [Table]
[Table] [Table]
Sample latent goal use
Sample state steps do 9:
Get a
Get next
Store ( s a s into
Sample transition (
Encode
=
14: (Probability 0 . 5 ) replace with z (cid:48)
∼ p (
15: (Probability replace g with e ( s (cid:48) where ( s a, s (cid:48)
∼ using p
Compute new reward r = −|| z (cid:48) − z g
|| .
Minimize
Bellman
Error using ( z, a, z (cid:48) , z g , r ) .
18: end for 19: for t = 0 , ..., H − 1 steps do 20: for i = 0 , ..., k − 1 steps do 21:
Sample future state s h
i , t < h i ≤ H − 1 . 22:
Store ( s t , a t , s t +1 , e ( s h i )) into R . 23: end for 24: end for 25:
Construct skewed replay buffer distribution p φ using data from R with Equation26: if total _ steps < 5000 then 27:
Fine-tune β -VAE on data uniformly sampled from R according to VAE Training Schedule.
28: else 29:
Fine-tune β -VAE on data uniformly sampled from R according to VAE Training Schedule.
Fine-tune β -VAE on data sampled from R using p φ according to VAE Training Schedule.
31: end if 32: end for
C. O RACLE 2D N AVIGATION E XPERIMENTS
We initialize the VAE to only output points in the bottom left corner of the environment.
Both the encoder and decoder have ReLU hidden activations, 2 hidden layers with 32 units, and no output activations.
The VAE has a latent dimension of 16 and a Gaussian decoder trained with mean-squared error loss, batch size of 500 , and 100 epochs per iteration.
For Skew-Fit hyperparameters, α = − 0 .and
N = 10000 .
D E NVIRONMENT D ETAILS
Point-Mass : In this environment, an agent must learn to navigate a square-shaped corridor (see
Figure 3).
The observation is the 2D position, and the agent must specify a velocity as the 2D action.
The reward at each time step is the negative distance between the achieved position and desired position.
Visual Pusher : A MuJoCo environment with a 7-DoF Sawyer arm and a small puck on a table that the arm must push to a target position.
The agent controls the arm by commanding x, y position for the end effector (EE).
The underlying state is the EE position, e and puck position p .
The evaluation metric is the distance between the goal and ﬁnal puck positions.
The hand goal/state space is a 10x cm 2 box and the puck goal/state space is a 30x20 cm 2 box.
Both the hand and puck spaces are centered around the origin.
The action space ranges in the interval [ − 1 , 1] in the x and y dimensions.
Visual Door : A MuJoCo environment with a 7-DoF Sawyer arm and a door on a table that the arm must pull open to a target angle.
Control is the same as in Visual Pusher .
The evaluation metric is the distance between the goal and ﬁnal door angle, measured in radians.
In this environment, we do not reset the position of the hand or door at the end of each trajectory.
The state/goal space is a 5x20x cm 3 box in the x, y, z dimension respectively for the arm and an angle between [0 , .
83] radians.
The action space ranges in the interval [ − 1 , 1] in the x, y and z dimensions.
Visual Pickup : A MuJoCo environment with the same robot as Visual Pusher , but now with a different object.
The object is cube-shaped, but a larger intangible sphere is overlaid on top so that it is easier for the agent to see.
Moreover, the robot is constrained to move in 2 dimension: it only controls the y, z arm positions.
The x position of both the arm and the object is ﬁxed.
The evaluation metric is the distance between the goal and ﬁnal object position.
For the purpose of evaluation, 75% of the goals have the object in the air and 25% have the object on the ground.
The state/goal space for both the object and the arm is 10cm in the y dimension and 13cm in the z dimension.
The action space ranges in the interval [ − 1 , 1] in the y and z dimensions.
Real World Visual Door : A Rethink Sawyer Robot with a door on a table.
The arm must pull the door open to a target angle.
The agent controls the arm by commanding the x, y, z velocity of the EE.
Our controller commands actions at a rate of up to 10Hz with the scale of actions ranging up to 1cm in magnitude.
The underlying state and goal is the same as in Visual Door .
Again we do not reset the position of the hand or door at the end of each trajectory.
We obtain images using a Kinect Sensor.
The state/goal space for the environment is a 10x10x10 cm 3 box.
The action space ranges in the interval [ − 1 , 1] (in cm) in the x, y and z dimensions.
The door angle lies in the range [0 , 45] degrees.
E G OAL -C ONDITIONED R EINFORCEMENT L EARNING M INIMIZES H ( G | S )
Some goal-conditioned RL methods such as Warde-Farley et al.
; Nair et al.
(2018) present methods for minimizing a lower bound for H ( G | S ) , by approximating log p ( G | S ) and using it as the reward.
Other goal-conditioned RL methods (Kaelbling, 1993; Lillicrap et al., 2016; Schaul et al., 2015; Andrychowicz et al., 2017; Pong et al., 2018; Florensa et al., 2018a) are not developed with the intention of minimizing the conditional entropy H ( G | S ) . Nevertheless, one can see that goal-conditioned RL generally minimizes H ( G | S ) by noting that the optimal goal-conditioned policy will deterministically reach the goal. The corresponding conditional entropy of the goal given the state, H ( G | S ) , would be zero, since given the current state, there would be no uncertainty over the goal (the goal must have been the current state since the policy is optimal). So, the objective of goal-conditioned RL can be interpreted as ﬁnding a policy such that H ( G | S )
= 0 .
Since zero is the minimum value of H ( G | S ) , then goal-conditioned RL can be interpreted as minimizing H ( G | S ) .