H IERARCHICAL F ORESIGHT : S ELF -S UPERVISED L EARNING OF L ONG -H ORIZON T ASKS VIA V ISUAL S UBGOAL G ENERATION
A BSTRACT
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only selfsupervision, reaching novel goals in cluttered scenes with unseen objects.
However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one signiﬁcant limitation of these approaches is the ability to plan over long horizons to reach distant goals.
To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which generates subgoal images conditioned on a goal image, and uses them for planning.
The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identiﬁes semantically meaningful states as subgoals.
Across four simulated vision-based manipulation tasks, we ﬁnd that our method achieves more than 20% absolute performance improvement over planning without subgoals and model-free RL approaches.
Further, our experiments illustrate that our approach extends to real, cluttered visual scenes.
I NTRODUCTION
Developing robotic systems that can complete long horizon visual control tasks, while generalizing to novel scenes and objectives, remains an unsolved and challenging problem.
Generalization to unseen objects and scenes requires robots to be trained across diverse environments, meaning that detailed supervision during data collection in not practical to provide.
Furthermore, reasoning over long-horizon tasks introduces two additional major challenges.
First, the robot must handle large amounts of uncertainty as the horizon increases.
And second, the robot must identify how to reach distant goals when only provided with the ﬁnal goal state, a sparse indication of the task, as opposed to a shaped cost that implicitly encodes how to get there.
In this work, we aim to develop a method that can start to address these challenges, leveraging self-supervised models learned using only unlabeled data, to solve novel temporally-extended tasks.
Model-based reinforcement learning has shown promise in generalizing to novel objects and tasks, as learned dynamics models have been shown to generalize to new objects (Finn & Levine, 2016;
Ebert et al., 2018b), and can be used in conjunction with planning to reach goals unseen during training.
However, planning to reach temporally distant goals is difﬁcult.
As the planning horizon increases model error compounds, and the cost function often provides only a noisy or sparse signal of the objective.
Both of these challenges are exacerbated when planning in visual space.
In this work, the key insight that we leverage is that while model error and sparse cost signals can make long horizon planning difﬁcult, we can mitigate these issues by learning to break down longhorizon tasks into short horizon segments .
Consider, for example, the task of opening a drawer and putting a book in it, given supervision only in the form of the ﬁnal image of the open drawer containing the book.
The goal image provides nearly no useful cost signal until the last stage of the task, and model predictions are likely to become inaccurate beyond the ﬁrst stage of the task.
However, if we can generate a sequence of good subgoals, such as (1) the robot arm grasping the drawer handle, (2) the open drawer, and (3) the robot arm reaching for the book, planning from the initial state to (1), from (1) to (2), from (2) to (3), and from (3) to the ﬁnal goal, the problem becomes substantially easier.
The subgoals break the problem into short horizon subsegments each with some useful cost signal coming from the next subgoal image.
Our main contribution is a self-supervised hierarchical planning framework, hierarchical visual foresight (HVF), which combines generative models of images and model predictive control to decompose a long-horizon visual task into a sequence of subgoals.
In particular, we propose optimizing over subgoals such that the resulting task subsegments have low expected planning cost.
However, in the case of visual planning, optimizing over subgoals corresponds to optimizing within the space of natural images .
To address this challenge, we train a generative latent variable model over images from the robot’s environment and optimize over subgoals in the latent space of this model.
This allows us to optimize over the manifold of images with only a small number of optimization variables.
When combined with visual model predictive control, we observe that this subgoal optimization naturally identiﬁes semantically meaningful states in a long horizon tasks as subgoals, and that when using these subgoals during planning, we achieve signiﬁcantly higher success rates on long horizon, multi-stage visual tasks.
Furthermore, since our method outputs subgoals conditioned on a goal image, we can use the same model and approach to plan to solve many different long-horizon tasks, even with previously unseen objects.
We ﬁrst demonstrate our approach in simulation on a continuous control navigation task with tight bottlenecks, and then evaluate on a set of four different multi-stage object manipulation tasks in a simulated desk environment, which require interacting with up to 3 different objects.
In the challenging desk environment, we ﬁnd that our method yields at least a 20% absolute performance improvement over prior approaches, including model-free reinforcement learning and a state of the art subgoal identiﬁcation method.
Finally, we show that our approach generates realistic subgoals on real robot manipulation data.
R ELATED W ORK
Developing robots that can execute complex behaviours from only pixel inputs has been a well studied problem, for example with visual servoing (Mohta et al., 2014;
Espiau et al., 1992; Wilson et al., 1996; Yoshimi & Allen, 1994; Jagersand et al., 1997; Lampe & Riedmiller, 2013; Sadeghi et al., 2018; Sadeghi, 2019).
Recently, reinforcement learning has shown promise in completing complex tasks from pixels (Ghadirzadeh et al., 2017; Levine et al., 2015;
Kalashnikov et al., 2018;
Lange et al., 2012; OpenAI et al., 2018;
Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including in goal-conditioned settings (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Sadeghi et al., 2018; Sadeghi, 2019; Nair et al., 2018a).
While model-free RL approaches have illustrated the ability to generalize to new objects (Kalashnikov et al., 2018) and learn tasks such as grasping and pushing through self-supervision (Pinto & Gupta, 2015; Zeng et al., 2018), pure model-free approaches generally lack the ability to explicitly reason over temporally-extended plans, making them ill-suited for the problem of learning long-horizon tasks with limited supervision.
Video prediction and planning have also shown promise in enabling robots to complete a diverse set of visuomotor tasks while generalizing to novel objects (Finn & Levine, 2016; Kalchbrenner et al., 2016; Boots et al., 2014; Byravan & Fox, 2016).
Since then, a number of video prediction frameworks have been developed speciﬁcally for robotics (Babaeizadeh et al., 2017; Lee et al., 2018;
Ebert et al., 2017), which combined with planning have been used to complete diverse behaviors (Nair et al., 2018b; Ebert et al., 2018b; Paxton et al., 2018;
Xie et al., 2019).
However, these approaches still struggle with long horizon tasks, which we speciﬁcally focus on.
One approach to handle long horizon tasks is to add compositional structure to policies, either from demonstrations (Krishnan et al., 2017; Fox et al., 2018), with manually-speciﬁed primitives (Xu et al., 2017; Huang et al., 2018), learned temporal abstractions (Neitz et al., 2018), or through model-free reinforcement learning (Sutton et al., 1999; Barto & Mahadevan, 2003;
Bacon et al., 2016; Nachum et al., 2018; Levy et al., 2019).
These works have studied such hierarchy in grid worlds (Bacon et al., 2016) and simulated control tasks (Nachum et al., 2018;
Eysenbach et al., 2018; Levy et al., 2019) with known reward functions.
In contrast, we study how to incorporate compositional structure in learned model-based planning with video prediction models.
Our approach is entirely self-supervised , without motion primitives, demonstrations, or shaped rewards, and scales to vision-based manipulation tasks.
Classical planning methods have been successful in solving long-horizon tasks (LaValle, 2006;
Choset et al., 2005), but make restrictive assumptions about the state space and reachability between states, limiting their applicability to complex visual manipulation tasks.
Similarly, completing long horizon tasks has also been explored with symbolic models (Toussaint et al., 2019) and Task and
Motion Planning (TAMP) (Kaelbling & Lozano-Perez, 2011; Srivastava et al., 2014).
However, unlike these approaches our method requires no additional knowledge about the objects in the scene nor any predeﬁned symbolic states.
Recently, there have been several works that have explored planning in learned latent spaces (Kurutach et al., 2018; Ichter & Pavone, 2018; Watter et al., 2015; Srinivas et al., 2018).
This has enabled planning in higher dimensional spaces, however these methods still struggle with long-horizon tasks.
Furthermore, our hierarchical planning framework is agnostic to state space, and could directly operate in one of the above latent spaces.
A number of recent works have explored reaching novel goals using only self-supervision (Finn & Levine, 2016;
Eysenbach et al., 2019; Kurutach et al., 2018; Wang et al., 2019; Jayaraman et al., 2019; Nair et al., 2018a).
In particular, time-agnostic prediction (TAP) (Jayaraman et al., 2019) aims to identify bottlenecks in long-horizon visual tasks, while other prior works (Nair et al., 2018a; Finn & Levine, 2016) reach novel goals using model-free or model-based RL.
We compare to all three of these methods in Section 5 and ﬁnd that HVF signiﬁcantly outperforms all of them.
P RELIMINARIES
We formalize our problem setting as a goal-conditioned Markov decision process (MDP) deﬁned by the tuple ( S , A , p, G , C, λ ) where s ∈ S is the state space, which in our case corresponds to images, a ∈ A is the action space, p ( s t +1 | s t , a t ) governs the environment dynamics, G ⊂ S represents the set of goals which is a subset of possible states, C ( s t , s g ) represents the cost of being in state s t ∈ S given that the goal is s g ∈ G , and λ is the discount factor.
In practice, acquiring cost functions that accurately reﬂect the distance between two images is a challenging problem (Yu et al., 2019).
We make no assumptions about having a shaped cost, assuming the simple yet sparse distance metric of (cid:96) 2 distance in pixel space in all of our experiments.
Approaches that aim to recover more shaped visual cost functions are complementary to the contributions of this work.
In visual foresight, or equivalently, visual MPC (Finn & Levine, 2016; Ebert et al., 2018b), the robot collects a data set of random interactions [( s 1 , a 1 ) , ( s 2 , a 2 ) , ..., ( s T , a T )] from a pre-determined policy.
This dataset is used to learn a model of dynamics f
θ ( s t +1 , s t +2 , ..., s t
+ h | s t , a t , a t +1 , ..., a t + h − 1 ) through maximum likelihood supervised learning.
Note the states are camera images, and thus f θ is an action-conditioned video prediction model.
Once the model is trained, the robot is given some objective and plans a sequence of actions that optimize the objective via the cross entropy method (CEM) (Rubinstein & Kroese, 2004).
In this work, we will assume the objective is speciﬁed in the form of an image of the goal s g , while CEM aims to ﬁnd a sequence of actions that minimize the cost C between the predicted future frames from f θ and the goal image.
While standard visual foresight struggles with long-horizon tasks due to uncertainty in f θ as the prediction horizon h increases and sparse C for CEM to optimize, in the next section we describe how our proposed approach uses subgoal generation to mitigate these issues.
H IERARCHICAL V ISUAL F ORESIGHT
Overview: We propose hierarchical visual foresight (HVF), a planning framework built on top of visual foresight (Finn & Levine, 2016; Ebert et al., 2018b) for long horizon visuomotor tasks.
We observe that when planning to reach a long horizon goal given only a goal image, standard planning frameworks struggle with (1) sparse or noisy cost signals and (2) compounding model error.
Critically, we observe that if given the ability to sample possible states, there is potential to decompose a long horizon planning task into shorter horizon tasks.
While this idea has been explored in classical planning (Betts, 2010), state generation is a signiﬁcant challenge when dealing with high dimensional image observations.
One of our key insights is that we can train a deep generative model, trained exclusively on selfsupervised data, as a means to sample possible states.
Once we can sample states, we also need to evaluate how easy it is to get from one sampled state to another, to determine if a state makes for a good subgoal.
We can do so through planning: running visual MPC to get from one state to another and measuring the predicted cost of the planned action sequence.
Thus by leveraging the low-dimensional space of a generative model and the cost acquired by visual MPC, we can optimize over a sequence of subgoals that lead to the goal image.
In particular, we can explicitly search in
[Figure]
Hierarchical Visual Foresight
[Figure]
predictionmodel
[Figure]
generativemodel
[Figure]
subgoals
[Figure]
latent image space for subgoals, such that no segment is too long-horizon, mitigating the issues around sparse costs and compounding model error.
In the following sections, we will describe more formally how HVF works, how we learn the generative model, how we learn the dynamics model, how goal-conditioned planning with visual MPC is executed, and lastly how subgoals are optimized.
Hierarchical visual foresight: Formally, we assume the goal conditioned MDP setting in Section 3 where the agent has a current state s 0 , goal state s g , cost function C , and dataset of environment interaction { ( s 1 , a 1 , s 2 , a 2 , ..., s T , a T ) } .
This data can come from any exploration policy; in practice, we ﬁnd that interaction from a uniformly random policy in the continuous action space of the agent works well.
From this data, we train both a dynamics model f
θ using maximum likelihood supervised learning, as well as a generative model s ∼ g φ .
Now given s 0 and s g , the objective is to ﬁnd K subgoals s 1 , s 2 , ..., s K that enable easier completion of the task.
Our hope is that the subgoals will identify steps in the task such that, for each subsegment, the planning problem is easier and the horizon is short.
While one way to do this might be to ﬁnd subgoals that minimize the total planning cost, we observe that this does not necessarily encourage splitting the task into shorter segments.
Consider planning in a straight line: using any point on that line as a subgoal would equally minimize the total cost.
Therefore, we instead optimize for subgoals that minimize the worst expected planning cost across any segment.
This corresponds to the following objective:
[Equation]
where C plan ( s i , s j ) is the cost achieved by the planner when planning from s i to s j , which we compute by planning a sequence of actions to reach s j from s i using f θ and measuring the predicted cost achieved by that action sequence 1 .
Once the subgoals are found, then the agent simply plans using visual MPC (Finn & Levine, 2016; Ebert et al., 2018b) from each s k − 1 to s k until a cost threshold is reached or for a ﬁxed, maximum number of timesteps, then from s ∗ k to s k +1 , until planning to the goal state, where s ∗ k is the actual state reached when running MPC to get to s k .
For a full summary of the algorithm, see Algorithm 1.
Next, we describe individual components in detail.
Generative model:
To optimize subgoals, directly optimizing in image space is nearly impossible due to the high dimensionality of images and the thin manifold on which they lie.
Thus, we learn a
# Sample latent subgoal lists
# Map latent to image subgoal 9:
# Optimal actions and planning cost 10: θ m
C
} # Max planning cost across segments 13:
# Rank by Cost m 15:
Set generative model s ∼ g φ ( z ) such that samples represent possible futures in the scene, and optimize in the low-dimensional latent space z . In settings where aspects of the scene or objects change across episodes, we only want to sample plausible future states, rather than sampling states corresponding to all scenes.
To handle this setting, we condition the generative model on the current image, which contains information about the current scene and the objects within view.
Hence, in our experiments, we use either a variational autoencoder (VAE) or a conditional variational autoencoder (CVAE), with a diagonal Gaussian prior over z , i.e. z ∼ N ( 0 , I ) .
In the case of the CVAE, the decoder also takes as input an encoding of the conditioned image, i.e. s ∼ g φ ( z, s 0 ) .
In practice, we use the inital state s 0 as the conditioned image.
We train the generative model on randomly collected interaction data.
The architecture and training details can be found in Appendix A.
Dynamics model: The forward dynamics model f θ can be any forward dynamics model that estimates p ( s t +1 , s t +2 , ..., s t
+ h | s t , a t , a t +1 , ..., a t + h − 1 ) .
In our case states are images, so we use an action conditioned video prediction model, stochastic variational video prediction (SV2P) (Babaeizadeh et al., 2017) for f θ .
We train the dynamics model on randomly collected interaction data.
Architecture and training parameters are identical to those used in (Babaeizadeh et al., 2017); details can be found in Appendix A.4.
MPC & planning with subgoals: When optimizing subgoals, we need some way to measure the ease of planning to and from the subgoals.
We explicitly compute this expected planning cost between two states s k , s k +1 by running model predictive control A, C = MPC ( f θ , s k , s k +1 ) 2 as done in previous work (Finn & Levine, 2016; Nair et al., 2018b; Ebert et al., 2018b).
This uses the model f θ to compute the optimal trajectory between s k and s k +1 , and returns the optimal action
A and associated cost C .
Note this does not step any actions in the real environment, but simply produces an estimated planning cost.
Details on this procedure can be found in Appendices A.1 and
A.2.
Once the subgoals have been computed, the same procedure is run MPC ( f θ , s k − 1 , s k ) until s k is reached (measured by a cost threshold) or for a ﬁxed, maximum number of timesteps, then from
MPC ( f θ , s ∗ k , s k +1 ) , ..., MPC ( f θ , s ∗ K , s g )
(Alg. 1 Lines 17:21), where s ∗ k represents the state actually reached after trying to plan to s k .
In this case, the best action at each step is actually applied in the environment until the task is completed, or the horizon runs out.
Note we only compute subgoals once in the beginning calling HVF ( f θ , g φ ( z ) , s 0 , s g ) , the plan with those subgoals.
In principle
HVF can be called at every step, but for computational efﬁciency we only call HVF once.
Subgoal optimization: Since we need to search in the space of subgoals to optimize Equation 1, we perform subgoal optimization using the cross entropy method (CEM) (Rubinstein & Kroese, 2004) in the latent space of our generative model g φ ( z ) .
Note this subgoal optimization CEM is distinct [Figure] from the CEM used for computing the planning cost, which we use as a subroutine within this outer level of CEM.
At a high-level, the subgoal optimization samples a list of subgoals, evaluates their effectiveness towards reaching the goal, and then iteratively reﬁnes the subgoals through resampling and reevaluation.
We begin by drawing M samples from a diagonal Gaussian distribution q =
N ( 0 , I ) where the dimensionality of q is K ∗ L , where K is the number of subgoals and L is the size of the latent dimension z of the generative model g φ ( z ) (Alg. 1 Line 6).
Thus, one sample z from q gives us K latents each of size L , each of which is decoded into a subgoal image (Alg. 1 Line 8).
Then, as described in Equation 1, the cost for one sample (one set of K subgoals) is computed as the maximum planning cost across the subsegments (Alg. 1 Lines 9:12).
The M samples are ranked by this cost, and then q is reﬁt to the latents z of the best M ∗ samples (Alg.
1 Lines 14:15).
In all of our experiments we use M = 200 , M ∗ = 40 and L = 8 .
E XPERIMENTS
In our experiments, we aim to evaluate (1) if, by using HVF, robots can perform challenging goal-conditioned long-horizon tasks from raw pixel observations more effectively than prior selfsupervised approaches, (2) if HVF is capable of generating realistic and semantically signiﬁcant subgoal images, and (3) if HVF can scale to real images of cluttered scenes.
To do so, we test on three domains: simulated visual maze navigation, simulated desk manipulation, and real robot manipulation of diverse objects.
The simulation environments use the MuJoCo physics engine (Todorov et al., 2012).
We compare against three prior methods: (a) visual foresight (Finn & Levine, 2016;
Ebert et al., 2018b), which uses no subgoals, (b) RIG (Nair et al., 2018a) which trains a model-free policy to reach generated goals using latent space distance as the cost, and (c) visual foresight with subgoals generated by time-agnostic prediction (TAP) (Jayaraman et al., 2019), a state-of-the-art method for self-supervised generation of visual subgoals, which generates subgoals by predicting the most likely frame between the current and goal state.
5. M AZE N AVIGATION
[Figure]
First, we study HVF in a 2D maze with clear bottleneck states, as it allows us to study
how
HVF compares to oracle subgoals.
In the the maze navigation environment, the objective is for the agent (the green block) to move to a goal position, which is always in the rightmost section.
To do so, the agent must navigate through narrow gaps in walls, where the position of the gaps, goal, and initial state of the agent are randomized within each episode.
The agent’s action space is 2D Cartesian movement, and the state space is top down images.
We consider three levels of difﬁculty, “Easy” where the agent spawns in the rightmost section, “Medium” where the agent spawns in the middle, and “Hard” where the agent spawns in the leftmost section.
Details in Appendix B.1.
Videos/code are available at: google stanford.
[Figure]
Results:
In Figure 2, we observe that using HVF subgoals consistently improves success rate over visual foresight (Finn & Levine, 2016) without subgoals, indicating that it is able to ﬁnd subgoals that make the long-horizon task more manageable.
Additionally, we compare to the “Ground Truth
Bottleneck” that uses manually-designed subgoals, where the subgoal is exactly at the gaps in the walls (or the midpoint between states in the “easy” case).
We ﬁnd that while using the oracle subgoals tends to yield the highest performance, the oracle subgoals do not perform perfectly, suggesting that a non-trivial amount of performance gains are to be had from improving the consistency of the video prediction model and cost function for short-horizon problems, as opposed to the subgoals.
We also show that HVF outperforms time agnostic prediction (TAP)
(Jayaraman et al., 2019) in Appendix C.3.
We next qualitatively examine the subgoals discovered by HVF in Figure 3, and ﬁnd empirically that they seem to correspond to semantically meaningful states.
In this task there are clear bottlenecksspeciﬁcally reaching the gaps in the walls and ﬁnd that HVF outputs close to these states as subgoals.
For example, when the agent starts in the leftmost section, and has to pass through two narrow gaps in walls, we observe the ﬁrst subgoal goal corresponds to the agent around the bottleneck for the ﬁrst gap and the second subgoal is near the second gap.
5. S IMULATED D ESK M ANIPULATION
[Figure]
We study the performance improvement and subgoal quality of HVF in a challenging simulated robotic manipulation domain.
Speciﬁcally, a simulated Franka Emika Panda robot arm is mounted in front of a desk (as used in (Lynch et al., 2019)).
The desk consists of 3 blocks, a sliding door, three buttons, and a drawer.
We explore four tasks in this space: door closing, 2 block pushing, door closing + block pushing, and door closing + 2 block pushing.
Example start and goal images for each task are visualized in Figure 5, and task details are in Appendix B.2.
The arm is controlled with 3D Cartesian velocity control of the endeffector position.
Across the 4 different tasks in this environment, we use a single dynamics model f θ and generative model g φ ( z ) .
Experimental details are in the Appendix B.2.
Results: As seen in Figure 4, we ﬁnd that using HVF subgoals dramatically improves performance, providing at least a 20% absolute improvement in success rate across the board.
In the task with the longest horizon, closing the door and sliding two blocks off the table, we ﬁnd that using no subgoals or 1 subgoal has approximately 15% performance, but 2 subgoals leads to over 42% success rate.
We compare to subgoals generated by time agnostic prediction (TAP) (Jayaraman et al., 2019) and ﬁnd that while it does generate plausible subgoals, they are very close to the start or goal, leading to no beneﬁt in planning.
We also compare against RIG (Nair et al., 2018a), where we train a model free policy in the latent space of the VAE to reach “imagined” goals, then try and reach the unseen goals.
[Figure]
However, due to the complexity of the environment, we ﬁnd that RIG struggles to reach even the sampled goals during training, and thus fails on the unseen goals.
Qualitatively, in Figure 5, we observe that HVF outputs meaningful subgoals on the desk manipulation tasks, often produces subgoals corresponding to grasping the handle, sliding the door, or reaching to a block.
5. R EAL R OBOT M ANIPULATION
Lastly, we aim to study whether HVF can extend to real images and cluttered scenes.
To do so, we explore the qualitative performance of our method on the BAIR robot pushing dataset (Ebert et al., 2018a).
We train f θ and g φ ( z, s 0 ) on the training set, and sample current and goal states from the beginning and end of test trajectories.
We then qualitatively evaluate the subgoals outputted by HVF.
Further implementation details are in the Appendix B.
. Results:
Our results are illustrated in Fig.
We observe that even in cluttered scenes, HVF produces meaningful subgoals, such grasping objects which need to be moved.
For more examples, see Figure C.2.
We observe that in both the BAIR dataset and the desk manipulation experiments, the most common failure cases corresponded to the subgoal prediction and model ignoring the objects and focusing exclusively on the arm.
Cost functions which can more effectively capture objects and their poses would be a step towards addresing this.
5. A BLATIONS
[Table]
In our ablations
[Table]
we explore three primary questions: (1) What is the optimal number of subgoals?
, (2) Is there a difference between HVF using the max and mean cost across subsegments?, and (3) Is HVF as valuable when the samples used for visual MPC is signiﬁcantly increased?
We evaluate these questions in the maze navigation task on the “Hard” difﬁculty.
All results report success rates over 100 trials using random initialization/goals.
Additional ablations on planning horizion and latent space cost can be found in Appendix C.
[Table]
[Table]
Number of Subgoals: We explore how HVF performs as we increase the number of subgoals in Table 1.
Interestingly, we observe that as we scale the number of subgoals beyond 2, the performance starts to drop, and with 5 or more subgoals the method fails.
We conclude that this is due to the increasing complexity of the subgoal search problem: the sampling budget allocated for subgoal optimization is likely insufﬁcient to ﬁnd a large sequence of subgoals
[Table]
[Table]
Max vs Mean: In our HVF formulation, we deﬁne the subgoal optimization objective as ﬁnding the subgoals that minimize the max cost across subsegments.
Table 2 compares to using the mean cost.
We ﬁnd that using the max cost is marginally better.
Sample Quantity: In Table 3, we examine how the number of action samples affects the relative improvement of
HVF.
Using more samples should also mitigate the challenges of sparse costs, so one might suspect that HVF would be less valuable in these settings.
On the contrary, we ﬁnd that HVF still signiﬁcantly outperforms no subgoals, and the improvement between using 0 and 1 subgoals is even more signiﬁcant.
C ONCLUSION AND L IMITATIONS
We presented an self-supervised approach for hierarchical planning with vision-based tasks, hierarchical visual foresight (HVF), which decomposes a visual goal into a sequence of subgoals.
By explicitly optimizing for subgoals that minimize planning cost, HVF is able to discover semantically meaningful goals in visual space, and when using these goal for planning, perform a variety of challenging, long-horizon vision-based tasks.
Further, HVF learns these tasks in an entirely selfsupervised manner without rewards or demonstrations.
While HVF signiﬁcantly extends the capabilities of prior methods, a number of limitations remain.
First, HVF assumes access to an exploration policy which can cover enough of the state space to learn a good model.
While the random policy used works for our environments, more complex environments may require better exploration techniques, such as intrinsic motivation methods.
Another limitation of the current framework is its computational requirements, as the nested optimization procedure requires many iterations of MPC with expensive video prediction models.
Speciﬁcally, assume that the horizon of the task is T , one iteration of visual MPC has computational cost C , and HVF is using K subgoals, and searching in a space of N subgoal sequences.
Then normal visual foresight or TAP would have cost T × C , while HVF would have cost ( T × C )
+ ( K × N × C ) .
However note - the computation of ( K × N × C ) can be heavily parallelized because it is done completely ofﬂine (without environment interaction).
We also expect that this can be mitigated by reformulating the subgoal prediction problem as policy inference, and training a subgoal generation policy in the loop of training, an interesting direction for future work.
Further, while the development of accurate visual cost functions and predictive models were not the main aim of this work, the performance with oracle subgoals suggests this to be a primary bottleneck in furthering the performance of HVF.
Speciﬁcally, learning dynamics models and cost functions which can more effectively capture the state of all the objects in the scene would signiﬁcantly improve the performance of HVF.
Lastly, development of better generative models that can capture the scene and generate possible futures remains an open and challenging problem.
Work in generative models that enable better generation of cluttered scenes with novel objects could improve the applicability of HVF in realworld settings. A CKNOWLEDGMENTS
We would like to thank Sudeep Dasari, Vikash Kumar, Sherry Moore, Michael Ahn and Tuna
Tokosz for help with infrastructure, as well as Sergey Levine, Abhishek Gupta, Dumitru Erhan,
Brian Ichter and Vincent Vanhouke for valuable discussions.
This work was supported in part by an NSF GRFP award. Chelsea Finn is a CIFAR Fellow in the Learning in Machines & Brains program A M ETHOD D ETAILS A. V ISUAL MPC
In HVF, evaluating the cost of potential subgoals, as well as actually taking actions given computed subgoals uses visual MPC.
One step of visual MPC is described in Algorithm 1.
Given the current state and goal state, MPC samples action trajectories of length H , then feeds them through the model f θ .
Then the cost of the output images are computed relative to the goal image, which is then used to sort the actions and reﬁt the action distribution.
After 5 iterations or convergence the best action is returned.
Receive current state s t and goal state s g from environment
Initialize N ( µ, σ 2 ) =
N (0 , 1) , cost function C ( s i , s j ) while ( σ 2 > 1 e − 3 ) or (iterations ≤ 5 ) do a 1 , ..., a D ∼ N ( µ, σ 2 ) s t
+ H, 1 , ..., s t
+ H,D = f θ ( s t , a 1 , ..., a D ) l 1 , ..., l D =
[ C ( s t
+ H, 1 , s g ) , ..., C ( s t
+ H,D , s g )] a sorted =
Sort ([ a 1 , .., a D ])
Reﬁt N ( µ, σ 2 ) to a sorted [1 − D ∗ ] end while
Return l sorted [1] , a sorted [1] where D = 200 , D ∗ =
A.
P LANNING WITH S UBGOALS
In the main text we describe how given, s 0 and s g , HVF produces subgoal images s 1 , s 2 , ..., s K .
Given these subgoals, planning with them is executed as follows.
For an episode starting at sof maximum length T , the agent plans using visual MPC from s 0 to s 1 until some cost threshold
C ( s t , s 1 ) <
x or for some maximum number of steps T ∗ .
Once this criteria is reached, the agent plans from its current state to the next subgoal s 2 , again until the cost threshold C ( s t , s 2 ) <
x or some maximum number of steps T ∗ , and so until the agent is planning form s t to the true goal s g , at which point it simply plans until the environment returns that the task has been a success or the total horizon T runs out.
A. G ENERATIVE M ODEL
The generative model we use is either a Variational Autoencoder s ∼ g φ ( z ) or a Conditional Variational Autoencoder s ∼ g φ ( z, s 0 ) .
In the VAE, training is done by sampling images from the dataset, and each image is encoded using an image encoder into the mean and standard deviation of a normal distribution of dimension 8.
Then a sample from the distribution are decoded back to the original input image.
This is trained with a maximum likelihood loss as well as a KL penalty on the normal distribution restricting it to be a unit gaussian N (0 , 1) .
In the conditional VAE, training is done by sampling pairs of images from the dataset, speciﬁcally images from random episodes s t as well as the corresponding ﬁrst image of that episode s 0 .
Then both s 0 and s t are encoded using the same image encoder, which are then ﬂattened and concatenated.
This is then encoded into the mean and standard deviation of a normal distribution of dimension 8.
A sample from the resulting distribution is then fed through fully connected layers and reshaped into the output shape of the encoder, concatenated with the spatial embedding of the conditioned image from the encoder and then decoded.
This is done to enable easier conditioning on the scene in the decoding.
The resulting image is trained with a maximum likelihood loss as well as a KL penalty on the normal distribution restricting it to be a unit gaussian N (0 , 1) .
In the Simulated Maze Navigation experiments, we use a Conditional VAE which encodes the 64x
RGB image with 4 convolutional layers ([16, 32, 64, 128] ﬁlters, kernel size 3x3, stride 2), as well as 3 fully connect layers of size 256 for the generated image and 3 fully connected layers of size [512, 512, 256] for the conditioned image, and a decoder with the same architecture as the encoder inverted.
In the Simulated Desk Manipulation experiments, we use a VAE which encodes the 64x
RGB image with 7 convolutional layers ([8, 16, 32, 32, 32, 64, 128] ﬁlters, kernel size 3, stride alternating between 1 and 2), as well as a single fully connected layer of size 512 before mapping the latent distribution, which is decoded with the inverted encoder architecture.
In the Real Robot
Manipulation experiments we use a Conditional VAE which encodes the 48x64 RGB image withconvolutional layers ([8, 16, 32, 64, 64, 128, 128, 256, 512] ﬁlters, with kernel size 3, and stride sizes [1,1,1,1,1,1,2,2,2]), as well as 3 fully connect layers of size 256 for the generated image andfully connected layers of size [512, 512, 256] for the conditioned image, and a decoder which is the inverted architecture of the encoder.
The three experiments use Adam optimizer with learning rate 1e-4, 1e-3, 1e-4 respectively.
[Figure]
The dynamics model is an action conditioned video prediction model, stochastic variational video prediction (SV2P) (Babaeizadeh et al., 2017).
See architecture below:
For all experimental settings the dynamics models are trained for approximately 300K iterations.
B E XPERIMENT D ETAILS B. M AZE N AVIGATION
Data Collection: For the maze navigation environment data is collected through random policy interaction.
That is, for each action we sample uniformly in the delta x, y action space of the green block.
We collect 10000 episodes, each with random initialization of the walls, block, and goal.
Each episode contains 100 transitions.
The images are of size 64x64x3.
Cost Function:
In this experiment, the cost function C ( s i , s j ) is simply the squared (cid:96) 2 pixel distance between the images, that is ||
s
i − s j
||
Planning Parameters: When planning in the maze navigation environment, MPC samples action trajectories of length H = 5 .
Additionally, the cost of a sequence of 5 actions is the cost of the last of the subsequent frames, where the cost of each frame is the temporal cost C described above.
Lastly, we use T = 50 and T ∗ = 10 in these experiments.
B. S IMULATED D ESK M ANIPULATION
Task Details: Door: The ﬁrst task is reaching to and closing the sliding door.
From the initialization position of the arm, it needs to reach around the door handle into the correct position, then slide the door shut.
The position of the door and distractor blocks on the table are randomized each episode.
The agent is given 50 timesteps to complete the task.
2 Block: The second task requires the agent to push two different blocks off the table, one located on the left end of the table and one located in the middle.
It requires pushing ﬁrst one block, then re positioning to push the other block off the table.
The agent is given 50 timesteps to complete the task.
Door + Block: The agent must both slide the door closed from a random position, as well as push a block off of the table.
This requires both grasping and sliding the door from the previous task, as well as positioning the end effector behind the block to slide it off the table.
The agent is given 100 timesteps to complete the task.
Door +
Block: A combination of the 2 Block and Door + Block task.
The agent must close the door and slide both blocks off the table within 100 timesteps.
Data Collection:
For the simulated desk manipulation environment data is again collected through random policy interaction.
That is, for each action we sample uniformly in the delta x, y, z action space of the robot end effector.
We collect 10000 episodes, each with random initialization of the desk door/drawer and blocks.
Each episode contains 100 transitions.
The images are of size 64x64x3.
Cost Function:
In this experiment, the cost function C ( s i , s j ) is simply the squared (cid:96) 2 pixel distance between the images, that is ||
s
i − s j ||
Planning Parameters: When planning in the desk manipulation environment MPC samples action trajectories of length H = 15 , which actually consists of 5 actions, each repeated 3 times.
Additionally, the cost of a sequence of 15 actions is the (cid:96) 2 pixel cost C of the last frame only .
Note this is distinct from the maze environment.
Lastly, we use T = 50 or 100 and T ∗ = 20 in these experiments, depending on the task.
B. R EAL R OBOT M ANIPULATION
Data: We use the BAIR robot manipulation dataset from (Ebert et al., 2018a), which consists of roughly 15K trajectories split into a train/test split.
We train f θ and g φ on the train set and display qualitative results on the test set.
Cost Function: Like the desk manipulation set up, the cost function C ( s i , s j ) is the squared (cid:96)pixel distance between the images, that is ||
s
i − s j
||
Planning Parameters: When planning in the desk manipulation environment MPC samples action trajectories of length H = 15 , which actually consists of 5 actions, each repeated 3 times.
Additionally, the cost of a sequence of 15 actions is the (cid:96) 2 pixel cost C of the last frame only .
C A DDITIONAL R ESULTS
C.
A DDITIONAL A BLATIONS
C.1.
P LANNING H ORIZON
[Table]
[Table]
In the maze environment, on the hard difﬁculty, we study how increasing the planning horizon of visual MPC impacts the beneﬁt of using
HVF (See Table 4).
Interestingly, we ﬁnd that for longer planning horizons, performance does not necessarily improve (as a longer planning horizon constitutes a harder search problem).
This supports the idea that simply using longer planning horizon is not necessarily the solution to doing long-horizon tasks.
Further, we ﬁnd that even with longer planning horizons, HVF outperforms standard visual foresight, but that performance with 2 subgoals is worse than with 1 subgoal.
C.1. VAE L ATENT S PACE C OST
To address the sparsity of pixel cost, we explore using distance in the latent space of the VAE as a cost signal.
However, we ﬁnd that this cost actually performs worse across all difﬁculties.
In Table 5 we show the success rate of standard visual foresight for either pixel (cid:96) 2 cost or VAE latent space (cid:96) 2 cost.
We ﬁnd that the reason for the poor performance of the latent space cost is that it provides close to zero cost anytime the green blocks are reasonably close together, leading to CEM often getting stuck close to the goal but not actually at the goal.
C.
[Figure]
C. T IME A GNOSTIC P REDICTION (J AYARAMAN ET AL ., 2019) IN THE M AZE N AVIGATION T ASK
We observe that while TAP gets similar performance to HVF on the “Easy” and “Medium” cases, it has signiﬁcantly lower performance in the longest horizon “Hard” setting.
[Table]
Additionally we see that Recursive TAP withsubgoals has lower performance across all difﬁculties as the subgoals it outputs are very close to current/goal state (See Figure 9).
[Figure]