Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
In this paper, we study the problem of solving long-horizon multistage manipulation tasks from raw image observations.
Reinforcement learning(RL) is a promising approach for acquiring skills that can be sequenced to solvesuch tasks.
However, the focus of RL algorithms has largely been on the success ofthose low-level skills, more so than satisfying the demands of extended multi-stagetasks that we ultimately want robots to complete.
The latter demands robustnessand persistence, as errors can compound over time, and may require the robot tohave a number of primitive skills in its repertoire, rather than just one.
To this end,we introduce EMBR, a model-based RL method that can learn primitive skills thatare suitable for solving long-horizon visuomotor tasks.
EMBR learns and plans using a learned model, critic, and success classiﬁer, where the success classiﬁerserves both as a reward function for RL and as a grounding mechanism to continuously detect if the robot should retry a skill when unsuccessful or under perturbations.
Further, the learned model is task-agnostic and trained using data from alltasks, enabling the robot to efﬁciently learn a number of distinct primitives.
Thesevisuomotor primitive skills and their associated pre- and post-conditions can thenbe combined with off-the-shelf symbolic planners to solve long-horizon tasks.
Ona Franka Emika robot arm, we ﬁnd that EMBR enables the robot to solve threelong-horizon visuomotor tasks at 85% success rate, such as organizing an ofﬁcedesk, a ﬁle cabinet, and drawers, which require sequencing up to 12 skills, involve14 unique learned primitives, and demand generalization to novel objects.
Videosare at https://sites.google.com/view/anonymous-ember .
Introduction
We want robots to robustly perform a variety of long-horizon visuomotor manipulation tasks.
Sucha capability can enable robots to gradually become more useful in our physical world.
Long-horizonmanipulation tasks can be solved by composing a sequence of low-level primitive skills, otherwiseknown as “manipulation primitives” [1] or “options” [2].
For example, the long-horizon task of“organizing an ofﬁce desk” often means performing a long sequence of shorter tasks, such as pickingobjects up, opening or closing drawers, placing objects into drawers, and putting pens into penholders.
However, solving such a long-horizon task may require the robot to learn many low-levelvisuomotor skills, compose them sequentially, and generalize across novel objects.
During task execution, the robot also needs to detect its own failures (e.g. accidentally dropping an object) andcorrect them (e.g. picking the object up again) before continuing to the remaining parts of the task.
We hope to endow robots with such generalization and robustness.
Approaches such as symbolic planning [3], hierarchical RL [4, 5, 6, 7, 8], and hierarchical planning [9] can accomplish compositional task generalization by reasoning in an abstract or symbolicspace.
While these approaches have shown promising results on tasks such as simulated tool use [10]and non-vision-based manipulation [9], they rely on the ability to acquire a repertoire of robust,low-level motor skills.
These low-level skills can in principle be acquired through reinforcementlearning, but a number of challenges remain.
First, the acquisition of robust and persistent low-levelvisuomotor manipulation skills is still challenging, and the errors in completing any sub-sequence ofa long-horizon task can compound over time and lead to overall task failure.
Second, acquiring therequisite low-level visuomotor skills requires data-efﬁcient learning and reward or task speciﬁcationacross a broad set of skills.
Finally, composing the learned primitive skills to perform a long-horizontask requires persistent grounding of the robot’s visual observation within the broader abstract plan.
To address these challenges, we introduce example-driven model-based RL (EMBR), an algorithmthat learns a dynamics model and a set of Q-functions and success classiﬁers to efﬁciently acquire arepertoire of visuomotor skills that are both robust and can be grounded into symbols for symbolicplanning of a long-horizon visuomotor task.
More speciﬁcally, the success classiﬁers in EMBRenable both robustness and skill grounding, as they allow the robot to determine when a skill shouldbe re-tried and can be used to determine the pre- and post-conditions of a skill.
For example, areward function and post-condition for the skill of opening a drawer can be obtained from learninga binary image classiﬁer of “whether the drawer is open” given the robot’s pixel observation.
At thesame time, this classiﬁer also enables the robot to persistently try to open the drawer until it is ﬁnallyopen.
Further, these success classiﬁers require only a modest number of example images, makingit easy to specify a variety of skills.
Finally, by learning a task-agnostic dynamics model, EMBRallows data to be shared across all tasks, leading to greater data-efﬁciency and generalization.
The main contribution of this work is a framework for learning grounded visuomotor skills fromhuman-provided example images that can be sequenced to solve long-horizon tasks with off-theshelf symbolic planners.
Experiments on a Franka Emika Panda robot arm ﬁnd that EMBR enablesthe robot to solve three long-horizon tasks from raw pixels with 85% success, such as organizinga desk, a cabinet, and two drawers, which require up to 14 learned unique primitives, sequencingup to 12 skills, and generalization to novel objects.
Moreover, our ablation studies conﬁrm thatall components of our method including the dynamics model, success classiﬁers, Q-functions, andstages are essential to performance, particularly for tackling challenging long-horizon tasks.
Related Work
Reinforcement learning (RL) has a rich history of being used for robotic control problems on tasksfrom locomotion [11, 12, 13] to object manipulation [14, 15, 16].
In this work, we focus on learningvision-based robotic manipulation skills, a problem studied by many prior works [17, 18, 19, 20].
Model-free deep RL has been effective on short-horizon skills like object grasping [21, 20, 22],pushing and throwing [19, 23, 24], and multi-task learning [23, 25] from images.
Alternatively,model-based approaches [26, 27], which explicitly learn the environment forward dynamics fromimages have also been employed for multi-task vision-based robotic manipulation tasks, either withplanning algorithms [28, 29, 30, 31, 32, 33, 34, 35] or for optimizing a parametric policy [36].
Unlikethese prior works, our algorithm uses a model in conjunction with task-speciﬁc success classiﬁersand Q-functions to learn a wide range of skills suitable for sequencing into long-horizon tasks.
In
Section 5, we ﬁnd that both components of our approach are essential for robust performance.
Even with powerful reinforcement learning methods, task speciﬁcation on real robots remains challenging, as engineering rewards on physical systems can be costly and time-consuming [37].
Motivated by this, many works have studied the reward speciﬁcation, including inverse reinforcementlearning [38] with robot demonstrations [39, 40, 41, 42, 43], learning rewards from user preferences[44, 45, 46, 47], and learning rewards from videos of humans [48, 49].
One common approachin visual RL is to learn to reach a goal image using a coarse measure of reward like negative ` 2pixel distance [30, 50, 51] or temporal distance [52, 53, 34].
While these techniques have had somesuccess on real robots, such rewards often provide a sparse and difﬁcult to optimize reward signal.
Alternatively, a number of recent works learn a classiﬁer from a modest number of examples goalstates [54, 55, 24, 56, 57, 25] and tries to learn agents which maximize the classiﬁer score.
In this work, we extend this approach to multiple stages of human-provided goals, which we ﬁnd enablesmore effective learning of hard exploration skills, and ﬁnd that such reward speciﬁcation can be usedto learn 14 unique visuomotor skills on a real robot, without the need for any demonstrations.
Like many prior works, the goal of this work is to complete challenging long-horizon visuomotortasks.
One class of prior work is hierarchical RL, which aims to learn temporally-extended primitiveskills and a high-level policy over them [4, 5, 6, 7, 8, 58].
Such approaches include jointly learningthe primitive skills with the high-level policy through goal generation or an “options” framework[6, 8, 59], learning primitives through intrinsic motivation or other auxiliary objectives [60, 61, 62],from demonstration behavior [63, 64, 65, 66, 67], or from low-level, potentially goal conditioned,reward functions [7, 58, 68, 69].
EMBR is not a hierarchical RL algorithm, as it only learns lowlevel skills provided to the high-level planner and does not learn a model over options.
Nevertheless,our work is similar to the last group, except that we learn visuomotor primitives on a real robotfrom rewards derived from human examples.
Other works have also explored learning hierarchicalpolicies on top of hand-designed skills [70, 71, 72].
These methods are complementary to EMBR,as skills learned via EMBR can be incorporated into most hierarchical policy learning algorithm.
Alternatively, a number of recent works have studied long horizon visual planning [52, 53, 73, 74,75, 50, 76, 77, 78], often by combining a structured search approach with learned components likegenerative models [73, 50, 76] or distance functions [53, 34].
While these approaches have demonstrated impressive results on problems such as visual navigation [52, 53], unlike prior work we learnmanipulation tasks on a real robots that consists of up to 12 unique skills and 250+ timesteps.
Lastly, the task and motion planning (TAMP) literature [79, 80] has extensively studied long-horizonrobotic tasks.
Unlike these works, we do not assume pre-deﬁned state representations or accuratestate estimators, making it possible to handle novel objects in cluttered scenes.
Like our work,a number of works have also explored combining learned models and skills with hand-designedor symbolic planning [10, 9, 81] and learning to ground skills for planning [82].
Unlike thesetechniques, a key insight in this work is that we can jointly learn visuomotor skills and the associatedgrounding of their pre- and post-conditions, allowing us to leverage off-the-shelf symbolic plannersto execute long horizon tasks in a closed-loop fashion directly from pixels on a real robot.
Preliminaries
Tasks and skills .
We consider the problem of solving a long-horizon visuomotor task M by learninga repertoire of visuomotor skills.
We model the robot’s environment as a controlled Markov process
E = h S , ⇢ 0 , A , T ,  , H i , with an image observation space s 2 S , an initial state distribution ⇢ 0 , an 120 action space a [Equation] , a dynamics model
[Equation]
R , a discount factor  2 [0 , 1) , and aﬁnite horizon H .
We notice that many long-horizon tasks are composed of a sequence of lower-levelskills.
For example, to organize an ofﬁce desk, the robot needs to acquire a number of skills suchas grasping an object, opening a drawer, and placing an object in a drawer.
Let K denote the totalnumber of unique skills the robot has acquired, and k 2 [1 , K ] denote the k th skill in the robot’s skillrepertoire.
Depending on the environment state, the order in which the skills should be executed tosuccessfully complete a long-horizon task varies.
For example, to organize a desk, the robot maynot need to use the skill of closing a drawer if the drawer is already closed.
Here, each skill is an
MDP M k = h E ,
R k
i , where the robot’s environment E is shared across all tasks and skills, and
R k : S ⇥ A !
R is the reward function for the k th skill (e.g. the skill of opening a drawer).
Problem Deﬁnition .
Formally, EMBR takes as input N human-provided example images k
=s 1 , . . .
, s N for each skill k
we intend to robot to learn.
Our goal (i.e. EMBR’s output) is to learnthe symbolic grounding of the pre- and post-conditions g kpre , g kpost : S ! { 0 , 1 } , and a policy⇡ k : S !
⇧ ( A ) for each skill k , where ⇧ ( · ) deﬁnes a probability distribution over a set.
Eachpolicy has a Q-function, Q k : S ⇥ A !
R , which estimates the discounted cumulative reward fortaking action a from observation s and following ⇡ k onward.
Variational Autoencoders (VAEs) .
VAEs compress high-dimensional observations s such as images into an embedding z .
VAEs can be optimized by maximizing the evidence lower bound(ELBO): max p,q L vae ( s ) , where L vae ( s ) =
E q ( z | s ) h
[Equation]
p [Equation] 1),where p denotes the generative model, and q denotes the variational distribution.
Symbolic planning .
A symbolic planner performs task planning in the symbolic domain.
Concretely, given a set of candidate actions, each action’s associated pre- and post-conditions, the current condition h and a goal condition g , a symbolic planner outputs the sequence of actions that allows the robot to reach from h to g .
The pre-condition of an action is a set of predicates that deﬁnesthe condition that needs to be satisﬁed in order for the action to be executable.
The post-conditionof a skill is a set of predicates that deﬁnes the effect of executing an action.
For example, the precondition and post-condition of an “Open left drawer” action is “Drawer is closed” and “Drawer isopen” respectively.
In our context, the candidate actions are the primitive skills learned by EMBR.
Example-Driven Model-Based RL (EMBR)
To learn a large repertoire of skills that can be sequenced to solve long-horizon robotic manipulation, we propose Example-driven Model-based RL, or EMBR.
Below we ﬁrst describe how EMBRlearns each visuomotor skill without any demonstration skills or trajectories during training, andthen elaborate on how these learned skills can be composed to solve a long-horizon task at test time.
4.1 Learning a Repertoire of Visuomotor Skills Autonomously without Demonstrations
During training, EMBR learns a repertoire of low-level persistent visuomotor skills, such as openinga drawer or inserting a marker, using a set of example success images provided by a person foreach task.
The model learned by EMBR, denoted as f , is composed of four components: a globalvariational autoencoder (VAE) f vae , a global latent dynamics model f T , stage-speciﬁc binary rewardfunctions f k R , and stage-speciﬁc Q-functions f kQ .
The VAE and latent dynamics model jointly learna low-dimensional representation of the image observations to address the challenge of efﬁcientlylearning many primitive skills; both are trained on data collected across all skills.
The skill-speciﬁcbinary classiﬁers allow us to learn a reward function for each skill from human-provided images.
Weuse the latent dynamics model to perform model predictive control with the skill-speciﬁc Q-functionas the terminal planning objective.
The skill-speciﬁc Q-functions and reward functions are trainedon skill-speciﬁc data.
Next, we elaborate on each of these components.
Learning a low-dimensional latent space with a VAE and a latent dynamics model .
Performinga long-horizon task using a large number of visuomotor skills demands the algorithm to be sampleefﬁcient and fast at execution time.
To address both of these demands, we learn a single VAEf vae
= { f enc , f dec } to construct a latent space Z .
This VAE consists of an encoder f enc : S !
⇧ ( Z ) 169 and a decoder f dec : Z ! ⇧ ( S ) .
We train f vae using data collected across all skills via Eq.
1, where 170 p ( s | z ) ⌘ f dec ( s t | z t ) , p ( z ) ⇠ N (0 , 1) , and q ( z | s ) ⌘ f enc ( z t | s t ) .
By learning from all skills, 171 the VAE can improve data efﬁciency.
In practice, we use the greedy hierarchical VAE [51] as f vaewith the current image observation as a context frame.
Based on this latent space, EMBR learns alatent dynamics model f T jointly with f vae with data collected across all skills via l 2 loss:
[Equation]
where z t ⇠ f enc ( · | s t ) , L is the rollout length of f T and D is the dataset collected by the robot.
Obtaining rewards by learning image classiﬁers .
Obtaining a reward function in RL typicallyrequires some forms of human supervision such as reward engineering or demonstrations.
In EMBR, binary image classiﬁers serve not only as reward functions for learning primitive visuomotor skills,but also as success detectors as well as grounding of raw pixels for long-horizon symbolic planning.
Concretely, we learn a reward function f k R for each skill M k by training binary image classiﬁersto distinguish human-provided example images from robot-collected non-example images using thebinary cross-entropy objective [55, 56].
The reward learning objective for skill k is:
[Equation]
Here, D + is the dataset of images labeled as positive by a person (typically 100-200 images), s +are the images sampled from D + , and s  are sampled from all images in D , the dataset collectedby the robot, and are considered negative by default.
In our experiments, these positive examplescan be collected in 15-30 minutes per visuomotor skill, and 345 minutes in total for all 14 skills.
Nodemonstration trajectories are collected.
For skills that are longer horizon, such as grasping objects,we break the skill down into two “stages” and learn a reward function for each stage.
Details of suchskill breakdown and the example-image collection process are in Appendix C and M respectively.
At test time, the learned image classiﬁers are re-purposed as post-condition grounding of each skillfor long-horizon task execution.
Some of the pre-conditions the task planner requires can be derived from existing post-conditions.
For example, for a drawer-opening skill, the pre-condition of“drawer is not open” can be derived from the post-condition of “drawer is open” by negating thispost-condition.
For the remaining pre-conditions that cannot be derived from any existing postconditions, we train additional image classiﬁers using Eq. 3.
Learning Q-functions for model-based control .
While one can directly use the latent dynamicsmodel for model-based planning, the difﬁculty of learning an accurate dynamics model for the fullhorizon of a primitive skill increases with the length of the skill itself.
To alleviate this challengeand further accelerate skill learning, we learn a Q-function f kQ in the latent space.
Concretely, weperform Q-learning via the following objective for skill k :
[Equation]
where f k R ( z ) ⌘ 1 { f k R ( z ) > 0 . 5 } , z t ⇠ f vae ( · | s t ) , z t +1 ⇠ f vae ( · | s t +1 ) , and the Q-value target iscomputed by maximizing over m 1 randomly and uniformly sampled actions a 1: m 1 t
+1 .
In the case of alonger-horizon skill such as grasping objects, we break the skill down into two “stages” and learn areward function for each stage (Appendix C).
At both training and test time, we perform model-based planning in the latent space by ﬁrst rollingout f T for L timesteps into the future and then evaluating the predicted future latent space observa-
[Figure]
tion at timestep t + L using the learned Q-function, a process that is illustrated in Fig. 2:
[Equation]
[Equation]
Here, the outer and inner maximization operations are taken over m 0 and m 1 samples respectively.
Integrating these four components, EMBR is trained simultaneously across all components fromscratch (i.e. without pre-training) for learning a repertoire of visuomotor primitive skills (Alg. 1).
Concretely, the robot begins with an empty dataset D and can learn each skill in almost any order(Appendix D.2).
During skill learning, the robot collects a new trajectory using actions selectedfrom Eq.
5 and updates all model components using D .
This process repeats until the current skillreaches a desirable success rate as computed by the success classiﬁers, before the robot moves on tolearn the next skill.
As the robot acquires more skills, the dataset D accumulates and improves thegeneralization of the visual dynamics model, which in-turn accelerates future skill learning.
Wallclock training time per skill and hyperparameters are detailed in Appendix O and P respectively.
4.2 Composing Learned Visuomotor Primitive Skills for Long-Horizon Task Execution
At test time, EMBR provides the learned primitive skills and the symbolic grounding of each skill’spost-conditions (in the form of success classiﬁers) to a high-level task planner (standard PDDL
STRIPS symoblic planner in our case) to perform long-horizon task.
As elaborated in Alg. 2, thehuman ﬁrst speciﬁes a long-horizon task using a symbolic goal condition g .
Next, EMBR computesthe current symbolic condition of the environment from raw pixel observations by passing the currentimage observations to each of the success classiﬁers.
Using both the symbolic goal condition andthe current symbolic condition, the symbolic planner computes the sequence of skills to perform andexecutes the ﬁrst skill of this sequence using actions selected from Eq. 5.
This procedure repeats untilthe current condition matches the goal condition, after which task execution terminates.
Note thatthe arm position of the robot is reset after executing each skill.
The predicate deﬁnitions, the pre- andpost-conditions and the full details of the symbolic planner used in Section 5 are in Appendix D.3.
Experiments
Our experiments aim to answer four key questions: 1) how does EMBR compare to prior RL algorithms on both performing primitive skills and when sequencing those skills to solve long-horizontasks?
2) how necessary are stage-speciﬁc reward functions (as elaborated in Appendix C) compared to a single skill-speciﬁc reward function?
3) how important is replanning using the classiﬁers for the robustness of the algorithm?
and 4) how important is each of the ﬁve camera observations to the performance of the primitive skills?
To answer these questions, we conduct experiments across 14 primitive skills and three long-horizon tasks, using the experimental setup in Fig.
3, in which a Franka Emika Panda robot has access to ﬁve cameras, each capturing 64 {+⇥ 
+} RGB images.
One of the ﬁve cameras is mounted on the wrist.
To test object generalization, allobjects used in all evaluations are novel and unseen during training, except for the desk, drawers, cabinet, and the transparent marker holder.
Fig.
3 visualizes the train, validation, and test
[Table]
Here, EMBR outperforms the three prior methods by 15-25%, 20-30%, and 85% respectively
[Table]
Comparisons.
We compare EMBR to three prior methods: (1) EMBR w/o f T , which corresponds to
EMBR without the dynamics model, i.e. using only the Q-function and the VAE for action selection;(2)
EMBR w/o f vae , f T (Qt-Opt) [20], which is our re-implemented version of Qt-Opt [20] with a
VICE-like success classiﬁer [55] that does not learn a latent space and uses only the Q-function foraction selection; and (3) BEE [56], which uses the success classiﬁer instead of the Q-function formodel predictive control.
Comparison to Prior Method 4, the Deep Skill Chaining [59] hierarchical
RL method, is in Appendix G. All comparisons are representative of prior methods for vision-basedrobotic RL.
For all experiments, the data used to train each method is the same dataset collected by
EMBR using Alg. 1, which contains 300-400 trajectories collected in 16 hours per skill on average.
This provides a more direct comparison and in effect makes the prior methods stronger since thechallenge of online exploration is addressed by EMBR during data collection.
5.1 Short-horizon Primitive Skill Performance
To answer the ﬁrst question, we compare all methods across 14 visuomotor skills.
In Table 1, weobserve that “EMBR (Ours)” can reliably solve all 14 skills with at least 90% success.
Compared to“EMBR w/o f T ” and “EMBR w/o f vae , f T (Qt-Opt)”, “EMBR (Ours)” improves success rates by10-20% and 15-25% across high-precision skills, 5-10% and 15-20% for medium-precision skillsand 0% for low-precision skills.
We hypothesize that EMBR’s improvement over “EMBR w/o f T ”is due to its ability to alleviate overestimation of Q-values in the presence of limited training trajectories (300-400 per skill), and the addition success rate improvement of EMBR over “EMBR w/of vae , f T ” is due to less overﬁtting of the success classiﬁers and Q-functions in the VAE latent space.
Finally, EMBR outperforms BEE [56] by 50-65% on high-precision tasks because BEE [56] cannotcapture action consequences beyond its planning horizon and the model is not always accurate forthe full horizon of a skill.
We elaborate on these hypotheses in Appendix L.
5.2 Long-horizon Task Performance
Next, we compare EMBR to the three prior methods when sequencing the learned visuomotor skillswith symbolic planning to solve three challenging long-horizon tasks (Fig. 4):
Organize Desk and Cabinet (8 Skills) - putting three novel objects cluttered on the desk into the cabinet.
Organize Markers (9 Skills) - picking up three markers and inserting them into a transparent holder.
Detailed task descriptions are in Appendix D.7.
In Table 2, we ﬁnd that “EMBR (Ours)” solves theselong-horizon visuomotor tasks with 85% success, including the most challenging tasks that requireup to 12 unique skills.
Compared to “EMBR w/o f T ” and “EMBR w/o f vae , f T (Qt-Opt)”, EMBRleads to 10-20% and 20-30% improvement in success rates respectively because EMBR achieveshigher success rate on individual primitive skills that require higher precision, such as insertingmarker and grasping object from desk or drawers.
Finally, BEE [56], is unable to perform anylong-horizon task due to low performance on skills with medium or high precision.
This reﬂects theimportance of learning robust, high-success-rate primitive skills using EMBR.
5.3 Ablations
Next, we conduct six ablations to examine the importance of various components of EMBR.
Ablation 3, which compares EMBR with vs. without any one camera observation, is in Appendix J.
Comparisons to the original BEE and training without stages are in Appendix H and I.
Ablation 1: Stage-speciﬁc vs. skill-speciﬁc reward function .
While EMBR accelerates skillacquisition by learning a reward function for each stage of a skill, one can in principle learn just onereward function for each skill.
In this ablation, we answer Question 2 by comparing EMBR to anablated EMBR that learns each skill with a single skill-speciﬁc reward function.
This does not affectdata collection and exploration, which stage-speciﬁc rewards can also help.
In Table 1, we ﬁnd thatusing stage-speciﬁc reward functions outperforms using a single skill-speciﬁc reward function by10-30% for higher precision skills such as grasping an object in various locations, indicating theimportance of using stage-speciﬁc reward functions in long-horizon or higher precision skills.
Ablation 2: Task performance with vs. without classiﬁer-based replanning .
While EMBRimproves the robustness of long-horizon task execution by using success classiﬁers during planning,one can in principle perform model-based planning without these classiﬁers.
In this ablation, weanswer Question 3 by comparing EMBR to an ablation that executes each skill of a task sequentiallywithout using the success classiﬁers to determine when the skill is successful.
In Table 2, we ﬁndthat long-horizon task execution with classiﬁers improves performance by 30-35% success rate, dueto the robot’s ability to detect success in a closed-loop manner until successful.
These ablations and
Table 2 suggest that all components of EMBR are critical to solving long-horizon tasks.
Conclusion and Future Work
In this work, we propose EMBR – an example-driven model-based RL algorithm for learning visuomotor skills that can be sequenced to perform long-horizon tasks from raw pixels.
Across threelong-horizon visuomotor tasks which require up to 14 unique primitives and sequencing up to 12skills, we ﬁnd that EMBR consistently achieves 85% success rates, outperforms prior methods, andgeneralizes to novel objects.
Nonetheless, EMBR and our experiments also have a number of important limitations, including the limited scope of tasks, the amount of supervision required to solvethese complex long-horizon tasks, and the limited degree of generalization especially across different scenes.
We discuss these and other limitations in detail in Appendix Q, all of which leaveexciting avenues for further pushing the capabilities of robot learning systems in future work.