Guided Imitation of Task and Motion Planning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: While modern policy optimization methods can do complex manipulation from sensor data, they struggle on problems with extended time horizonsand multiple sub-goals.
On the other hand, task and motion planning (TAMP)methods scale to long horizons but they are computationally expensive and needto precisely track world state.
We propose a method that tries to get the best ofboth worlds: we train a policy to imitate a TAMP solver’s output.
This produces afeed-forward policy that can accomplish multi-step tasks from sensory data.
First,we build an asynchronous distributed TAMP solver that can produce supervisiondata fast enough for imitation learning.
Then, we propose a hierarchical policy architecture that lets us use partially trained control policies to speed up the TAMPsolver.
In robotic manipulation tasks, the partially trained policies reduce the timeneeded for planning by a factor of up to 2.5.
We show results in a range of pickplace tasks, solving the 4 object pick-place task from RoboSuite 88% of the timefrom object pose observations and solving the RoboDesk 9-goal multitask benchmark 67% of the time from RGB images.
Introduction
This paper describes a policy learning approach that leverages task-and-motion planning (TAMP)to train robot manipulation policies for long-horizon tasks.
Policy learning has demonstrated animpressive ability to accomplish robotic control tasks from complex sensory input [1, 2, 3].
But thatsuccess has largely been limited to short-horizon tasks. It is an open problem to learn policies thatexecute long sequences of manipulation actions [4, 5].
By contrast, TAMP methods readily solveproblems that require dozens of abstract actions that satisfy complex geometric constraints in highdimensional conﬁguration spaces [6, 7].
The need to robustly track world state and budget time forplanning constrains the application of these methods to controlled settings [8, 9].
To address these concerns two lines of work have emerged.
The ﬁrst line of work uses machinelearning to identify policies or heuristics that reduce planning time [10, 11, 12, 13, 14, 15].
Thisextends the domains where TAMP can be applied, but still relies on explicit state estimation.
Thesecond line of work learns policies that imitate the output of TAMP solvers.
These approachesreduce dependence on state estimation by, e.g., learning predictors to ground the logical state [16] ortraining recurrent models to predict the feasibility of a task plan [17].
These learned policies operatedirectly on perceptual data, and do not need hand-coded state estimation.
However, TAMP solversdeﬁne a highly non-linear mapping from observations to controls and this makes imitation difﬁcult. Furthermore, the compute resources required for planning limits the availability of training data forlearning.
In this paper, we build on both approaches.
We take inspiration from guided expert imitation [18, 19]and train feed-forward policies to imitate TAMP.
Our key insight is that Srivastava et al.
[20]’s modular TAMP framework supports a distributed TAMP architecture that: 1) is optimized for throughput(as opposed to low latency, the typical focus in TAMP research); 2) trains policies to imitate theplanner output; and 3) uses those policies to, in turn, amortize planning across solutions and reducecompute costs.
The result is a virtuous cycle where learning accelerates planning, and faster planning provides more supervision.
We propose a policy architecture that leverages the deﬁnition of the TAMP domain to improve learning performance.
Task plans from our system supervise a task-levelmodel that learns to predict a parameterized action.
This output selects from and parameterizes a set of control networks, one for each action schema.
We show that this architecture can executemanipulation skills in high-dimensional environments from complex sensory input.
As a result, oursystem is able to compile task and motion planning into a single feed-forward policy.
The contributions of this work are as follows: 1) we show a design for a distributed, asynchronous,high-throughput task and motion planning system that leverages policy learning to speed up planning; 2) we propose a hierarchical policy architecture that leverages the TAMP problem speciﬁcation; and 3) we implement and evaluate this method to show that it can learn to accomplish multiplegoals over (comparatively) long horizons with high-dimensional sensor data and action spaces.
In a2D pick-place task, we train policies that can place 3 objects precisely onto targets 83% of the timefrom RGB images.
In the RoboSuite [21] pick-place benchmark we train policies for joint controlthat can place all 4 objects 88% of the time from object pose vectors.
We reach 67% success on the RoboDesk [22] 9-goal multitask benchmark from RGB images.
Background
Task and Motion Planning.
Task and motion planning (TAMP) divides a robot control probleminto two components: a symbolic representation of actions (e.g., grasp ) and a geometric encodingof the world.
Task planning operates on a logical representation of the world.
It ﬁnds sequences ofabstract actions to accomplish a goal, e.g., pick( obj 1 ), place( obj 1 , targ 1 ), etc.
Each action encodesa motion problem that must be reﬁned (i.e., solved) to obtain a feasible trajectory that satisﬁesspeciﬁed constraints.
We represent TAMP problems with the formalism introduced by Hadﬁeld-Menell et al. A
TAMP problem is a tuple h X, F, G, U, f, x 0 , A i .
X is the space of valid world conﬁgurations.
F isa set of ﬂuents , binary functions of the world state that characterize the task space f : X ! { 0 , 1 } ,e.g. at( obj 3 , targ 2 ) or holding( obj 1 ).
G is the goal state, deﬁned as a conjunction of ﬂuents { g i } .
U is the control space of the robot.
f describes the world dynamics: f ( x t , u ) = x
t +1 .
x 0 describesthe initial world conﬁguration x 0 2 X .
Finally, A is the set of abstract action schemas .
Each actionschema a has four components: (1) a.params : the parameters of the action (e.g., which object tograsp); (2) a.pre : a set of parameter-dependent ﬂuents that deﬁnes the states when this action canbe taken; (3) a.mid : a set of ﬂuents that constrains the allowable controls for this action; and (4)a.post : a set of ﬂuents that will be true after the action is executed.
Solutions to such a problem are apair of sequences ( ~ a, ~⌧ ) , where ~ a encodes abstract actions and ~⌧ encodes reﬁned motion trajectories.
Modular TAMP.
Our approach parallelizes the modular TAMP approach of Srivastava et al.
They use a graph representation of TAMP, where nodes correspond to different task plans.
A nodecan be either reﬁned or expanded.
Reﬁnement uses motion planning to search for trajectories thataccomplish the actions in the plan.
Reﬁnement failures are tracked along with the unsatisﬁableconstraints that caused the failure.
When a node is expanded, one of these reﬁnement failures is usedto generate a new plan.
The error information (e.g., a collision with an obstacle) is used to updatethe abstract domain so that the planner can identify an alternative plan (e.g., one that moves theobstruction out of the way).
To solve a TAMP problem, Srivastava et al. [20] interleave reﬁnementand expansion to ﬁnd a plan that reaches the goal.
To implement plan reﬁnement, we use the sequential convex optimization method described in [23].
This formulates motion planning as an optimization problem.
The objective is a smoothing costk ⌧ k 2 =
[-τ (cid:107) 
 = (cid:80)-]P t k ⌧ t +1  ⌧ t k 2 .
The constraints are determined by the pre, post, and mid conditions 88 of an action schema: ⌧✓ a.pre , ⌧ T ✓ a.post , and ⌧ ✓ a.mid .
E.g., for a grasp action thepreconditions constrain the initial state to position the gripper near the object with proper orientation.
The postconditions constrain the ﬁnal state so that the object is in a valid grasp.
The midconditionsconstrain the intermediate states to avoid collisions.
If this optimization fails, we use the unsatisﬁedconstraints to expand the associated plan node, as described above.
Method
This section has three parts.
First, we propose a distributed TAMP solver.
Then, we describe ahierarchical policy architecture that leverages the TAMP domain description to make it easier tomodel TAMP behavior.
Finally, we show how to incorporate feedback from the learned policies toprevent trajectory drift.
3.1 Distributed Planning and Training Architecture
Figure 2 shows our overall design.
Our system has four types of nodes that operate asynchronously:(1) policy training, (2) task planning, (3) motion planning, (4) structured exploration.
The task-levelpolicy ⇡ hi and the motion-level policy ⇡ lo train within their respective nodes without backpropagation between the two.
For this work we found standard supervised learning sufﬁcient for goodperformance, however the system’s modularity makes it straightforward to apply more complex procedures (e.g. generative adversarial imitation learning [24] or inverse reinforcement learning [25]).
Speciﬁc network architectures and hyperparameters are included in the Supplemental Materials.
Algorithm 1 outlines our procedure for task planning.
This node reads from a shared priority queue
Q task that tracks task planning problems, represented as tuples of: an initial state x 0 , a logical state 0 , a goal g , and a reﬁned trajectory preﬁx ⌧ 0 .
At random intervals, or when the queue is empty, wesample a new planning problem from the base problem distribution, with an empty trajectory preﬁx⌧ 0 = ; .
A symbolic planner, such as FastFoward [26], computes a valid action sequence ~ a suchthat g ⇢ ~ a (  ) and pushes the problem ( x 0 ,  0 , g, ⌧ 0 ) and action sequence ~ a to the motion queue Q motion .
Algorithm 2 outlines our procedure for motion planning.
This node pulls from Q motion .
It uses thereﬁnement procedure outlined in section 2 to reﬁne each action a i into a valid trajectory segment⌧ i .
If reﬁnement fails, it computes the set of unsatisﬁable constraints { e i } that describe why themotion planner could not reﬁne a i , appends these to the original logical description  0 , and pushesthe updated problem instance ( x 0 ,  0 [ { e i } , g, ⌧ 0: i  1 ) to Q task .
If reﬁnement succeeds, each ⌧ iis pushed to the motion dataset D motion to supervise the appropriate control policy.
When the goalis reached (i.e., ⌧ T ⇢ g ), it pushes the action sequence ~ a to the task dataset D task to supervise thetask-level policy.
3.2 Hierarchical Task and Motion Policies
Learning to imitate TAMP solutions with a single policy is difﬁcult because small changes in theobservation can lead to different task plans and, thus, radically different controls.
We deal with thisthrough a hierarchical policy architecture that mirrors the TAMP domain description.
The policyis split into two parts: a task-level policy ⇡ hi to select parameterized actions a t and a motion-levelpolicy ⇡ lo to select controls u t as a function of a t and world state.
Figure 1 illustrates our design.
Require:
Shared queues Q task , Q motion
Require: Problem distribution P 1: while not terminated do 2: if is empty( Q task ) then 3: ( x 0 ,  , g , ⌧ ) ⇠ P prob 4: else 5: ( x 0 ,  , g , ⌧ ) pop ( Q task ) 6: ~ a task plan (  , g ) 7: push( Q motion , ( x 0 ,  , g, ~ a ))
Require:
Shared queues Q motion , Q task
Require: Expert datasets D motion , D task
Require: Motion policy ⇡ motion 1: while not terminated do 2: ( x ,  , g, ⌧ , ~ a )
Q motion 3: for a i 2 ~ a do 4: ˆ ⌧ i rollout  x , ⇡ lo ( ⇤ | a i )  5: ⌧ i , success motion plan  x, a i , ˆ ⌧ i  6: if success then 7: x ⌧ i [  1] 8: append( ⌧ , ⌧ i ) 9: push( D motion , ( a i , ⌧ i ) 10: else 11: ## add unsatisﬁable constraints { e i } 12: push ( Q task , ( x,  [ { e i } , g , ⌧ )) 13: break 14: push( D task , ( ⌧ , ~ a, g))
⇡
hi maps continuous observations and goalsto the discrete space of parameterized actionschemas.
We use the factored encoding schemefrom Van et al.
⇡ hi outputs a sequence ofvectors.
The ﬁrst speciﬁes a one-hot encodingof the choice of action-type (e.g. place).
Eachsubsequent vector provides a one-hot encodingof the ordered action parameters (e.g. obj 1 ,targ 2 ).
Training data for this policy is pulledfrom the task dataset D task .
The low-level policy ⇡ lo consists of two stages:an attention module and action-speciﬁc controlnetworks.
The attention module maps the continuous observation and discrete action parameters to a continuous parameterization.
This stepis ﬂexible.
In our system, it represents the relevant objects in a particular geometric frame.
E.g., for the action place ( obj 1 , targ 2) this isthe pose of obj 1 in the targ 2 frame.
When thepolicy has access to state data, we hard codethis step.
In other situations, this is learnedfrom supervision alongside the control policy.
The abstract action, the continuous parameterization, and the original observation pass to thesecond stage to predict the next control.
Thisstage contains a set of separate control
3.3 Policy-Aware Supervision
A common challenge in imitation learning is that small deviations from training supervision buildup over time [27].
To account for this, we propose task and motion supervision methods basedon Dataset Aggregation Ross et al. For controls, we bias trajectory optimization to providesupervision near states encountered by the motion policies. After an initial training period, wewarm-start optimization with a rollout from the appropriate motion policy.
Then, to keep supervisionclose to this trajectory, we modify the optimization objective with a rollout deviation cost , based onthe acceleration kernel from Dragan et al. This uses a generalization of Dynamic Movement
Primitives [30] that adapts the rollout trajectory to satisfy optimization constraints. For a sampledtrajectory ˆ ⌧ i , the optimization cost is k ˆ ⌧ i  ⌧ i k 2 =
[Equation]
This is 166 conceptually similar to the regularization penalty from Guided Policy Search [18]. To generate task-level supervision, the exploration node identiﬁes states where the task policy generates invalid actions.
We sample a problem from the problem distribution and rollout the combinedpolicy with an execution monitor that ensures that high-level action a is only started when a.pre aresatisﬁed and is executed until a.post are satisﬁed. We use this to build up a dataset of negative examples for the task policy and to identify states where TAMP supervision may improve performance.
Details and pseudo-code are in the Supplemental Materials.
[-τ i − τ opt (cid:107) 
-]
Experimental Results
We evaluate our system in three domains: a 2D pick-place simulator, the RoboSuite benchmark[21], and the RoboDesk benchmark [22].
Screenshots of the simulation environments are shownin Figure 1.
We used FastForward [26] as the task planner.
Unless otherwise stated, evaluationsused 2 processes for task planning, 18 for motion planning, and, when applicable, 10 for supervisedexploration.
Experiments ran for 5 random seeds unless otherwise noted.
Policy architecture detailsand hyperparameters are in the Supplemental Materials.
4.1 2D Pick-Place Experiments
Setup.
We begin with a simple, synthetic, pick-place domain.
The robot is circular with a parallel jaw gripper and its goal is to transfer up to 5 objects each to one of 8 randomly assignedtargets.
The domain has three action schemas: moveto-and-grasp ( obj ) , transfer ( obj, targ ) , placeand-retreat ( obj, targ ) .
The full space includes a grounding of these for each object and target.
With 5 objects and 8 targets, this gives 85 possible actions.
The control space outputs x, y,and rotational velocities on the robot body and an open/close gripper signal.
We used DMControl [31] for the underlying physics engine.
The goal requires each object overlap with its target (i.e.|| pos obj  pos targ
|| 2  radius obj ).
We use a simulated LIDAR sensor with 39 distance readings to 189 facilitate collision avoidance.
Manipulated Variables and Dependent Measures.
Our primary variable is the training procedureused.
We compare against: 1) ﬂat imitation learning (ﬂat-IL), a single feedforward network trainedto imitate TAMP output; 2) reinforcement learning with a dense reward function, trained with doublethe compute of our method (60 vCPUs); and 3) passive imitation learning (no-feedback), which usesthe hierarchical architecture but turns off policy-aware supervision (20 vCPUs).
We used two RLmethods: PPO [32], a ﬂat RL method, and HIRO [33], a hierarchical method.
See the Supplemental
Materials for training details.
We used two performance measures: 1) success rate (Succ.), the rateat which trained policies achieve the goal; and 2) distance reduced (Dist), the percent reduction inthe distance from each object to its target.
Results.
Table 1 shows the results of these comparisons.
With 1 object, we observed a 12% successrate for PPO and a 1% success rate for HIRO.
Flat-IL succeeded 21% of the time in the 1 objectvariant.
With two objects, it reliably reduced distance to the goal, but only succeeded 1.5% of the time.
With 5 objects, we observed an 80% success rate for the hierarchical policy withoutfeedback.
This highlights the strong inductive bias our hierarchical architecture provides.
Withfeedback, we observed success rates of 99.7%, 98.7%, and 88% for 1, 2, and 5 objects. With 5objects, performance plateaued with approximately 3 . 6 ⇥ 10 5 environment transitions of supervision,generated by 1600 successful plans over 2 hours.
See Figure 3 (left) for a depiction of performanceover the course of training. [Figure] [Table]
Camera Observations.
We next investigated the ability of our approach to trainpolicies from RGB data.
Figure 3 showsperformance over time for solving problems with 1, 2, and 3 objects.
After 4hours of training (120 cpu hours) our policy successfully solves the 3 object task83% of the time.
In all cases, performance plateaued with less than 3 . 4 ⇥ 10 5timesteps of supervision from the planner. Generalization. We tested the learnedpolicies’ ability to generalize to novelgoals and dynamics at test time.
First, weintroduced 6 additional obstacles to represent ‘humans’ in the environment.
The policy observations were the same as before, so the robot hadto use the LIDAR sensor to avoid the humans.
At training time the humans stayed in ﬁxed, randomlocations. At test time, the humans use model predictive control to take actions that move towardsa random goal location.
Their objective penalized collisions with other humans, but not the robot.
Rollouts terminated if a human collided with the robot.
We observed a 50% success rate placing 3objects in this condition.
Next, we moved the target locations for objects from training to test time. We arranged the targets in a square at the center of the free space at training time. At test time, wemoved the target locations to the edges of the space.
This placed all targets outside the region seenduring training.
We observed success rates of 99%, 97%, and 39% for problems with 1, 2, and 4objects, respectively.
4.2 RoboSuite Experiments
Setup.
We measured performance in the RoboSuite pick-place benchmark with a simulated 7-DoF Sawyer arm [21].
The goal is to move a cereal box, milk carton, soda can, and loaf of bread fromrandom initial locations in a bin on the left to its target bin on the right. The world state is thejoint conﬁguration and the positions of the objects.
The action space speciﬁes velocities for the7 joints and a binary open/close signal for the gripper.
The symbolic domain has 16 high-levelactions across 4 action schemas: moveto ( obj ) , grasp ( obj ) , moveholding ( obj ) , and putdown ( obj ) .
The policy observations were joint angles, gripper state, the translation and rotational displacementsfrom the gripper to each object, and the displacement from each object to its target.
Our mainexperiment had 4 conditions: 1) can-only, to compare with prior work; 2) cereal-and-milk, the most difﬁcult 2 object condition; 3) all, the standard benchmark goal; and 4) all-noisy, which added
Gaussian observation noise to the policy observations with standard deviation equal to 1% of thepossible deviation in initial values.
We performed an ablation study in the cereal-and-milk conditionto compare performance across feedback types: 1) no-feedback; 2) task-feedback (only); 3) motionfeedback (only); and 4) tamp-feedback (i.e., both feedback types).
Dependent Measures.
To compare with published results in RL, we report the average dense returnof the trained policies over 500-step horizons for can-only. Our primary measure of performanceis success rate. We also measured the number of timesteps taken to reach the goal on successfultrajectories.
For the ablation study, we tracked the amount of time spent during motion optimizationcalls to measure the speedup from feedback. Comparison to Prior Work.
First, we compare against published results from RL and LfD. Theofﬁcial benchmark reports returns near 50 when using Soft-Actor Critic to move the can [21, 34]. In comparison, we observed an average return of 324 for this task.
Fan et al.
[35] report an averagereturn below 600 over a 2000-step horizon. Our return for this setting can be lower bounded as 1734.
We compare with LfD results on success rate. Mandlekar et al.
[36] trained from the RoboTurk cansdataset [37] and observed success rates of 31% from state data and 43% from RGB camera images.
In comparison, we observed a 98% success rate from state data for the can-only condition of ourexperiments. Results.
Figure 3 (center) shows success rates over time for the four conditions described above.
Our learned policies reach an 88% success rate in the all condition, 90% in the cereal-and-milkcondition, and 98% in can-only condition.
These correspond to average (successful) rollout lengthsof 244, 120, and 55, respectively.
In the all-noisy condition, we observed a success rate of 64% ,compared with a 15% success rate executing plans from the TAMP solver directly. Figure 4 (center)shows success rate against training time for the ablation study. Both motion-feedback and tampfeedback reach 80% success within 2 hours. Task-feedback shows steady improvement but does notreach the performance of the other variants within the time limit.
(See the Supplemental Materialsfor a non-stationary condition where task-feedback improves on motion-feedback.) The no-feedbackcondition does not get above 20% success. We observe up to a 2x speedup of the motion optimizationcode with tamp-feedback compared to no-feedback. Figure 4 (left) illustrates the results.
Generalization. We ran a followup experiment to test the ability of the learned policies to generalizeto unseen goals and dynamics. First, we held out goals that moved 3 and 4 objects and tested withother sizes of goals. When trained on problems that move 1 and 2 objects, the system was able tosolve 3 object problems 48% of the time.
When trained on 1, 2, and 4 object goals, it was ableto solve 3 object problems 58% of the time. Next, we trained on 1, 2, and 3 object problems andobserved a 60% success rate when tested with a goal to move all 4 objects.
Note that this condition,which learns a multitask policy, is harder than the RoboSuite benchmark, which only attempts tolearn a single-task policy. Next, we used the domain randomization functionality of RoboSuite to test the ability of the previously trained policies to generalize to different dynamics. This varies properties of the physicsengine such as the inertia and mass of the robot and objects, the parameters of the solver for contact forces, position and quaternion offsets of bodies, and physical properties of the joints.
Usingthe default randomization parameters, we observed a success rate of 34%. When randomizing allparameters except the positional and quaternion parameters, we observed a 73% success rate.
4.3 RoboDesk Experiments
Background.
We completed the core of this work on May 19 th , with the above domains [38].
Our ﬁnal experiment anecdotally measures the engineering effort required to adapt the system toa previously unseen environment.
The RoboDesk multitask benchmark had been released 6 daysprior, on May 13 th [22].
The results in this section reﬂect one engineer’s effort over the course ofthree weeks.
The primary engineering steps were: 1) extension of existing action schemas to handlenew tasks; 2) integration of the existing system with the new simulator; and 3) iteration on motionspeciﬁcation and ML hyperparameters.
Setup.
RoboDesk provides a 7 DoF PANDA arm with 9 goals: open slide, press button, lift block,open drawer, block in bin, block off table, lift ball, stack blocks, and block in shelf.
We added acomposite goal: block in shelf with door closed.
Our observations are RGB camera images and joint positions.
The controls are on joint velocities and the gripper.
Our domain has 7 actionschemas: moveto ( obj ) , lift ( obj ) , stack ( obj 1 , obj 2) , place ( obj, targ ) , press ( button ) , open ( door ) ,close ( door ) .
This gives 31 possible high-level actions.
Results.
Figure 3 shows the results of training for 4 hours on an Nvidia DGX Station (40 vCPUs, {+
+}
GPUs).
The system learned to complete both tasks with an average success rate of 67% .
We reportthe benchmark performance for each goal in the Supplemental Material.
The system performedworst (32% success) for the lift ball task, which requires especially precise motion.
We observed a2.6x reduction in motion optimization time from feedback and report the results in Figure 4 (left).
Related Work
Speeding up TAMP.
There is a growing ﬁeld of work that uses machine learning on previous experience to reduce planning for TAMP systems.
Much of this work focuses on guiding the task-levelsearch — something we do not explicitly address in our system.
Chitnis et al.
[15] trained linearheuristics from expert demonstrations.
Kim et al.
[11] used a score-space representation to guidethe search by transferring knowledge from previous plans.
Wells et al.
[10] learned a classiﬁer toassess motion feasibility and incorporated it as a heuristic into the search.
At the motion level, Ichnowski et al.
[13] also learned policies to warm-start trajectory optimization.
The key differencesare that they predicted full trajectories, rather than executing the policy, and they did not penalizeoptimization for deviating from the policy controls.
We note that our focus is on the learned policies themselves, and so we do not directly compare the speedup produced by our system to othermethods.
Imitating TAMP.
Several other methods distill all or part of TAMP into learned policies.
Paxtonet al.
[14] used imitation learning to train policies in an abstract task space and deep q-learningin a control space.
Like our system, they learned feed-forward policies for both task and motionprediction and used those policies to guide supervision.
The key difference is the type of plannerused.
Ours is more compute intensive, but scales to larger problems.
Kase et al.
[16] proposed amethod to imitate hand-engineered trajectories to train a model to predict logical state and controllersto execute actions.
Their method does not directly predict tasks, but allows a TAMP system to runonline.
In contrast, we train a single policy that maps observations directly to controls.
The approach most similar to ours is that of Driess et al.
They also train hierarchical policiesto match TAMP output. They show that their system is able to integrate geometric and high-levelreasoning for a reaching task that, depending on the position of the target, potentially requires theuse of tools. The primary difference is that their approach predicts the feasibility of a given actionsequence, based on the initial state, while we predict actions and controls given the current state. This means that they need to search over candidate high-level action sequences at test time.
In oursystem, this search is handled by the task planning node during training (and bypassed entirely attest time).
The other large difference is that they represent their control policies with learned energyfunctions that are minimized online. Task transitions occur when this system reaches an equilibriumstate. In contrast, we learn policies that directly choose when to transition between tasks.
Future Work
First, this system should be deployed on physical robots. We hope that our policies will transfer morereadily than those produced, e.g., by RL because TAMP solutions do not rely on detailed dynamicsmodels.
Next, it would be interesting to experiment with parameter sharing for the motion policiesto save space and potentially speed up training.
It would also be interesting to experiment with theoutput space of the high-level policy to directly output continuous parameters, which may allowpolicies to generalize more effectively. There is likely room to improve on both the task and motionpolicy learning: sequence modelling methods from NLP [39] may be better suited to represent taskplans and modern imitation learning procedures, such as GAIL [24] or related works, may reducetraining time or improve generalization.
Note that each of these changes is relatively straightforwardto implement, as it only modiﬁes a single node of the training architecture. Finally, it would beinteresting to identify theoretical guarantees on the learned policies. E.g., it may be possible toformalize this as optimizing policy parameters with respect to a mixed discrete-continuous costfunction deﬁned by the TAMP domain.