Learning Neuro-Symbolic Skills for Bilevel Planning
Abstract: Decision-making is challenging in robotics environments with continuous object-centric states, continuous actions, long horizons, and sparse feedback.
Hierarchical approaches, such as task and motion planning (TAMP), address these challenges by decomposing decision-making into two or more levels of abstraction.
In a setting where demonstrations and symbolic predicates are given, prior work has shown how to learn symbolic operators and neural samplers for TAMP with manually designed parameterized policies.
Our main contribution is a method for learning parameterized polices in combination with operators and samplers.
These components are packaged into modular neurosymbolic skills and sequenced together with search-then-sample TAMP to solve new tasks.
In experiments in four robotics domains, we show that our approach — bilevel planning with neuro-symbolic skills — can solve a wide range of tasks with varying initial states, goals, and objects, outperforming six baselines and ablations. Video: https://youtu.be/PbFZP8rPuGg Code: https: //tinyurl.com/skill-learning
Introduction
Decision-making in robotics environments with continuous state and action spaces is especially challenging when tasks have long horizons and goal-based objectives.
A common strategy is to decompose decision-making into a high level (“what to do”) and a low level (“how to do it”)
Seminal work by Konidaris et al.
[4] considers such a hierarchy where symbolic AI planning [5] is used to sequence together temporally-extended continuous skills [6].
Their main contribution is a method for learning symbols from known skills.
In this work, we study the inverse: learning skills from known symbols [7, 8, 9, 10, 11].
We are motivated by cases where it is easier to design [12, 13, 14, 15, 16] or learn [17, 18, 19, 20, 21] symbols than it is to design skills.
In the long term, we envision a continually learning robot that uses symbols to learn skills and vice versa.
We consider symbols in the form of predicates — discrete relations between objects.
A set of predicates induces a state abstraction [3, 22] of a continuous object-centric state space.
For example, consider the Stick Button environment (Figure 1, third column), where a robot must press buttons either with its gripper, or by using a stick as a tool.
Predicates include Grasped and Pressed , inducing an abstract state such as { Grasped(robot,stick) , Pressed(button1) , . . . } .
We leverage symbolic predicates to learn skills that generalize substantially from limited demonstration data.
Concretely, we learn skills that achieve certain symbolic effects (e.g., Grasped(robot, stick) ) from states where certain symbolic preconditions hold (e.g., HandEmpty(robot) ).
Symbolic scaffolding for skill learning has two major beneﬁts.
First, by drawing on AI planning techniques, we can efﬁciently chain together long sequences of learned skills [4, 5, 10].
This beneﬁt is key in our setting where planning in the low-level transition space is practically infeasible, even though a black-box transition function is deterministic and known — actions are continuous, solutions can exceed 100 actions, and there is no extrinsic feedback available before a task goal is reached.
Second, because each skill acts in a conﬁned region of the state space, learning is easier than it would be for a monolithic goal-conditioned policy (e.g., with behavioral cloning [23]).
While symbols offer useful structure for skill learning and planning, they also present challenges.
In particular, symbolic state abstractions are often lossy [12, 13, 24], in that they discard information that may be important for decision-making, like the reachability of buttons or the relative stick grasp in Stick Button.
This lossiness can be mitigated to some degree by inventing new predicates [4, 21], but in robotics environments, there are often kinematic and geometric constraints that are hard to perfectly abstract away [1, 25, 26].
These difﬁculties lead us to the following key desiderata.
Key Desideratum 1 (KD1).
Because the abstraction is lossy, some environment states that correspond to the same symbolic state may be better than others in terms of completing a task.
Thus, a skill should be able to reach many different environment states (“subgoals”) that correspond to the same abstract state.
For example, a skill that achieves Grasped(robot, stick) should not always lead to the same relative grasp: the grasp should be lower when buttons are farther away, and higher if collisions with the gray stick holder would prevent a bottom grasp.
This desideratum implies that lookahead may be necessary, where the choice of skill early in the plan should be inﬂuenced by soft or hard constraints later in the plan.
It also implies that we should not rely on the (abstract) subgoal condition in the framework of Konidaris et al.
Key Desideratum 2 (KD2).
When the state abstraction is lossy, it is not always possible to take actions in the environment that correspond to a particular abstract state sequence.
Thus, an agent should be able to consider multiple skill sequences that reach the same goal from the same initial abstract state.
For example, without a predicate that reﬂects button reachability, the agent may ﬁrst plan to press all buttons without the stick, but if some buttons are unreachable, it will need the stick.
To address these two key desiderata, we propose an approach for learning and planning with neurosymbolic skills .
A neuro-symbolic skill consists of a symbolic operator [27], a neural subgoalconditioned policy, and a neural subgoal sampler [28, 29].
The subgoals produced by the sampler and consumed by the policy correspond to different environment states that are consistent with the abstract effects of the operator, addressing KD1.
The symbolic operators enable a bilevel planning algorithm [30, 31, 32], with AI planning in an outer loop and sampling in an inner loop.
If sampling fails for an operator sequence, the outer loop continues to another sequence, addressing KD2.
Skills are learned from demonstrations without any prior knowledge about the number of skills.
Main Contributions.
Prior works by Silver et al.
[12] and Chitnis et al.
[13] propose a pipeline for learning operators and samplers for bilevel planning when given a set of parameterized policies 1 .
While some policies may be human-designed and generally reusable, e.g., navigation policies that use motion planners, others are more domain-speciﬁc, such as pouring a pot of coffee or opening a door (Figure 1).
Our main contribution is a method for learning these policies to complete the neurosymbolic skills.
We conduct experiments in four robotics environments (Figure 1) that require a wide range of behaviors, such as pouring, opening, pressing, picking, and placing.
We ﬁnd that skills learned from 100–250 demonstrations generalize to novel states, goals, and objects.
Furthermore, our approach — bilevel planning with neuro-symbolic skills (BPNS) — outperforms six baselines and ablations, including learning graph neural network-based metacontrollers [15, 33].
Problem Setting
We consider the problem of learning from demonstrations in deterministic, fully-observed environments with object-centric states, continuous actions, and a known transition function.
An environment is characterized by a tuple
h
⇤ , U , f, i .
An object type  2 ⇤ has a name (e.g., button , robot ) and a tuple of real-valued features (e.g., ( x , y , z , radius , color , ...)) of dimension dim (  ) .
An object o 2 O has a name (e.g., button1 ) and a type, denoted type ( o ) 2 ⇤ .
A state x 2 X is an assignment of objects to feature vectors, that is, x ( o ) 2 R dim ( type ( o )) for o 2 O .
The action space is denoted U ✓ R m .
The transition function is a known, deterministic mapping from a state and action to a next state, denoted f : X ⇥ U !
X .
A predicate has a name (e.g., Touching ) and a tuple of types (e.g., ( stick , button )).
A ground atom is a predicate and a mapping from its type tuple to objects (e.g., Touching ( stick , button1 )).
A lifted atom instead has a mapping to typed variables, which are placeholders for objects (e.g., Touching(?s, ?b) ).
Predicates induce a state abstraction: abstract ( x ) denotes the set of ground atoms that hold true in x , with all others assumed false.
We use s 2 S to denote an abstract state, i.e., abstract : X ! S .
An environment is associated with a task distribution T , where each T 2 T is characterized by a tuple
h
O , x 0 , g, H i .
The set of objects in the task is denoted O .
Importantly, objects vary between tasks.
The initial state of the task is denoted x2 X .
The goal, a set of ground atoms with predicates in and objects in O , is denoted g .
A goal represents a set of abstract states: for example, there are many possible abstract states where g = { Pressed(button2) } holds.
The task horizon, the limit for how many actions can be taken in a task, is denoted H 2 Z +
A solution to a task is a sequence of actions that achieves the goal and does not exceed the task horizon, i.e. u = ( u 1 , . . .
, u k ) such that k  H , g ✓ abstract ( x k ) , and x i = f
( x i  1 , u i ) for 1  i  k .
We consider a standard learning setting where train tasks drawn from from T are available at training time, and held-out evaluation tasks drawn from T are used for evaluation.
Our objective is to maximize the number of evaluation tasks solved within a wall-clock timeout.
Each train task T is associated with one demonstration h T, u, x i , where u is a solution for T and x = ( x 0 , . . .
, x k ) is the sequence of states visited.
In experiments, tasks are randomly sampled, and demonstrations are generated with handcrafted environment-speciﬁc policies (but see (§C.4)).
Neuro-Symbolic Skills
How can we leverage the available demonstrations, predicates, and known transition model to maximize the number of evaluation tasks solved before timeout?
We propose to learn and plan with neuro-symbolic skills.
In this section, we deﬁne these skills formally; in (§4), we discuss planning; and in (§5), we address learning.
See Figure 2 for a summary of the architecture.
A skill is a tuple  = h v, ! , ⇡ ,  i where v is a tuple of arguments ; !
is an operator ; ⇡ is a subgoalconditioned policy ; and  is a subgoal sampler .
A set of skills is denoted  .
Arguments are variables with types in ⇤ and are shared by ! , ⇡ , and  .
An operator is a tuple !
=
h v, P, E + , E  i where
P, E + , E  are preconditions , add effects , and delete effects respectively, each a set of lifted atoms over the predicates and arguments v .
A subgoal-conditioned policy is a mapping ⇡ : O | v | ⇥ X ⇥
X !
U that takes as input a tuple of objects for the arguments v , the current state x , and a subgoal state x 0 , and maps it to an action u .
A subgoal sampler is a generator of elements from a distribution of subgoal states conditioned on object arguments and a current state, that is,  : O | v | ⇥ X !  ( X ) . A ground skill is a skill with objects that match its arguments, denoted by a tuple  = h  , o i where o is a tuple of objects in
O with types matching v . (We use underlines to denote grounded entities.) The corresponding ground operator , ground policy , and ground sampler are deﬁned analogously, associated with objects match The contract between a skill’s operator, policy, and sampler is the following: in states where the preconditions of a ground skill hold, i.e., P ✓ abstract ( x ) , the sampler should generate a subgoal x 0 ⇠  ( x ) that satisﬁes the skill’s effects, i.e. F ( abstract ( x ) ,  ) =
abstract ( x 0 ) , and the actions proposed by the policy ⇡ ( · , x 0 ) should lead to x 0 from x .
In practice, we initiate a ground skill only when the ground operator preconditions hold, and terminate at corresponding abstract successor state (cf. options [6]).
Next, we show how to leverage this contract for efﬁcient planning.
Bilevel Planning with Neuro-Symbolic Skills
Given skills and a new task, we search for a solution with bilevel planning.
See Figure 3, Algorithm 1, and [13] for an extended description.
Bilevel planning begins with an outer loop (Line 2), where skill operators are used to generate up to N abstract abstract plans .
An abstract plan for task T =
h O , x 0 , g, H i is a sequence of ground skills that achieves the goal, i.e.  =
(  1 , . . .
,  ` ) such that s {+
+}
= abstract ( x 0 ) , g ✓ s ` , `  H + 1 , and s i
= F ( s i  1 ,  i ) for 1  i  ` .
plan( T,  , f ) //
Params: N abstract , H skill , N samples sabstract ( x 0 ) //
Uses operators for  in top-k
( s 0 , g ,  , N abstract ) do // Uses samplers and policies u reﬁne(  , T,  , f, H skill , N samples ) if u is not null then //
Planning succeeded return u //
Planning failed return null
Algorithm 1: Bilevel planning.
Much work in the AI planning literature is devoted to efﬁciently generating these abstract plans, with most state-of-the-art techniques building on heuristic search.
In experiments, we use A ⇤ search with the LMCut heuristic [34].
The AI planner details are not important for this work; we have also experimented with other planners and heuristics including
GBFS, K ⇤ [35], FF [36], and various conﬁgurations of the Fast Downward planner [5].
Unlike typical AI planning, we continue the search after the ﬁrst plan is found, generating a potentially inﬁnite sequence of candidate plans (cf. top-
k planning [35, 37, 38]). Given a candidate abstract plan, we use the samplers and policies to perform a backtracking search over action sequences (Line 3).
For each ground skill in the abstract plan, we use the ground sampler to produce a subgoal, and simulate the ground policy until either (1) the abstract transition is completed; or (2) a maximum skill horizon H skill is reached.
In case (1), we continue on to the next ground skill, or terminate if the goal is reached (Line 5), while in case (2) we backtrack up to N samples times per step.
If backtracking fails, we continue on to the next abstract plan 2 .
We now address the problem of learning skills  from a dataset of demonstrations D = { h T, u, x i } .
Recall that the data comprise raw sequences of states and actions for entire tasks.
We preprocess the data in three steps: segmenting trajectories temporally, partitioning the segments into skill datasets , and lifting the objects within each skill dataset to variables.
Then, for each skill dataset, we learn an operator, a subgoal-conditioned policy, and a subgoal sampler.
We detail these steps below.
This pipeline extends that of Chitnis et al.
[13], who assume parameterized policies are given.
5.1 Data Preprocessing
Segmentation.
For each demonstration h T, u, x i 2 D , we identify a set of switch points 1 < i 1 < · · · < i ` < | u | .
Each pair of consecutive switch points ( j, k ) induces a segment  =
h u j : k , x j  1: k i , where u j : k = ( u j , u j +1 , . . .
, u k  1 ) is a subsequence of actions from u and x j  1: k is deﬁned analogously.
Temporal segmentation is a much studied problem in the literature [40, 41, 42].
In this work, inspired by the notion of contact-based modes (e.g., [43]), we create a new switch point whenever there is a change in the truth-value of any contact-related ground atom (§A).
Intuitively, using more (fewer) predicates here instead would lead to ﬁner (coarser) temporal abstractions.
Partitioning.
Next, we partition the set of all segments into a collection of skill datasets .
We group segments into the same skill dataset if their abstract effects are equivalent up to object substitution.
Formally, given a segment  = h u j : k , x j  1: k i , let s =
abstract ( x j  1 ) and s 0 =
abstract ( x k  1 ) .
The effects of the segment are
[Equation]
The affected object set O e comprises all objects that appear in either e + or e  .
Two segments  1 ,  2 with effects h e +1 , e  1 i , h e +2 , e  2 i and affected objects O e 1 , O e 2 are equivalent (  1 ⌘  2 ) if there exists an injective mapping  :
O e 1 !
O esuch that  ( e +1 ) =
e +2 and  ( e  1 ) =
e  2 and where each object in O e 1 shares the type of the object it is mapped to in O e 2 .
The relation ⌘ induces the partition: each partition contains equivalent segments.
Lifting.
The segments in each skill dataset involve different objects.
When we perform lifting, we replace all the objects in a segment with variables that have the objects’ types.
For example, in Stick Button, the robot may use the stick to press button1 in some segment, and the robot may use the stick to press button2 in another segment.
After lifting, these objects would be replaced with ?robot , ?
stick , and ?button .
Formally, for each skill dataset  i , for an arbitrary segment  0 2  i with affected object set O e 0 , we create one placeholder variable v for each object in O e 0 .
We order these variables arbitrarily into a tuple v .
Using the injective mappings  described above, it is then possible to map the affected object set for any segment in  i to the variables v .
In addition to aligning segments with different objects, variables deﬁne the skill scope : when computing abstract transitions, subgoals, and actions, the skills will reference only the object states for those variables.
The ﬁnal output of preprocessing is a set of lifted skill datasets, i.e., skill datasets  i , variables v , and mappings from segment objects to variables that we will use in skill learning, described next.
5.2 Skill Learning
Operator Learning.
Following previous work [13, 18, 21, 44], we use a simple linear-time approach to learn symbolic operators.
We induce one operator from each lifted skill dataset  i .
The variables v constructed via lifting comprise the operator arguments.
The operator add and delete effects follow immediately from lifting a segment’s effects h e + , e  i : we replace all objects with the corresponding variables in v , to arrive at lifted atom sets h E + , E  i .
To compute preconditions, we use all common lifted atoms in the initial segment states.
For a segment  2  i with states x j  1: k , let s
= abstract ( x j  1 ) , and let s v  denote the corresponding lifted atom set, with all affected objects in s  replaced with variables in v , and with any ground atoms in s  involving objects not in v discarded.
The preconditions P are then the intersection of lifted atom sets for each segment, denoted P = [Equation]
[Equation] s [Equation] , completing the operator !
=
h v, P, [Equation] E  i .
Policy Learning.
Recall the responsibility of a ground policy ⇡ ( x, x 0 ) is to output an action u that will lead the agent closer to the subgoal x 0 from the state x .
At this point in the pipeline, for each skill, we have available a dataset of subgoals being achieved: the ﬁnal state of each segment in the skill dataset is a subgoal, and the preceding states and actions demonstrate behavior towards that subgoal.
We can therefore use these data for supervised learning of a subgoal-conditioned policy.
First, we use the skill scope to convert the segment states into ﬁxed-dimensional vectors by concate- nating the features of the objects in the scope together with the subgoal.
Formally, for each state x in each segment  2  i , we construct a vector x v = x ( v 1 )  · · ·  x ( v n ) , where v = ( v 1 , . .
. , v n ) , x ( v i ) =
x ( o i ) for the corresponding o i under lifting, and  denotes vector concatenation.
We then create a dataset of input-output pairs ( x vi  1  x vk  1 , u i ) for j  i < k in each segment with states x j  1: k , where x vk  1 is the segment subgoal.
We are now left with the problem of learning a policy from vector-based supervised data.
In this work, we use behavioral cloning with fully-connected neural networks (see (§B) for details), but many other choices are possible 3 .
At evaluation time, if the policy is grounded with objects o =
( o 1 , . . .
, o n ) , and queried in state x , then x ( o 1 )  · · ·  x ( o n ) is the state input to the neural network, while the subgoal input is generated by the ground sampler.
Sampler Learning.
The role of the subgoal sampler is to propose different speciﬁc states, among the inﬁnitely many consistent with the effects of the operator, for the policy to pursue.
This ﬂexibility is important for KD1 (§1).
To learn samplers, we can largely reuse the datasets from policy learning: for each ( x  y, u ) in the policy learning dataset, with y denoting the subgoal, we create a ( x, y ) inputoutput pair for sampler learning.
Following [13], we learn two neural networks for each sampler.
The ﬁrst network outputs the mean and diagonal covariance of a Gaussian distribution.
The second network is a binary classiﬁer that takes a candidate ( x, y ) sampled from the Gaussian as input and accepts or rejects.
These two networks are capable of modeling a richer class of distributions than the Gaussian alone, e.g., multimodal distributions.
See (§B) for architecture and training details 4 .
We make two modiﬁcations to the approach described above to promote generalization.
First, we use the relative subgoal x vk  1  x vi for each step i in the segment, rather than the absolute subgoal x vk  1 (cf.
goal relabelling [29, 46]).
At evaluation time, we derive the absolute subgoal from the relative subgoal when the policy is ﬁrst initialized, and update the relative subgoal after each transition.
Second, we detect and remove subgoal dimensions that are static across all data (e.g., object mass).
Limitations.
Our approach assumes that the given predicates comprise a useful state abstraction of the underlying state space with respect to the task distribution.
With random or meaningless predicates, we would not expect to outperform non-symbolic baselines.
However, previous work [12, 21] suggests that operator and sampler learning are robust to a limited degree to predicate ablation or addition; we expect the same for policy learning 5 .
We assume that effective behavior can be determined from the skill scope alone and do not learn skills that handle a variable number of objects. Our method would create a separate skill for each unique skill dataset arising from affected object sets of different sizes, which might be preferred in some scenarios (picking up one, vs. ten, vs. two-hundred coins off the ground) but not others (tying multiple sticks together with a piece of string).
Relational neural networks (e.g., GNNs) offer one possible path to address these two limitations.
Through the lens of parameterized policy learning [47, 48, 49, 50], subgoals are just one form of parameterization among many possible alternatives.
We also aspire to learn from human demonstrations on real robots; data efﬁciency results in the next section suggest that this is feasible.
Experimental Results
Our experiments address ﬁve questions about bilevel planning with neuro-symbolic skills (BPNS):
Q1 .
How many train tasks are required by BPNS to effectively solve held-out evaluation tasks? Q2 .
Does BPNS generalize to unseen numbers of objects?
Q3 .
Can BPNS learn skills to complement existing general-purpose skills?
Q4 .
How important is the ability to sample multiple subgoals per abstract plan step (KD1)?
Q5 .
How important is the ability to generate multiple candidate abstract plans (KD2)?
Environments.
We describe environments here at a high level, with details in (§A).
The Cover environment was introduced by [12, 13, 21].
A robot must pick and place blocks to cover target regions on a 1D line.
The robot can only initiate and release grasps in certain allowed regions.
For example, if the robot grasps a block on the left side, it may not be able to place it to completely cover a target, depending on the allowed regions.
Tasks have two blocks and two targets.
There is random variation in the initial poses of the blocks, targets, allowed regions, and robot gripper.
Doors is loosely inspired by the classic Four Rooms [6].
A robot must navigate to a target room while avoiding obstacles and opening doors.
Doors have handles that require different rotations to open.
In this environment, to address ( Q3 ), a skill the conﬁgurations a motion planner (BiRRT) is provided; door opening must be learned.
Tasks have –25 rooms with random connectivity.
The goals, obstacles, initial robot pose, and door parameters also
The Stick Button environment is described in (§1).
Train tasks have 1–2 buttons and evaluation have vary in the initial poses of the stick, , robot,
In Coffee , a robot must move a pot of coffee to a hot plate, turn on the hot plate by pressing a button, and then move to pour the coffee into cups.
The cups have different sizes and require different amounts of coffee.
Depending on its orientation, the pot may need to be rotated before the handle can be grasped; the orientation is not reﬂected in the predicates.
Overﬁlling a cup or pouring from too far results in a spill (failure).
Train tasks have 1–2 cups and evaluation tasks have 2–3 cups
Tasks vary in the poses of the robot, pot, and cups, and in cup sizes.
The goal is to ﬁll all cups.
Approaches.
We brieﬂy
describe
the
approaches
we evaluate with details in (§B).
• BPNS :
Our main approach, bilevel planning with learned neuro-symbolic skills.
• BPNS No Subgoal : Our main approach, but with no samplers, and with subgoal-conditioned policies replaced with regular policies ⇡ : X !
U .
This approach is inspired by [7, 9, 51]. • GNN Meta : Instead of AI planning, this approach learns a graph neural network (GNN) metacontroller [9, 15, 33] that outputs a next skill based on the current state, abstract state, and goal. • GNN Meta No Subgoal :
Same as GNN Meta, but without samplers, and with regular policies.
This baseline is the closest to existing methods in the AI planning literature [9, 10, 51, 52]. • GNN BC : Learns a monolithic goal-conditioned GNN policy by behavioral cloning (BC). • Samples=1 : An ablation of BPNS where N samples = 1 during planning.
• Abstract Plans=1 :
An ablation of BPNS where N abstract = 1 during planning.
All approaches are trained with the same demonstration data and use the same predicates.
All approaches also use the transition function f , except for GNN BC, which is model-free.
Experimental Setup.
For all environments and approaches, we use 50 evaluation tasks andrandom seeds.
The timeout for each evaluation task is 300 seconds and the horizon H = 1000 .
Experiments were run on Ubuntu 18.04 using 4 CPU cores of an Intel Xeon Platinum 8260 processor.
Results and Analysis.
Figure 4 shows performance as a function of demonstration count.
BPNS performs well in all environments after 100–250 demonstrations ( Q1 ).
Stick Button is the most challenging because many considered abstract plans are infeasible, which can lead to timeouts.
The strong performance in Stick Button and Coffee, which have more objects than seen during training, also allows us to afﬁrmatively resolve ( Q2 ).
Strong performance in Doors, where motion planners are provided, conﬁrms that BPNS is capable of incorporating general-purpose skills ( Q3 ).
The importance of generating multiple samples per abstract plan step ( Q4 ) is reﬂected in the No
Subgoal baselines and the Samples=1 ablation.
The ablation performs worse than BPNS in all environments, while the No Subgoal baselines perform worse in all except Doors with 1000 demonstrations.
We found evidence for two explanations in our analysis.
First, skill policies may fail to terminate with certain parameters, but succeed with others.
Second, even if a policy terminates, the agent may or may not be able to ﬁnish the rest of the task from that subgoal.
These results conﬁrm that KD1 (§1) is important for the strong performance of BPNS.
To analyze the importance of generating multiple candidate abstract plans ( Q5 ), we can examine the GNN Meta baselines and the Abstract Plans=1 ablation.
These approaches perform well in
Cover and Doors, where the ﬁrst abstract plan is usually reﬁnable into a solution.
In Coffee, the ﬁrst abstract plan does not rotate the pot, and therefore fails when the handle is out of reach.
See Figurefor an example of failure in Stick Button.
These results conﬁrm the importance of KD2 (§1).
[Table]
We also compare BPNS with GNN BC, a purely modelfree approach.
The table on the right reports the mean (standard deviation) percent of evaluation tasks solved per environment after training with 1000 demonstrations.
We also experimented with fewer demonstrations for GNN BC and saw consistently poor performance.
The failure of GNN BC is unsurprising given the limited data and the high degree of task variability.
In the appendix, we present additional results that investigate the planning time of BPNS (§C.1), the poor performance of GNN Meta (§C.2), comparison to [12, 13] (§C.3), learning from human demonstrations (§C.4), the impact of irrelevant predicates (§C.5) or objects (§C.6), the ﬁltering of low-data skills (§C.7), comparison to oracle skills (§C.8), the structure of the learned operators (§C.9), and skill learning with invented predicates (§C.10).
Related Work
Our work contributes to a long line of research in skill learning for robotics, with prior works considering both reinforcement learning [53, 54, 55, 56, 57, 58, 59, 60, 61, 62] and learning from demonstrations [8, 40, 63, 64, 65, 66, 67, 68, 33].
In the context of hierarchical RL [28, 29, 69], it is possible to interpret our skill samplers as managers and policies as workers.
Our work is also related to parameterized skill learning [47, 70, 48, 71, 49, 50] if we view subgoals as parameters.
In the AI planning literature, prior works use a set of given planning operators to learn skills [7, 9, 10, 72, 73, 51, 52, 74].
Achterhold et al.
[75] pursue a similar approach in a robotics setting.
Recent work by Guan et al.
[15] is closest to our contribution in that they use given operators to learn skills with latent parameters that can be used to reach a diverse set of terminal states [61], and then learn a metacontoller over the skills.
The “GNN Meta” baseline is inspired by this work.
Instead of operators, recent works have also used pretrained language models to generate abstract “instructions” and then learn language-conditioned skills [68, 76, 77, 78].
Our work can also be viewed as learning components of a task and motion planning (TAMP) system.
In this line, other work has concentrated on learning operators [79, 80, 81, 82, 44, 12, 83], learning samplers [39, 84, 85, 86, 13, 87], or learning predicates [17, 88, 4, 89, 90, 91, 18, 19, 20, 92, 21].
Our operator and sampler learning algorithms are most directly related to that of Chitnis et al.
[13], who assume given parameterized policies.
Driess et al.
[26] use given operators to learn energy-based controllers from images in the context of constraint-based TAMP [43].
Wang et al.
[86] learn to sample continuously parameterized skills to accomplish symbolic operator effects, such as pouring at a certain angle to ﬁll a cup.
To the best of our knowledge, our work is the ﬁrst to learn operators, samplers, and policies for search-then-sample TAMP [30, 31, 32].
Conclusion
In this work, we proposed bilevel planning with learned neuro-symbolic skills as a framework for decision-making in object-centric robotics environments.
We conﬁrmed experimentally that this approach leads to strong generalization and data efﬁciency.
There are many interesting open questions for future work.
(1) Can neuro-symbolic skills be learned from invented predicates?
In (§C.10), we offer preliminary evidence suggesting the viability of this direction.
(2) Can neuro-symbolic skills be learned with reinforcement learning?
(3) Can latent object feature learning [16] be used to drive skill learning?
Answering these questions will help us leverage hierarchical planning at scale. Acknowledgements
We gratefully acknowledge support from NSF grant 1723381; from AFOSR grant FA9550-17-10165; from ONR grant N00014-18-1-2847; and from MIT-IBM Watson Lab.
Tom is supported by an
NSF Graduate Research Fellowship.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of our sponsors.
We thank Rohan Chitnis, Nishanth Kumar, Willie B. McClinton, Aidan Curtis, Will Shen, and Anish Athalye for helpful comments on an earlier draft