Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning
Abstract
In robotic domains, learning and planning are complicated by continuous state spaces, continuous action spaces, and long task horizons.
In this work, we address these challenges with Neuro-Symbolic Relational Transition Models (NSRTs), a novel class of models that are data-efﬁcient to learn, com- patible with powerful robotic planning methods, and generalizable over objects.
NSRTs have both symbolic and neural components, enabling a bilevel planning scheme where symbolic AI planning in an outer loop guides continuous
plan- ning with neural models in an inner loop.
Experiments in four robotic planning domains show that NSRTs can be learned af- ter only tens or hundreds of training episodes, and then used for fast planning in new tasks that require up to 60 actions and involve many more objects than were seen during training.
Introduction
For robots to plan effectively in the world, they need to contend with continuous state spaces, continuous action spaces and long task horizons (Figure 1, bottom row).
Symbolic AI planning techniques are able to solve tasks with very long horizons, but typically assume discrete, factored spaces (Helmert 2006).
Neural network-based approaches have shown promise in continuous spaces but scaling to long horizons remains challenging (Hafner et al. 2020;
Chua et al. 2018).
How can symbolic and neural planning methods be combined to overcome the limitations of each?
In this paper we propose a new model-based approach for learning and planning in deterministic, goal-based, multitask settings with continuous state and action spaces.
Following previous work, we assume that a small number of discrete predicates (named relations over objects) are given, having been implemented by a human engineer (Lyu et al.
; Illanes et al.
Wang et al. 2021), or learned from previous experience in similar domains.
These predicates induce discrete state abstractions of the continuous environment state (Sacerdoti 1974; Kokel et al. 2021).
For example, H OLDING ( block1 ) abstracts away the continuous pose with which block1 is held.
Even when given predicates, the question of how to make use of them to learn effective models for planning in continuous state and action spaces is a hard problem that this paper seeks to address.
From the predicates, and through sequential interaction with an environment, we aim to learn: (1) abstract actions , which deﬁne transitions between abstract states; (2) an abstract transition model , with symbolic preconditions and effects akin to AI planning operators; (3) a neural transition model over the low-level, continuous state and action spaces; and (4) a set of neural action samplers , which deﬁne how abstract actions can be reﬁned into continuous actions.
We unify all of these with a new class of models that we term the N euro- S ymbolic R elational T ransition Model (NSRT) (pronounced “insert”).
NSRTs have both symbolic and neural components; all components are relational, permitting generalization to tasks with any number of objects and allowing sample-efﬁcient learning.
To plan with NSRTs, we borrow techniques from searchthen-sample task and motion planning (TAMP)
(Garrett et al. 2021), with symbolic AI planning in an outer loop serving as guidance for continuous planning with neural models in an inner loop.
This bilevel strategy allows for fast planning in environments with continuous state and action spaces, while avoiding the downward reﬁnability assumption , which would assume planning can be decomposed into separate symbolic and continuous planning steps (Bacchus and Yang 1994).
When modeling robotic domains symbolically, the predicates are often lossy , meaning that downward reﬁnability cannot be assumed (Figure 1, top and middle).
This paper focuses on how to learn NSRTs and how to use NSRTs for planning in continuous-space, long-horizon tasks.
We show in four robotic planning domains, across both the PyBullet (Coumans and Bai 2016) and AI2THOR (Kolve et al.
2017) simulators, that NSRTs are extremely data-efﬁcient: they can be learned in only tens or hundreds of training episodes.
We also show that learned
NSRTs allow for fast planning on new tasks, with many more objects than during training and long horizons of up to 60 actions.
Baseline comparisons conﬁrm that integrated neuro-symbolic reasoning is key to these successes.
Related Work
Model-Based Reinforcement Learning (MBRL).
Our work is related to multi-task MBRL in that we learn and plan with transition models from data collected by environment interaction.
Many recent approaches to deep
MBRL learn unstructured neural transition models on a latent vector-space representation, and therefore must resort to highly undirected planning strategies like the crossentropy method (Hamrick et al. 2021; Hafner et al. 2020, 2021).
Some recent MBRL work has used more powerful planners, such as RRTs (Ichter, Sermanet, and Lynch 2020), LQR (Chebotar et al. 2017) and divide-and-conquer
MCTS (Parascandolo et al. 2020).
Relational MBRL is a subﬁeld of MBRL that uses relational learning (Dˇzeroski,
De Raedt and Driessens 2001; Tadepalli, Givan, and
Driessens 2004) to learn object-centric factored transition models (Battaglia et al. 2016; Chang et al. 2017; Kansky et al. 2017) or to discover STRIPS operator models (Xia et al. 2019; Lang, Toussaint, and Kersting 2012) when given a set of predicates.
Our work also learns relational transition models, but with a bilevel structure that allows planning without assuming downward reﬁnability.
Symbolic AI Planning for RL.
Our work continues a recent line of investigation that seeks to leverage symbolic
AI planners for continuous states and actions.
For example, previous work learns propositional (Dittadi, Drachmann, and Bolander 2020; Konidaris, Kaelbling, and Lozano-Perez 2018) or lifted (Arora et al. 2018; Chitnis et al. 2021;
Asai 2019; Ames, Thackston, and Konidaris 2018; Ahmetoglu et al. 2020) symbolic transition models, and uses them with
AI planners (Hoffmann 2001; Helmert 2006).
Other related work has used symbolic planners as managers in hierarchical RL, where low-level option policies are learned (Lyu et al.
2019; Sarathy et al.
Illanes et al.
Yang et al. 2018; Kokel et al. 2021).
This interface between symbolic planner and low-level policies assumes downward reﬁnability, a critical assumption we do not make in this work.
Learning for Hierarchical Planning.
Reasoning at multiple levels of abstraction is a key theme in hierarchical planning (Bercher, Alford, and H¨oller 2019).
Prior work has considered learning transition models that are compatible with hierarchical planners, including those based on mixed-integer nonlinear programming (Say et al. 2017; Say 2021) or hierarchical task networks (Nejati, Langley, and
Konik 2006; Zhuo et al. 2009).
Task and motion planning (TAMP) systems (Garrett et al. 2021) can plan effectively at long horizons (Kaelbling and Lozano-P´erez 2011; Srivastava et al. 2014), but they typically require hand-speciﬁed operators and action samplers, and a known low-level transition model.
While recent work can learn one of the ﬁrst two components (Loula et al. 2020; Silver et al. 2021; Wang et al. 2021; Chitnis et al. 2016;
Kim, Kaelbling, and LozanoP´erez 2018), our approach learns all three: the operators, the action samplers, and the low-level transition model.
Problem Setting
We study a deterministic, goal-based, multi-task setting with continuous object-oriented states, continuous actions, and a ﬁxed, given set of predicates.
Formally, we consider an environment h T , d, A , f, P i and a collection of tasks , each of which is a tuple h s 0 , g, H i .
Environments.
T is a set of object types, and d : T ! N deﬁnes the dimensionality of the real-valued attribute (feature) vector of each object type.
For example, an object of type box might have an attribute vector describing its current pose, side length, and color.
An environment state s is a mapping from a set of typed objects o to attribute vectors of dimension d ( o ) , where d ( o ) is shorthand for the dimension of the attribute vector of the type of object o .
We use S to denote this object-oriented state space.
The A ✓ R m is the environment action space.
The f : S ⇥ A ! S [ { fail } is a deterministic transition function mapping a state s 2 S and action a 2 A to either a next state in S or a special failure state fail , which can be used, e.g., to capture undesirable behavior such as causing a collision.
The transition function f is unknown; the agent only observes states through online interaction with the environment.
P is a set of predicates given to the agent.
A predicate is a named, binary-valued relation among some number of objects.
A ground atom applies a predicate to speciﬁc objects, such as A BOVE ( o 1 , o 2 ), where the predicate is A BOVE .
A lifted atom applies a predicate to typed placeholder variables: A BOVE (? a , ?
b ).
Taken together, the set of ground atoms that hold in a continuous state deﬁne a discrete state abstraction ; let A BSTRACT ( s ) denote the abstract state for state s 2 S , and let S " denote the abstract state space.
For instance, a state s where objects o 1 , o 2 , and o 3 are stacked may be represented by the abstract state
A BSTRACT ( s ) =
{ O N ( o 1 , o 2 ) , O N ( o 2 , o 3 ) } ; note that this abstract state loses details about the geometry of the scene.
Tasks and Objective.
A task h s 0 , g, H i is an initial state s 0 2 S , a goal g , and a maximum horizon H .
We will generally denote the set of objects in s 0 as O .
This object set
O is ﬁxed within a task, but changes between tasks.
Goals g are sets of ground atoms over the object set O , such as { O N ( o 3 , o 2 ), O N ( o 2 , o 1 ) } .
The agent interacts with the environment episodically .
An episode begins with a task’s initial state s 0 .
The agent takes actions sequentially, observing the state at each timestep.
If it encounters a state s for which g ✓ A BSTRACT ( s ) (i.e., the goal holds), the episode is solved .
An episode ﬁnishes when it is solved, when the failure state fail is reached, or after H timesteps.
We consider a set of training tasks and a set of test tasks ; test tasks have more objects and longer horizons, and are unknown to the agent during training.
The agent’s objective is to maximize the number of episodes solved over the test tasks.
Data Collection.
We focus on the problems of learning and planning .
To isolate these, we use a simple, ﬁxed strategy for data collection that makes use of a behavior prior ⇡ 0 ( · | s ) , a state-conditioned distribution over A .
Recent work has studied learning behavior priors (Ajay et al. 2021)
; we are assuming it is given, but it could be learned.
Datagathering proceeds by running ⇡ 0 on tasks sampled from the set of training tasks.
We do not use ⇡ 0 at test time.
Since ⇡is ﬁxed and given, our setting can be seen as model-based ofﬂine reinforcement learning (Levine et al. 2020).
NSRT Representation
The next three sections introduce Neuro-Symbolic Relational Transition Models (NSRTs).
In this section, we describe the NSRT representation; in Section 5, we address planning with NSRTs; and in Section 6, we discuss learning
NSRTs.
Figure 2 illustrates the full pipeline.
We want models that are learnable , plannable , and generalizable .
To that end, we propose the following deﬁnition:
Deﬁnition 1.
A Neuro-Symbolic Relational Transition
Model (NSRT) is a tuple h O, P, E, h, ⇡ i , where:
• O =
( o 1 , . . .
, o k ) is an ordered list of parameters ; each o i is a variable of some type from type set T .
• P is a set of symbolic preconditions ; each precondition is a lifted atom over parameters O . • E
=
( E + , E  ) is a tuple of symbolic effects .
E + are add effects , and E  are delete effects ; both are sets of lifted atoms over parameters O .
• h : R d ( o 1 )+ ··· + d ( o k ) ⇥
A ! R d ( o 1 )+ ··· + d ( o k ) is a lowlevel transition model , a neural network that predicts next attribute values given current ones and an action.
• ⇡ ( a | v ) is an action sampler , a neural network deﬁning a conditional distribution over actions a 2 A , where v 2 R d ( o 1 )+ ··· + d ( o k ) is a vector of attribute values.
In this paper, we will learn and plan with a collection of
NSRTs.
Together with the object set O of a task, a collection of NSRTs jointly deﬁnes four things: an abstract action space for efﬁcient planning; a (partial) abstract transition model over the abstract state space S " and the abstract action space; a (partial) low-level transition model over environment states and actions; and action samplers to reﬁne abstract actions into environment actions.
The rest of this section describes how NSRTs deﬁne these four components.
First, we deﬁne the notion of an NSRT grounded with objects, which represents an abstract action for a task:
Deﬁnition 2.
Given an object set O , a ground NSRT is an
NSRT whose parameters o i 2 O are replaced by objects from O , following a bijective substitution  mapping each o i to an object.
The ground preconditions and effects under  are denoted P  and E  respectively.
Given a set of NSRTs and a task with object set O , the resulting set of ground NSRTs deﬁnes an abstract action space for that task, which we denote as A " .
Therefore, the phrases abstract action and ground NSRT are interchangeable.
For instance, say we wrote an NSRT called S TACK with two parameters ?
x and ?
y ; let  = { ?
x 7! o 3 , ?
y 7!
o 6 } .
Then S TACK ( o 3 , o 6 ) is an abstract action with substitution  .
Working toward a deﬁnition of the abstract transition model, we next deﬁne ground NSRT applicability .
Deﬁnition 3.
A ground NSRT with preconditions P  is applicable in state s 2 S if P  ✓ A BSTRACT ( s ) .
It is also applicable in abstract state s " [Equation] S " if P  ✓ s " .
In words, applicability simply checks that the ground
NSRT’s precondition atoms are a subset of the abstract state atoms.
A set of ground NSRTs deﬁnes a (partial) abstract transition model f " : S " ⇥ A " ! S " , which maps an ab- stract state and abstract action (ground NSRT) to a next abstract state.
The f " ( s " , a " ) is partial since it is only deﬁned when a " is applicable in s " ; when it is applicable, we have:
[Equation]
(Equation [Equation] where E  =
( E +  , E   ) are the effects for a " .
In words, this abstract transition model removes delete effects and includes add effects, as long as the preconditions of the ground NSRT are satisﬁed.
This symbolic representation is akin to operators in classical AI planning (Bonet and Geffner 2001); we use this to our advantage in Section 5.
What is the connection between the symbolic components of an NSRT ( P and E ) and the environment transitions?
To answer this question, we use the following deﬁnition:
Deﬁnition 4.
A ground NSRT a " with effects ( E +  , E   ) covers an environment transition ⌧
= ( s, a, s 0 ) , denoted a " | = ⌧ , if (1) the ground NSRT is applicable in s ; (2)
E +  = A BSTRACT ( s 0 ) \
A BSTRACT ( s ) ; and (3) E   =
A BSTRACT ( s ) \ A BSTRACT ( s 0 ) .
We assume that the following weak semantics connect P and E with the environment: for each ground NSRT a " , there exists a state s 2 S and there exists an action a 2 A s.t.
a " | = ( s, a, f ( s, a )) .
Importantly, this means that the abstraction deﬁned by the NSRTs does not satisfy downward reﬁnability (Marthi, Russell, and Wolfe 2007), which would have required the “there exists a state” to be “for all states.”
These weak semantics will make learning efﬁcient (Section 6), but will require integrated planning (Section 5).
To plan, it is important to be able to simulate the effects of actions on the continuous environment state.
The low-level transition model h , which we discuss next, is used for this.
Deﬁnition 5.
Given a state s and ground NSRT a " with substitution  , the context of s for a " is v  ( s ) =
s [  ( o 1 )]  · · ·  s [  ( o k )] , where v  ( s ) 2 R d ( o 1 )+ ··· + d ( o k ) , s [ · ] looks up an object’s attribute vector in s , and  is vector concatenation.
In words, the context for a ground NSRT is the subset of a state’s attribute vectors that correspond to the ground
NSRT’s objects, assembled into a vector.
The context is the input to the low-level neural transition model h :
[Equation]
where, recall, f is the unknown environment transition model.
All objects not in  are predicted to be unchanged.
Finally, the neural action sampler ⇡ of an NSRT connects the abstract and environment action spaces: it samples continuous actions from the environment action space
A that lead to the NSRT’s symbolic effects.
Given a state s and applicable ground NSRT with substitution  , if a ⇠ ⇡ ( · | v  ( s )) , then ( s, a, f ( s, a )) should ideally be covered by the ground NSRT.
The fact that ⇡ is stochastic can be useful for planning, where multiple samples may be required to achieve desired effects (see Figure 1, or Wang et al.
There are three key properties of NSRTs to take away from these deﬁnitions.
(1) NSRTs are fully relational, i.e., invariant over object identities.
This leads to data-efﬁcient learning and generalization to novel tasks and objects.
(2) NSRTs do not assume downward reﬁnability, as discussed above.
(3) NSRTs are locally scoped ; all components of a ground
NSRT are deﬁned only where it is applicable.
This modularity leads to independent learning problems; see Section 6.
Neuro-Symbolic Planning with NSRTs
We now describe how NSRTs can be used to plan in a given task.
Recall that the weak semantics of NSRTs (Section 4) do not guarantee downward reﬁnability: abstract actions that achieve a goal cannot necessarily be turned into environment actions achieving that goal.
Our strategy will be to perform integrated bilevel planning, with an outer search in the abstract space informing an inner loop producing environment actions.
This planning strategy falls under the broad class of search-then-sample TAMP techniques (Garrett et al. 2021).
See Appendix A.1 for pseudocode (Algorithm 1).
Symbolic Planning.
We perform an outer A ⇤ search from
A BSTRACT ( s 0 ) to g , with the abstract transition model of
Equation 1 and uniform action costs.
For the search heuristic, we use h add , a domain-independent heuristic from the symbolic planning literature (Bonet and Geffner 2001) that approximates the state-to-goal distance under a delete relaxation of the abstract model.
This A ⇤ search will ﬁnd candidate symbolic plans : sequences of ground NSRTs a " 2 A " .
Continuous Planning.
For each candidate symbolic plan, an inner loop attempts to reﬁne it into a plan — a sequence of actions a 2 A that achieves the goal g — using the neural components of the NSRTs.
We use the action sampler ⇡ and low-level transition model h of each ground NSRT in the symbolic plan to construct an imagined state-action trajectory starting from the initial state s 0 .
If the goal g holds in the ﬁnal imagined state, we are done.
If g does not hold, or if any state’s abstraction does not equal the expected abstract state according to the A ⇤ search, then we attempt to sample again.
After n trials (a hyperparameter) unsuccessful imagined trajectories, we return control to the A ⇤ search.
Handling Failures.
Recall that an episode ends if the failure state fail is reached.
Following Srivastava et al.
(2014), we would like to use the presence of a failure state during continuous planning to inform symbolic planning.
In Appendix A.2, we describe a simple domain-independent procedure for learning to predict transitions to the fail state, and using this learned model during planning.
This optimization propagates failure information back to the symbolic level and guides the A ⇤ away from repeating such situations.
Learning NSRTs
We now address learning the structure (Section 6.1), the symbolic components (Section 6.2), and the neural components (Section 6.3) of NSRTs.
See Appendix A.1 for pseu- (Algorithm 2) and an example.
6. Partitioning the Transition Data
Recall that data collection (Section 3) gathers a set of samples from the unknown transition model f : each sample is a state s 2 S , an action a 2 A , and either a next state s 0 2 S or the failure state fail .
We will ignore the transitions that led to fail here; see Appendix A.2.
We begin by partitioning the set of transitions ⌧ =
( s, a, s 0 ) so that each partition will correspond to a single NSRT, thus determining the number of learned NSRTs.
Two transitions belong to the same partition iff their symbolic effects can be uniﬁed :
Deﬁnition 6.
Two transitions ⌧ 1 and ⌧ 2 can be uniﬁed if there exists a bijective mapping  from the objects in E FF ( ⌧ 1 ) to the objects in E FF ( ⌧ 2 ) s.t.
[ E FF ( ⌧ 1 )] =
E FF ( ⌧ 2 ) , where E FF ( ⌧ ) =
( A BSTRACT ( s 0 ) \
A BSTRACT ( s ) , A BSTRACT ( s ) \
A BSTRACT ( s 0 )) , and  [ · ] denotes substitution following  .
These partitions can be computed in time linear in the number of transitions, objects, and atoms per effect set.
6. Learning the Symbolic Components
We now show how to learn NSRT parameters O , symbolic preconditions P , and symbolic effects E for each partition .
First, we deﬁne a mapping R EF that maps a transition ⌧ to a subset of objects in ⌧ that are “involved” in the transition.
In practice, we implement R EF ( ⌧ ) by selecting all objects that appear in E FF ( ⌧ ) .
By construction of our partitions, every transition ⌧will have equivalent R EF ( ⌧ ) , up to object renaming.
We thus introduce NSRT parameters
O corresponding to the types of all the objects in any arbitrarily chosen transition’s R EF ( ⌧ ) .
For each ⌧ 2 , let  ⌧ be a bijective mapping from these parameters O to the objects in R EF ( ⌧ ) .
The NSRT symbolic effects follow by construction: E =   1 ⌧ [ E FF ( ⌧ )] for any arbitrarily chosen ⌧ 2 .
To learn the symbolic preconditions P for the NSRT corresponding to partition , we use a simple inductive approach (Bonet, Frances, and Geffner 2019) that restricts learning by assuming that for each lifted effect set seen in the data, there is exactly one lifted precondition set.
By this assumption, the preconditions follow from an intersection:
P = \ ⌧ =( s, · , · )  1 ⌧ [ P ROJECT ( A BSTRACT ( s ))] , where P ROJECT maps A BSTRACT ( s ) to the subset of atoms whose objects are all in R EF ( ⌧ ) .
See the example in Appendix A.1.
Note that by construction, two different learned
NSRTs cannot cover (Deﬁnition 4) the same transition.
6. Learning the Neural Components
We now describe how to learn a low-level transition model h and action sampler ⇡ for each partition’s NSRT.
The key idea is to use the state projections computed during partitioning to create regression problems.
Recalling Deﬁnition 5, let
[Equation]
the context of state s from transition ⌧ , where ( o 1 , o 2 , . . .
, o k ) are the NSRT parameters.
In words, v  is a vector of the attribute values in state s corresponding to the objects that map the ground atoms E FF ( ⌧ ) of the transition to the lifted effects E of the
NSRT.
We can do the same to produce v  0 for s 0 .
Applying this to all transitions in gives us a dataset of ( v  , a, v  0 ) .
Recall that we want to learn h such that h ( v  ( s ) , a ) ⇡ v  ( f ( s, a )) .
With the dataset above, this learning problem now reduces to regression, with v  and a being the inputs and v  0 being the output.
We use a fully connected neural network (FCN) as the regressor, trained to minimize meansquared error.
Learning ⇡ requires distribution regression, where we ﬁt P ( a | v  ) to the transitions ( v  , a, · ) .
We use an FCN that takes v  as input and predicts the mean µ and covariance matrix ⌃ of a Gaussian.
This FCN is trained to maximize the likelihood of action a under N ( µ, ⌃ ) .
Since
Gaussians have limited expressivity, we also learn an applicability classiﬁer that maps pairs ( v  , a ) to 0 or 1, implemented as an FCN with binary cross-entropy loss.
To sample from ⇡ , we then rejection sample from the Gaussian.
Experiments
Our empirical evaluations address the following key questions: (Q1) Can NSRTs be learned data-efﬁciently? (Q2)
Can learned NSRTs be used to plan to long horizons, especially in tasks involving new and more objects than were seen during training?
(Q3)
Is bilevel planning efﬁcient and effective, and are both levels needed?
(Q4)
To what extent are learned action samplers useful for planning?
7. Experimental Setup
We evaluate Q1-Q4 by running eight methods on four environments.
All experiments were run on Ubuntu .04 using 4 CPU cores of an Intel Xeon Platinum 8260 processor.
Environments.
In this section, we describe our four environments at a high level, with details in Appendix A.3.
The environments are illustrated in Figure 1 (bottom row).
Each environment has three sets of tasks: training, “easy” test, and “hard” test.
“Hard” test tasks require generalization to more objects.
In all environments, we transition to the failure state fail whenever a geometric collision occurs.
• Environment 1:
In “PickPlace1D,” a robot must pick blocks and place them into designated target regions on a table.
All poses are 1D.
Some placements are obstructed by movable objects; none of the predicates capture obstructions, leading to a lack of downward reﬁnability.
• Environment 2:
In “Kitchen,” a robot waiter in 3D must pick cups, ﬁll them with water, wine, or coffee, and serve them to customers.
Some cups are too heavy to be lifted; the cup masses are not represented by the predicates, leading to a lack of downward reﬁnability.
• Environment 3:
In “Blocks,” a robot in 3D must stack blocks on a table to make towers.
In this environment only, the downward reﬁnability assumption holds.
• Environment 4:
In “Painting,” a robot in 3D must pick, wash, dry, paint, and place widgets into a box or shelf.
Placing into the box (resp. shelf) requires picking with a top (resp. side) grasp.
All widgets must be painted a particular color before being placed, which ﬁrst requires washing/drying if the widget starts off dirty or wet.
The box has a lid that may obstruct placements; whether the lid will obstruct a placement is not represented symbolically, leading to a lack of downward reﬁnability.
Methods Evaluated.
We evaluate the following methods.
See Appendix A.4 for additional details.
• Ours: Bilevel planning with NSRTs.
This is our main approach.
Plans are executed open-loop.
• B1:
Symbolic planning only.
This baseline performs symbolic planning using the symbolic components of the learned NSRTs.
When a symbolic plan is found that reaches the goal, it is immediately executed by calling the learned action samplers for the corresponding ground NSRTs in sequence, open-loop.
The low-level transition models are not used.
This baseline ablates away our integrated planner and assumes downward reﬁnability.
• B2:
Neural planning only with forward shooting.
This baseline randomly samples H -length sequences of ground NSRTs and uses their neural components to imagine a trajectory, repeating until it ﬁnds a trajectory where the ﬁnal state satisﬁes the goal.
This baseline does not use the symbolic components of the NSRTs, and thus can be seen as an ablation of the symbolic planning.
• B3:
Neural planning only with hill climbing.
This baseline performs local search over full plans.
At each iteration, a random plan step is resampled using the learned action sampler of a random NSRT.
The new plan is rejected unless it improves the number of goal atoms satisﬁed in the ﬁnal imagined state.
As in B2, the symbolic components of the NSRTs are not used.
• B4: GNN action-value function learning.
This “modelfree” baseline trains a goal-conditioned graph neural network (GNN) action-value function using ﬁtted Qiteration.
The GNN takes as input a continuous low-level state, the corresponding abstract state, and a continuous action; it outputs expected discounted future returns.
At evaluation time, given a state, we draw several candidate actions from the behavior prior ⇡ 0 , and take the best one.
• B5:
Behavior prior only.
This baseline takes actions that are directly sampled from the prior ⇡ 0 .
• B6:
Bilevel planning with prior.
This baseline is an ablation of our main approach that does not use the learned NSRT action samplers ⇡ .
Instead, actions are selected by rejection sampling from the prior ⇡ 0 , with rejections determined by checking the learned applicability classiﬁer.
• B7:
Forward shooting with prior.
This baseline uses the forward shooting of B2 with rejection sampling from ⇡ 0 , like B6.
Only the low-level transition models h are used.
All methods except B5 (non-learning) receive the same data.
7. Results and Discussion
See Figure 3 for learning curves.
The main observation is that in all environments, our method quickly learns to solve tasks within the allotted 3-second timeout.
Thus, Q1 and Q
[Table]
can be answered afﬁrmatively.
Turning to Q3 , we can study whether bilevel planning is effective by comparing Ours, B1, and B2.
The gap between Ours and B1 shows the importance of integrated bilevel planning.
B1 will not be effective in any environment where downward reﬁnability does not hold — only Blocks is downward reﬁnable, which explains the identical performance of Ours and B1 there.
B2 fails in most cases, conﬁrming the usefulness of the learned abstractions.
Both B3 and B4 are generally ineffective.
B3 performs local search, which is much weaker than our directed A ⇤ .
B4 is model-free, forgoing planning in favor of learning an action-value function directly; such strategies are known to be more data-hungry (Moerland, Broekens, and Jonker 2020).
B5 does not require any training, and is just included to illustrate the performance of the behavior prior alone.
To evaluate Q4 , we turn to an ablation study.
Table 1 compares our method with B6 and B7, both of which rejection sample from the generic behavior prior ⇡ 0 rather than using our learned NSRT action samplers.
First, comparing B and B7, bilevel planning is much better than shooting, which speaks to the beneﬁts of using the symbolic components of the NSRTs to guide the continuous planning; this conclusion was also supported by Figure 3.
Second, comparing Ours and B6, the learned action samplers help substantially versus rejection sampling from ⇡ 0 .
This is because the behavior prior is highly generic, not targeted toward any speciﬁc set of effects like NSRT action samplers are, so rejection sampling can take many tries to pass the applicability classiﬁer.
Conclusion, Limitations, and Future Work
We proposed NSRTs for long-horizon, goal-based, objectoriented planning tasks.
We showed that their neurosymbolic structure affords fast bilevel planning, and found experimentally that they are data-efﬁcient to learn and effective at generalization, outperforming several baselines.
Key limitations of the current work include: (1) that predicates are given; (2) that a behavior prior is given for data collection; (3) that environments are deterministic and fully observable.
To address (1), NSRTs could be combined with work on learning predicates from high-dimensional inputs (Asai 2019).
(2) could be addressed using skill prior learning techniques (Ajay et al. 2021).
For (3), we hope to draw on
TAMP techniques for handling stochasticity and partial observability (Hadﬁeld-Menell et al. 2015; Garrett et al. 2020).