L EARNING TO P LAN IN H IGH D IMENSIONS VIA N EURAL E XPLORATION -E XPLOITATION T REES
A BSTRACT
We propose a meta path planning algorithm named Neural Exploration-Exploitation
Trees (NEXT) for learning from prior experience for solving new path planning problems in high dimensional continuous state and action spaces
Compared to more classical sampling-based methods like RRT, our approach achieves much better sample efﬁciency in high-dimensions and can beneﬁt from prior experience of planning in similar environments.
More speciﬁcally, NEXT exploits a novel neural architecture which can learn promising search directions from problem structures.
The learned prior is then integrated into a UCB-type algorithm to achieve an online balance between exploration and exploitation when solving a new problem.
We conduct thorough experiments to show that NEXT accomplishes new planning problems with more compact search trees and signiﬁcantly outperforms state-of-the-art methods on several benchmarks
I NTRODUCTION
Path planning is a fundamental problem with many real-world applications, such as robot manipulation and autonomous driving.
A simple planning problem within low-dimensional state space can be solved by ﬁrst discretizing the continuous state space into a grid, and then searching for a path on top of it using graph search algorithms such as A ∗ (Hart et al., 1968).
However, due to the curse of dimensionality, these approaches do not scale well with the number of dimensions of the state space.
For high-dimensional planning problems, people often resort to sampling-based approaches to avoid explicit discretization.
Sampling-based planning algorithms, such as probabilistic roadmaps (PRM) (Kavraki et al., 1996), rapidly-exploring random trees (RRT)
(LaValle, 1998), and their variants (Karaman & Frazzoli, 2011) incrementally build an implicit representation of the state space using probing samples.
These generic algorithms typically employ a uniform sampler which does not make use of the structures of the problem.
Therefore they may require lots of samples to obtain a feasible solution for complicated problems.
To improve the sample efﬁciency, heuristic biased samplers, such as Gaussian sampler (Boor et al., 1999), bridge test (Hsu et al., 2003) and reachability-guided sampler (Shkolnik et al., 2009) have been proposed.
All these sampling heuristics are designed manually to address speciﬁc structural properties, which may or may not be valid for a new problem, and may lead to even worse performance compared to the uniform proposal.
Online adaptation in path planning has also been investigated for improving sample efﬁciency in current planning problem.
Speciﬁcally, Hsu et al.
(2005) exploits online algorithms to dynamically adapts the mixture weights of several manually designed biased samplers.
Burns & Brock (2005a;b) ﬁt a model for the planning environment incrementally and use the model for planning.
Yee et al.
(2016) mimics the Monte-Carlos tree search (MCTS) for problems with continuous state and action spaces.
These algorithms treat each planning problem independently , and the collected data from previous experiences and built model will be simply discarded when solving a new problem.
However, in practice, similar planning problems may be solved again and again, where the problems are different but sharing common structures.
For instance, grabbing a coffee cup on a table at different time are different problems, since the layout of paper and pens, the position and orientation of coffee cups may be different every time; however, all these problems show common structures of handling similar objects which are placed in similar fashions.
Intuitively, if the common characteristics across problems can be learned via some shared latent representation, a planner based on such representation can then be transferred to new problems with improved sample efﬁciency.
Several methods have been proposed recently to learn from past planning experiences to conduct more efﬁcient and generalizable planning for future problems.
These works are limited in one way or the other.
Zucker et al.
; Zhang et al.
; Huh & Lee (2018) treat the sampler in the sampling-based planner as a stochastic policy to be learned and apply policy gradient or TD-algorithm to improve the policy.
Finney et al.
(2007); Bowen & Alterovitz (2014)
; Ye & Alterovitz (2017);
Ichter et al.
; Kim et al.
; Kuo et al.
(2018) apply imitation learning based on the collected demonstrations to bias for better sampler via variants of probabilistic models, e.g. , (mixture of) Gaussians, conditional VAE, GAN, HMM and RNN.
However, many of these approaches either rely on specially designed local features or assume the problems are indexed by special parameters, which limits the generalization ability.
Deep representation learning provides a promising direction to extract the common structure among the planning problems, and thus mitigate such limitation on handdesigned features.
However, existing work, e.g. , motion planning networks (Qureshi et al., 2019), value iteration networks (VIN) (Tamar et al., 2016), and gated path planning networks (GPPN)
(Lee et al., 2018), either apply off-the-shelf MLP architecture ignoring special structures in planning problems or can only deal with discrete state and action spaces in low-dimensional settings.
In this paper, we present Neural EXploration-EXploitation Tree (NEXT) , a meta neural path planning algorithm for high-dimensional continuous state space problems.
The core contribution is a novel attention-based neural architecture that is capable of learning generalizable problem structures from previous experiences and produce promising search directions with automatic online exploration-exploitation balance adaption.
Compared to existing learning-based planners, • NEXT is more generic .
We propose an architecture that can embed high dimensional continuous state spaces into low dimensional discrete spaces, on which a neural planning module is used to extract planning representation.
These module will be learned end-to-end
• NEXT balances exploration-exploitation trade-off .
We integrate the learned neural prior into an upper conﬁdence bound (UCB) style algorithm to achieve an online balance between exploration and exploitation when solving a new problem.
Empirically, we show that NEXT can exploit past experiences to reduce the number of required samples drastically for solving new planning problems, and signiﬁcantly outperforms previous state-of-the-arts on several benchmark tasks.
1. R ELATED W ORKS
Designing non-uniform sampling strategies for random search to improve the planning efﬁciency has been considered as we discussed above.
Besides the mentioned algorithms, there are other works along this line, including informed RRT ∗ (Gammell et al., 2014) and batch informed Trees (BIT*) (Gammell et al., 2015) as the representative work.
Randomized A ∗ (Diankov & Kuffner, 2007) and sampling-based A ∗ (Persson & Sharf, 2014) expand the search tree with hand-designed heuristics.
These methods incorporate the human prior knowledge via hard-coded rules, which is ﬁxed and unable to adapt to problems, and thus, may not universally applicable.
Choudhury et al.
; Song et al.
(2018) attempt to learn search heuristics.
However, both methods are restricted to planning on discrete domains.
Meanwhile, the latter one always employs an unnecessary hierarchical structure for path planning, which leads to inferior sample efﬁciency and extra computation.
The online exploration-exploitation trade-off is also an important issue in planning.
For instance,
Rickert et al.
(2009) constructs a potential ﬁeld sampler and tuned the sampler variance based on collision rate for the trade-off heuristically.
Paxton et al.
(2017) separates the action space into high-level discrete options and low-level continuous actions, and only considered the trade-off at the discrete option level, ignoring the exploration-exploitation in the ﬁne action space.
These existing works address the trade-off in an ad-hoc way, which may be inferior for the balance.
There have been sevearl non-learning-based planning methods that can also leverage experiences (Kavraki et al., 1996; Phillips et al., 2012) by utilizing search graphs created in previous problems.
However, they are designed for largely ﬁxed obstacles and cannot be generalized to unseen tasks from the same planning problems distribution.
S ETTINGS FOR L EARNING TO P LAN
Let S ⊆
R q be the state space of the problem, e.g. , all the conﬁgurations of a robot and its base location in the workspace, S obs (cid:40) S be the obstacles set, S free :=
S \ S obs be the free space, s
init ∈ S free be the initial state and S goal (cid:40) S free be the goal region.
Then the space of all collisionfree paths can be deﬁned as a continuous function Ξ := { ξ ( · ) :
[0 , 1] → S free } .
Let c ( · ) :
Ξ (cid:55)→ R be the cost functional over a path.
The optimal path planning problem is to ﬁnd the optimal path in terms of cost c ( · ) from start s init to goal S goal in free space S free , i.e. ,
[Equation]
Traditionally (Karaman & Frazzoli, 2011), the planner has direct access to ( s init , S goal , c ( · )) and the workspace map (Ichter et al., 2018; Tamar et al., 2016; Lee et al., 2018), map ( · ) : R 2 or R 3 (cid:55)→ { 0 , 1 } , (0: free spaces and 1: obstacles).
Since S free often has a very irregular geometry (illustrated in Figure 10 in Appendix A), it is usually represented via a collision detection module which is able to detect the obstacles in a path segment.
For the same reason, the feasible paths in Ξ are hard to be described in parametric forms, and thus, the nonparametric ξ , such as a sequence of interconnected path segments [ s 0 , s 1 ] , [ s 1 , s 2 ] , . . .
, [ s T − 1 , s T ] ⊂ S with ξ (0) =
s
= s init and ξ (1)
= s T , is used with an additive cost (cid:80) Ti =1 c ([ s i − 1 , s i ]) .
Assuming given the planning problems { U i :
=
( s init , S goal , S , S free , map , c ( · )) }
Ni =1 sampled from some distribution U , we are interested in learning an algorithm alg ( · ) , which can produce the (nearly)-optimal path efﬁciently from the observed planning problems.
Formally, the learning to plan is deﬁned as
[Equation]
where A denotes the planning algorithm family, and (cid:96) ( · ) denotes some loss function which evaluates the quality of the generated path and the efﬁciency of the alg ( · ) , e.g. , size of the search tree.
We elaborate each component in Eq (2) in the following sections.
We ﬁrst introduce the tree-based sampling algorithm template in Section 3, upon which we instantiate the alg ( · ) via a novel attention-based neural parametrization in Section 4.2 with exploration-exploitation balance mechanism in Section 4.1.
We design the (cid:96) -loss function and the meta learning algorithm in Section 4.3.
Composing every component together, we obtain the neural exploration-exploitation trees (NEXT) which achieves outstanding performances in Section 5.
[Figure]
P RELIMINARIES
[Figure]
Problem:
U = (
[Equation]
; 2 Initialize T = ( V , E ) with V ←
[Equation]
s init [Equation]
The sampling-based planners are more practical and become dominant for high-dimensional path planning problems (Elbanhawi & Simic, 2014).
We describe a unifying view for many existing tree-based sampling algorithms (TSA), which we will also base our algorithm upon.
More speciﬁcally, this family of algorithms maintain a search tree T rooted at the initial point s init and connecting all sampled points V in the conﬁguration space with edge set E .
The tree will be expanded by incorporating more sampled states until some leaf reaches S goal .
Then, a feasible solution for the path planning problem will be extracted based on the tree T .
The template of tree-based sampling algorithms is summarized in Algorithm 1 and illustrated in Fig.
1(c).
A key component of the algorithm is the Expand operator, which generates the next exploration point s new and its parent s parent ∈ V .
To ensure the feasibility of the solution, the s new must be reachable from T , i.e. , [ s parent , s new ] is collision-free, which is checked by a collision detection function.
As we will discuss in Appendix B, by instantiating different Expand operators, we will arrive at many existing algorithms, such as RRT (LaValle, 1998) and EST (Hsu et al., 1997; Phillips et al., 2004).
One major limitation of existing TSAs is that they solve each problem independently from scratch and ignore past planning experiences in similar environments.
We introduce the neural components into TSA template to form the learnable planning algorithm family A , which can explicitly take advantages of the past successful experiences to bias the Expand towards more promising regions.
N EURAL E XPLORATION -E XPLOITATION T REES
Based on the TSA framework, we introduce a learnable neural based Expand operator, which can balance between exploration and exploitation, to instantiate A in Eq (2).
With the self-improving training, we obtain the meta NEXT algorithm illustrated in Figure 1.
4. G UIDED P ROGRESSIVE E XPANSION
Algorithm 2: NEXT :: Expand (
= ( V , E ) , U )
[Equation]
We start with our design of the Expand .
We assume having an estimated value function ˜
V ∗ ( s | U ) , which stands for the optimal cost from s to target in planning problem U , and a policy ˜ π ∗ ( s (cid:48) | s, U ) , which generates the promising action s (cid:48) from state s .
The concrete parametrization of ˜ V ∗ and ˜ π ∗ will be explained in Section 4.2 and learned in Section 4.3.
We will use these functions to construct the learnable Expand with explicit exploration-exploitation balancing.
The Expand operator will expand the current search tree T by a new neighboring state s new around
T .
We design the expansion as a two-step procedure: (i) select a state s parent from existing tree T ; (ii) expand a state s new in the neighborhood of s parent .
More speciﬁcally,
Selecting s parent from T in step (i).
Consider the negative value function − ˜ V ∗ ( s | U ) as the rewards r ( s ) , step (i) shares some similarity with the multi-armed bandit problem by viewing existing nodes s ∈ V as arms.
However, the vanilla UCB algorithm is not directly applicable, since the number of states is increasing as the algorithm proceeds and the value of these adjacent states are naturally correlated.
We address this challenge by modeling the correlation explicitly as in contextual bandits.
Speciﬁcally, we parametrize the UCB of the reward function as φ ( s ) , and select a node from T according to φ ( s )
[Equation]
where ¯ r t and σ t denote the average reward and variance estimator after t -calls to Expand .
Denote the sequence of t selected tree nodes so far as S t = { s 1 parent , . . .
, s tparent } , then we can use kernel smoothing estimator for ¯ r t ( s ) =
(cid:80) s (cid:48)∈S t k
( s (cid:48) ,s )
r
( s (cid:48) )
(cid:80) s (cid:48)∈S t k ( s (cid:48) ,s ) and
σ t ( s ) = (cid:114) log (cid:80) s (cid:48)∈S t w ( s (cid:48) )
w ( s ) where k ( s (cid:48) , s ) is a kernel function and w ( s )
= (cid:80) s (cid:48)
∈S t k ( s (cid:48) , s ) .
Other parametrizations of ¯ r t and σ t are also possible, such as Gaussian Process parametrization in Appendix C.
The average reward exploits more promising states, while the variance promotes exploration towards less frequently visited states; and the exploration versus exploitation is balanced by a tunable weight λ > 0 .
Expanding a reachable s new in step (ii).
Given the selected s parent , we consider expanding a reachable state in the neighborhood s parent as an inﬁnite-armed bandit problem.
Although one can ﬁrst samples k arms uniformly from a neighborhood around s parent and runs a UCB algorithm on the randomly generated ﬁnite arms (Wang et al., 2009), such uniform sampler ignores problem structures, and will lead to unnecessary samples.
Instead we will employ a policy ˜ π ∗ ( s (cid:48) | s, U ) 1 for guidance when generating the candidates.
The ﬁnal choice for next move will be selected from these candidates with max φ ( s ) deﬁned in (3).
As explained in more details in Section 4.2, ˜ π ∗ will be
[Figure]
trained to mimic previous successful planning experiences across different problems, that is, biasing the sampling towards the states with higher successful probability.
With these detailed step (i) and (ii) , we obtain NEXT :: Expand in Algorithm 2 (illustrated in Figure 1(b) and (c)).
Plugging it into the TSA in 1, we construct alg ( · ) ∈
A which will be learned.
The guided progressive expansion bears similarity to MCTS but deals with high dimensional continuous spaces.
Moreover, the essential difference lies in the way to select state in T for expansion:
the
MCTS only expands the leaf states in T due to the hierarchical assumption, limiting the exploration ability and incurring extra unnecessary UCB sampling for internal traversal in T ; while the proposed operation enables expansion from each visited state, particularly suitable for path planning problems.
4. N EURAL A RCHITECTURE FOR V ALUE F UNCTION AND E XPANSION P OLICY
In this section, we will introduce our neural architectures for ˜ V ∗ ( s | U ) and ˜ π ∗ ( s (cid:48) | s, U ) used in
NEXT :: Expand .
The proposed neural architectures can be understood as ﬁrst embedding the state and problem into a discrete latent representation via an attention-based module in Section 4.2.1, upon which the neuralized value iteration, introduced in Section 4.2.2, is performed to extract features for deﬁning
˜ V ∗ ( s | U ) and ˜ π ∗ ( s (cid:48) | s, U ) , as illustrated in Figure 2.
4.2. C ONFIGURATION S PACE E MBEDDING
Our network for embedding high-dimension conﬁguration space into a latent representation is designed based on an attention mechanism.
More speciﬁcally, let s w denote the workspace in state and s h denote the remaining dimensions of the state, i [Figure] e. s =
( s w , s h ) .
s w and s h will be embedded by different sub-neural networks and combined for the ﬁnal representation, as illustrated in Figure 3.
For simplicity of exposition [Figure] we will focus on the 2d workspace,
i
[Figure]
e. , s w ∈ R 2 .
However, our method applies to 3d workspace as well.
[Figure]
• Spatial attention.
The workspace information s w will be embedded as µ w ( s w ) ∈
R
d × d , d is a hyperparameter related to map (see remark below).
The spatial embedding module (upper part in Figure 3) is composed of k w convolution layers, i.e. ,
[Equation]
where θ wi denotes the convolution kernels, ⊕ denotes the convolution operator and f wi ( s w )
∈ R d × d × d i with d i channels.
The ﬁrst layer f w 0 is designed to represent s w into a d × d × 4 tensor as f w 0 ( s w ) ij =
[ s w 1 , s w 2 , i, j ] , i, j = 1 , . . .
, d , without any loss of information.
[Figure]
• Conﬁguration attention.
The remaining conﬁguration state information will be embedded as µ h ( s h ) through k h fully-connected layers (bottom-left part in Figure 3), i
e.
[Equation]
where
µ h ( s h ) ∈ R
d a and f h 0 ( s h ) =
s h .
The ﬁnal representation µ θ ( s ) will be obtained by multiplying µ w ( s w ) with µ h ( s h ) element-wisely, µ θ ( s ) ijl = µ w ( s w ) ij · µ h ( s h ) l , which is a d × d × d a tensor attention map with µ θ ( s ) ijl (cid:62) 0 , and (cid:80) ijl µ θ ( s ) ijl = 1 (bottom-right part in Figure 3).
Intuitively, one can think of d a as the level of the learned discretization of the conﬁguration space s h , and the entries in µ softly assign the actual state s to these discretized locations.
θ := ( { θ wi } k w − 1 i =0 , { θ hi , b i }
k
h − 1 i =0 ) are the parameters to be learned.
Remark (different map size): To process map using convolution neural networks, we resize it to a d × d image, with the same size as the spatial attention in Eq (4), where d is a hyperparameter.
4.2. N EURAL V ALUE I TERATION
We then apply neuralized value iteration on top of the conﬁguration space embedding to extract further planning features (Tamar et al., 2016; Lee et al., 2018).
Speciﬁcally, we ﬁrst produce the embedding µ θ ( s goal ) of the center of the goal region s goal .
We execute T steps of neuralized Bellman updates (planning module in Figure 4) in the embedding space,
[Equation]
[Equation] (cid:105)(cid:17) , with ( ν ∗ (0) , ˜ R )
=
σ ( W 0 ⊕ [ µ θ ( s goal ) , map ]) , and obtain ν
∗ ( T ) ∈ R
d × d × d a × p .
Both W 0 and W 1 are 3 d convolution kernels, min implements the pooling across channels.
Accordingly, ν ∗ ( T ) now can be understood as a latent representation of the value function ˜
V ∗ ( · ) in learned embedding space for the problem U with s goal in map .
To deﬁne the value function for particular state s , i.e. , ˜ V ∗ ( s | U ) , from the latent representation ν ∗ ( T ) , we ﬁrst construct another attention model between the embedding of state s using µ θ ( s ) and ν ∗ ( T ) , i.e. , ψ ( s )
k
= (cid:80) ijl ν ∗ ( T ) ijlk · µ θ ( s ) ijl , for k = 1 , . .
p .
Finally we deﬁne
[Equation]
and
[Equation]
where h W 2 and h W 3 are fully connected dense layers with parameters W 2 and W 3 respectively, and N ( h W 3 ( ψ ( s )) , σ 2 ) is a Gaussian distribution with variance σ 2 .
Note that we also parametrize the policy ˜ π ∗ ( s (cid:48) | s, U ) using the embedding ν ∗ ( T ) , since the policy is connected to the value function via π ∗ ( s (cid:48) | s, U ) =
argmin s (cid:48) ∈S c ([ s, s (cid:48) ])
+ V ∗ ( s (cid:48) | U ) .
It should also be emphasized that in our parametrization, the calculation of ν ∗ ( T ) only relies on the µ θ ( s goal ) , which can be reused for evaluating ˜ V ∗ ( s | U ) and ˜ π ∗ ( s (cid:48) | s, U ) over different s , saving computational resources.
Using this trick the algorithm runs 10 × - 100 × faster empirically.
The overall model architecture in alg ( · ) is illustrated in Figure 4.
The parameters W =
( W 0 , W 1 , W 2 , W 3 , θ ) will be learned together by our meta self-improving learning .
For the details of the parameterization and the size of convolution kernels in our implementation, please refer to Figure 12 in Appendix D.
4. M ETA SELF - IMPROVING LEARNING
The learning of the planner alg ( · ) reduces to learning the parameters in ˜ V ∗ ( s | U ) and ˜ π ∗ ( s (cid:48) | s, U ) and is carried out while planning experiences accumulate.
We do not have an explicit training and testing phase separation.
Particularly, we use a mixture of RRT :: Expand and NEXT ::
Expand with probability (cid:15) and 1 − (cid:15) , respectively, inside the TSA framework in Algorithm 1.
The RRT ∗ postprocessing step is used in the template.
The (cid:15) is set to be 1 initially since { ˜ V ∗ , ˜ π ∗ } are not well-trained, and thus, the algorithm behaves like RRT ∗ .
As the training proceeds, we anneal (cid:15) gradually as the sampler becomes more efﬁcient.
The dataset D n = { ( T j , U j ) } nj =1 for the n -th training epoch is collected from the previous successful planning experiences across multiple random problems .
We ﬁx the size of dataset and update D in the same way as experience reply buffer (Lin, 1992; Schaul et al., 2015).
For an experience ( T , U ) ∈ D n , we can reconstruct the successful path { s i } mi =1 from the search tree T ( m is the number of segments), and the value of each state s i in the path will be the sum of cost to the goal region, i.e. , y i :=
(cid:80) m − 1 l = i
c
([ s l , s l +1 ]) .
We learn { ˜ V ∗ , ˜ π ∗ } by optimizing objective min W (cid:88)
[Equation]
n
[Equation]
[Equation]
[Equation]
=
The loss (6) pushes the ˜ V ∗ and ˜ π ∗ to chase the successful trajectories, providing effective guidance in alg ( · ) for searching, and therefore leading to efﬁcient searching procedure with less sample complexity and better solution.
On one hand, the value function and policy estimation { ˜ V ∗ , ˜ π ∗ } is improved based upon the successful outcomes from NEXT itself on previous problems.
On the other hand, the updated { ˜ V ∗ , ˜ π ∗ } will be applied in the next epoch to improve the performance.
Therefore, the training is named as Meta Self-Improving Learning (MSIL) .
Since all the trajectories we collected for learning are feasible, the reachability of the proposed samples is enforced implicitly via imitating these successful paths.
By putting every components together into the learning to plan framework in Eq (2), the overall procedure is summarized in Algorithm 3 and illustrated in Figure 1.
E XPERIMENTS
In this section, we evaluate the proposed NEXT empirically on different planning tasks in a variety of environments.
Comparing to the existing planning algorithms, NEXT achieves the state-of-the-art performances, in terms of both success rate and the quality of the found solutions.
We further demonstrate the power of the proposed two components by the corresponding ablation study.
We also include a case study on a real-world robot arm control problem at the end of the section.
5. E XPERIMENT S ETUP
Benchmark environments.
We designed four benchmark tasks to demonstrate the effectiveness of our algorithm for high-dimensional planning.
The ﬁrst three involve planning in a 2d workspace with a 2 DoF (degrees of freedom) point robot, a 3 DoF stick robot and a 5 DoF snake robot, respectively.
The last one involves planning a 7 DoF spacecraft in a 3d workspace.
For all problems in each benchmark task, the workspace maps were randomly generated from a ﬁxed distribution; the initial and goal states were sampled uniformly randomly in the free space; the cost function c ( · ) was set as the sum of the Euclidean path length and the control effort penalty of rotating the robot joints.
Baselines.
We compared NEXT with RRT ∗ (Karaman & Frazzoli, 2011), BIT ∗ (Gammell et al., 2015), CVAE-plan (Ichter et al., 2018), Reinforce-plan (Zhang et al., 2018), and an improved
[Figure] [Figure]
version of GPPN (Lee et al., 2018) in terms of both planning time and solution quality.
RRT ∗ and BIT ∗ are two widely used effective instances of TSA in Algorithm 1.
In our experiments, we equipped RRT ∗ with the goal biasing heuristic to improve its performance.
BIT ∗ adopts the informed search strategy (Gammell et al., 2015) to accelerate planning.
CVAE-plan and Reinforce-plan are two learning-enhanced TSA planners proposed recently.
CVAE-plan learns a conditional VAE as the sampler (Sohn et al., 2015), which will be trained by near-optimal paths produced by RRT ∗ .
Reinforce-plan learns to do rejection sampling with policy gradient methods.
For the improved
GPPN, we combined its architecture for map with a fully-connected MLP for the rest state, such that it can be applied to high-dimensional continuous spaces.
Please refer to Appendix E for more details.
Settings.
For each task, we randomly generated 3000 different problems from the same distribution without duplicated maps.
We trained all learning-based baselines using the ﬁrst 2000 problems, and reserved the rest for testing.
The parameters for RRT ∗ and BIT ∗ are also tuned using the ﬁrstproblems.
For NEXT, we let it improve itself using MSIL over the ﬁrst 2000 problems.
In this period, for every 200 problems, we updated its parameters and annealed (cid:15) once.
5. R ESULTS AND A NALYSIS
Comparison results.
Examples of all four environments are illustrated in Appendix F.1 and Figure 6, where NEXT ﬁnds high-quality solutions as shown.
We also illustrated the comparison of the search trees on two 2d and 7d planning tasks between NEXT and RRT* in Figure 5 (a)-(c) and Figure(b) and (c).
Obviously, the proposed NEXT algorithm explores with guidance and achieves better quality solutions with fewer samples, while the RRT* expands randomly which may fail to ﬁnd a solution.
The learned ˜ V ∗ and ˜ π ∗ in the 2d task are also shown in Figure 5(d).
As we can see, they are consistent with our expectation, towards the ultimate target in the map.
For more search tree comparisons for all four experiments, please check Figure 18, 19, 20 and 21 in Appendix F.
To systematically evaluate the algorithms, we recorded the cost of time (measured by the number of collision checks used) to ﬁnd a collision-free path, the success rate within time limits, and the cost of the solution path for each run.
The results of the reserved 1000 test problems of each environment are shown in the top row of Figure 7.
We set the maximal number of samples as 500 for all algorithms.
Both the kernel smoothing (NEXT-KS) and the Gaussian process (NEXT-GP) version of NEXT achieves the state-of-the-art performances, under all three criteria in all test environments.
Although the BIT* utilizes the heuristic particularly suitable for 3d maze in 7d task and performs quite well,
[Figure]
the NEXT algorithm still outperform every competitor by a large margin, no matter learning-based or preﬁxed heuristic planner, demonstrating the advantages of the proposed NEXT algorithm.
Self-improving.
We plot the performance improvement curves of our algorithms on the 5d planning task in the bottom row of Figure 7.
For comparison, we also plot the performance of RRT ∗ .
At the beginning phase of self-improving, our algorithms are comparable to RRT ∗ .
They then gradually learn from previous experiences and improve themselves as they see more problems and better solutions.
In the end, NEXT-KS is able to match the performance of RRT ∗ -10k using only one-twentieth of its samples, while the competitors perform consistently without any improvements.
Due to the space limits, we put improvement curves on other environments in Figure 17 and the quantitative evaluation in Table 1, 2, and 3 in Appendix F.
Please refer to the details there.
5. A BLATION S TUDIES
Ablation study I: guided progressive expansion.
To demonstrate the power of NEXT :: Expand , we replace it with breadth-ﬁrst search (BFS) (Kim et al., 2018), another expanding strategy, while keeping other components the same.
Speciﬁcally, BFS uses a search queue in planning.
It repeatedly pops a state s out from the search queue, samples k states from π ( ·| s ) , and pushes all generated samples and state s back to the queue, until the goal is reached.
For fairness, we use the learned sampling policy π ( s (cid:48) | s, U ) by NEXT-KS in BFS.
As shown in Figure 7, BFS obtained worse paths with a much larger number of collision checks and far lower success rate, which justiﬁes the importance of the balance between exploration versus exploitation achieved by the proposed NEXT :: Expand .
Ablation study II: neural architecture.
To further demonstrate the beneﬁts of the proposed neural architecture for learning generalizable representations in high-dimension planning problems, We replaced our attention-based neural architecture with an improved GPPN, as explained in Appendix E, for ablation study.
We extended the GPPN for continuous space by adding an extra reactive policy network to its ﬁnal layers.
We emphasize the original GPPN is not applicable to the tasks in our experiments.
Intuitively, the improved GPPN ﬁrst produces a ‘rough plan’ by processing the robot’s discretized workspace positions.
Then the reactive policy network predicts a continuous action from both the workspace feature and the full conﬁguration state of the robot.
We provide more preference to the improved GPPN by training it to imitate the near-optimal paths produced by RRT* in the training problems.
During test time it is also combined with both versions of the guided progressive expansion operators.
As we can see, both GPPN-KS and GPPN-GP are clearly much worse than NEXT-KS and NEXT-GP, demonstrating the advantage of our proposed attention-based neural architecture in high-dimensional planning tasks.
Ablation study III: learning versus heuristic.
The NEXT algorithm in Figure 5 shows similar behavior as the Dijkstra heuristic, i.e. sampling on the shortest path connecting the start and the goal in workspace.
However, in higher dimensional space, the Dijkstra heuristic will fail.
To demonstrate that, we replace the policy and value network with Dijkstra heuristic, using the best direction in workspace to guide sampling.
NEXT performs much better than Dijkstra in all but the 2d case, in which the workspace happens to be the state space.
5. C ASE S TUDY : R OBOT A RM C ONTROL
We conduct a real-world case study on controlling robot arms to move objects on a shelf.
On this representative real-time task, we demonstrate the advantages of the NEXT in terms of the wall-clock.
In each planning task, there is a shelf of multiple levels, with each level horizontally divided into multiple bins.
The task is to plan a path from a location in one bin to another, i.e. , the end effectors of the start and goal conﬁgurations are in different bins.
The heights of levels, widths of bins, and the start and goal are randomly drawn from some ﬁxed distribution.
Different from previous experiments, the base of the robot is ﬁxed.
We consider the BIT* instead of RRT* as the imperfect expert intraining problems.
We then evaluate the algorithm on a separated 1000 testing problems.
We compare NEXT(-
KS) with the highly tuned BIT* and RRT* in OMPL, and also CVAE-plan and Reinforce-plan in Figure 8.
As seen from the visualization of the found paths in Figure 9, this is a very difﬁcult task.
Our NEXT outperforms the baselines by a large margin, requiring only 1 second to reach the same success rate as running 50 seconds of
BIT*.
Due to space limits, we put details of the experiment setups, more results and analysis in Appendix F.4.
C ONCLUSION
In this paper, we propose a self-improving planner, Neural EXploration-EXploitation Trees (NEXT), which can generalize and achieve better performance with experiences accumulated.
The algorithm achieves a delicate balance between exploration-exploitation via our carefully designed UCB-type expansion operation.
To obtain the generalizable ability across different problems, we proposed a new parametrization for the value function and policy, which captures the Bellman recursive structure in the high-dimensional continuous state and action space.
We demonstrate the power of the proposed algorithm by outperforming previous state-of-the-art planners with signiﬁcant margins on planning problems in a variety of different environments.
A CKNOWLEDGEMENT
We thank the Google Research Brain team members for helpful thoughts and discussions as well as the anonymous reviewers for their insightful comments and suggestions.
This work is supported in part by NSF grants CDS&E-1900017 D3SC, CCF-1836936 FMitF, IIS-1841351, CAREER IISto L.S.
Appendix A I LLUSTRATION OF THE D IFFICULTY IN P LANNING P ROBLEMS
The Figure 10(a) illustrates a concrete planning problem for a stick robot in 2d workspace.
With one extra continuous action for rotation, the conﬁguration state is visualized in Figure 10(b), which is highly irregular and unknown to the planner.
[Figure]
B M ORE P RELIMINARIES Tree-based
sampling planner
The tree-based sampling planner algorithm is illustrated in Figure 11.
The Expand in Algorithm 1 operator returns an existing node in the tree s parent ∈ V and a new state s new ∈ S sampled from the neighborhood of s parent .
Then the line segment [ s parent , s new ] is passed to function ObstacleFree for collision checking.
If the line segment [ s parent , s new ] is collision-free (no obstacle in the middle, or called reachable from T ), then s new is added to the tree vertex set V , and the line segment is added to the tree edge set E .
If the newly added node s new has reached the target S goal , the algorithm will return.
Optionally, some concrete algorithms can deﬁne a
Postprocess operator to reﬁne the search tree.
For an example of the Expand operator, as shown in
Figure 1 (c), since there is no obstacle on the dotted edge [ s parent , s new ] , i.e. , s new is reachable, the new state and edge will be added to the search tree (connected by the solid edges).
[Figure]
Now we will provide two concrete algorithm examples.
For instance,
• If we instantiate the Expand operator as Algorithm 4, then we obtain the rapidly-exploring random trees (RRT) algorithm (LaValle, 1998), which ﬁrst samples a state s from the conﬁguration space
Algorithm 4: RRT :: Expand ( T , U )
Data: T =
( V , E ) ,
U =
( s init , S goal , S , S free , map , c ( · )) 1 s rand ← U nif ( S ) ; (cid:46) Sample conﬁguration space 2 s parent ← argmin s ∈V (cid:107) s
rand − s (cid:107) ; (cid:46) Pull to a tree node 3 s new ← argmin s ∈B ( s parent ,η ) (cid:107) s − s rand (cid:107) ; 4 return s parent , s new ;
Algorithm 5: EST :: Expand ( T , U )
Data: T =
( V , E ) ,
U =
( s init , S goal , S , S free , map , c ( · ))
1 s parent ∼ φ ( s ) , s ∈ V ; (cid:46) Sample a tree node 2 s new ← U nif ( B ( s parent )) ; (cid:46) Sample neighborhood 3 return s nearest , s new ;
S and then pulls it toward the neighborhood of current tree T measured by a ball of radius η :
B ( s, η ) =
{ s (cid:48)
∈ S | (cid:107) s (cid:48)
− s (cid:107) (cid:54) η } .
Moreover, if the Postprocess operator is introduced to modify the maintained search tree as in RRT ∗ (Karaman & Frazzoli, 2011), the algorithm is provable to obtain the optimal path asymptotically.
• If we instantiate the Expand operator as Algorithm 5, then we obtain the expansive-space trees (EST) algorithm
(Hsu et al., 1997; Phillips et al., 2004)
, which samples a state s from the nodes of the existing tree, and then draw a sample from the neighborhood of s .
UCB-based algorithms
Speciﬁcally, in a K -armed bandit problem, the UCB algorithm will ﬁrst play each of the arms once, and then keep track of the average reward ¯ r i and the visitation count n i for each arm.
After T rounds of trials, the UCB algorithm will maintain a set of information { (¯ r i , n i ) }
Ki =1 with (cid:80) Ki =
1 n i = T .
Then, for the next round, the UCB algorithm will select the next arm based on the one-sided conﬁdence interval estimation provided by the Chernoff-Hoeffding bound,
[Equation]
where λ controls the exploration-exploration trade-off.
It has been shown that the UCB algorithm achieves O (log T ) regret.
However, the MCTS is not directly applicable to continuous state-action spaces.
There have been many attempts to generalize the UCB and UCT algorithms to continuous state-action spaces (Chu et al., 2011; Krause & Ong, 2011; Couëtoux et al., 2011; Yee et al., 2016).
For instance, contextual bandit algorithms allow continuous arms but involve a non-trivial high dimensional non-convex optimization to select the next arm.
In UCT, the progressive widening technique has been designed to deal with continuous actions (Wang et al., 2009).
Even with these extensions, the
MCTS restricts the exploration only from leaves states, implicitly adding an unnecessary hierarchical structure for path planning, resulting inferior exploration efﬁciency and extra computation in path planning tasks.
Although these off-the-shelf algorithms are not directly applicable to our path planning setting, their successes show the importance of exploration-exploitation trade-off and will provide the principles for our algorithm for continuous state-action planning problems.
Planning networks Value iteration networks (Tamar et al., 2016)
employ neural networks to embed the value iteration algorithm from planning and then use this embedded algorithm to extract input features and deﬁne downstream models such as value functions and policies.
Speciﬁcally, VIN mimics the following recursive application of Bellman update operator G to value function V ∗ ,
[Equation] [Equation]
where P ( s (cid:48) | s, a ) is the state transition model.
When the state space for s and action space for a are low dimensional, these spaces can be discretized into grids.
Then, the local cost function c ([ s, s (cid:48) ]) and the value function V ∗ ( s (cid:48) | U ) can be represented as matrices (2d) or tensors (3d) with each entry indexed by grid locations.
Furthermore, if the transition model P ( s (cid:48) | s, a ) is local, that is P ( s (cid:48) | s, a ) =
for s (cid:48) / ∈ B ( s ) , it resembles a set of convolution kernels, each indexed by a discrete action a .
And the Bellman update operator essentially convolves P ( s (cid:48) | s, a ) with c ([ s, s (cid:48) ]) and V ∗ ( s (cid:48) | U ) , and then performs a min -pooling operation across the convolution channels.
Inspired by the above computation pattern of the Bellman operator, value iteration networks design the neural architecture as follows,
[Equation]
where ⊕ is the convolution operation, both map , ˜
V ∗ t and ˜ R are d × d matrices, and the parameter
W 1 are k c convolution kernels of size k × k .
The min implements the pooling across k c convolution channels.
The gated path planning networks (GPPN) (Lee et al., 2018) improves the VIN by replacing the VIN cell (9) with the well-established LSTM update, i.e. ,
[Equation]
where the summation is taking over all the k c convolution channels.
After constructing the VIN and GPPN, the parameters of the model, i.e. , (cid:110)
˜ R, W 1 (cid:111) can be learned by imitation learning or reinforcement learning.
The application of planning networks are restricted in low-dimension tasks.
However, their success enlightens our neural architecture for generalizable representation for high-dimension planning tasks.
C P ARAMETRIZED UCB A LGORITHMS
We list two examples of parametrized UCB as the instantiation of (3) used in GPE:
• GP-UCB : The GP-UCB Chu et al.
(2011) is derived by parameterizing via Gaussian Processes (GP) with kernel k ( s, s (cid:48) )
, i
[Equation]
e. , E
[ r ( s ) |T , U ] ∼ GP (0 , k ) , GP-UCB maintains an UCB of the reward after t -step
[Equation]
where
[Equation]
with k t ( s ) =
[ k ( s i , s )]
s i ∈S t , K t =
[ k ( s, s (cid:48) )] s,s (cid:48) ∈S t , and S t = { s 1 , s 2 , . . .
, s t } denotes the sequence of selected nodes in current trees.
The variance estimation σ 2 t ( s ) takes the number of visits into account in an implicit way: the variance will reduce, as the neighborhood of s is visited more frequently (Srinivas et al., 2009).
• KS-UCB : We can also use kernel regression as an alternative parametrization for (7) (Yee et al., 2016), which leads to an UCB of the reward after t -step as
[Equation]
where
[Equation]
with w ( s ) =
(cid:80) s (cid:48)
∈S t k ( s (cid:48) , s ) .
Clearly, the variance estimation is to promote exploration towards less frequently visited states.
As we can see, in both two examples of the parametrized UCB, we parametrize the observed rewards, leading to generalizable UCB for increased states by considering the correlations.
D P OLICY AND V ALUE N ETWORK A RCHITECTURE
We explain the implementation details of the proposed parametrization for policy and value function.
Figure 12 and Figure 13 are neural architectures for the attention module, the policy/value network, and the planning module, respectively.
[Figure]
In the ﬁgures, we use rectangle blocks to denote inputs, intermediate results and outputs, stadium shape blocks to denote operations, and rounded rectangle blocks to denote modules.
We use different colors for different operations.
In particular, we use blue for convolutional/LSTM layers, green for dense layers, and orange for anything else.
For convolutional layers, "Conv 1 × 1 , 32, relu" denotes a layer with 1 × 1 kernels, 32 channels, followed by a rectiﬁed linear unit; for dense layers, "Dense, 64, relu" denotes a layer of size 64, followed by a rectiﬁed linear unit.
The attention module (Figure 12-left) embeds a state to a d × d × d a tensor.
The planning module (Figure 13) is a one-step LSTM update which takes the result of a convolutional layer as input.
Both the input and hidden size of the LSTM cell are d e .
All d × d locations share one set of parameters and are processed by the LSTM in one batch.
The main architecture is illustrated in Figure 12-right.
It takes maze map, state and goal as input, and outputs the action and the value.
Refer to Section 4.2 for details for computing ψ ( s ) .
In our experiments, we set the values of the hyper-parameters to be ( d, d e , d a , p ) =
[Figure]
E E XPERIMENT D ETAILS E. B ENCHMARK E NVIRONMENTS
We used four benchmark environments in our experiment.
For the ﬁrst three, the workspace dimension is 2d.
We generated the maze maps with the recursive backtracker algorithm using the following implementation: https://github.com/lileee/gated-path-planning-networks/blob/master/generate_dataset.py.
Examples of the workspace are shown in Figure 15.
Three environments differ in the choice of robots:
• Workspace planning (2d).
The robot is abstracted with a point mass moving in the plane.
Without higher dimensions
this problem reduces to planning in
the workspace.
• Rigid body navigation (3d).
A rigid body robot, abstracted as a thin rectangle, is used here.
The extra rotation dimension is added to the planning problem.
This robot can rotate and move freely without any constraints in the free space.
• 3-link snake (5d).
The robot is a 5 DoF snake with two joints.
Two more angle dimensions are added to the planning task.
To prevent links from folding, we restrict the angles to the range of [ − π/ 4 , π/ 4] .
The fourth environment has a 3d workspace.
Cuboid obstacles were generated uniformly randomly in space with density ≈ 20% .
Example of the workspace is shown in Figure 6 and 16, where the blue cuboids are obstacles.
The environment is described below:
• Spacecraft planning (7d).
The robot is a spacecraft with a cuboid body and two 2 DoF arms connecting to two opposite sides of the body.
There is a joint in the middle of each arm.
The outer arm can rotate around this joint.
Each arm can also rotate as a whole around its connection point with the body.
All rotation
angles are restricted in the range of [0 , π/ 2] .
The spacecraft itself cannot rotate.
E. H YPERPARAMETER FOR MSIL
During self-improving over the ﬁrst 2000 problems, NEXT updated its parameters and annealed (cid:15) once for every 200 problems.
The value of the annealing (cid:15) was set as the following:
[Equation]
with i denoting the problem number.
E. B ASELINE : THE I MPROVED GPPN
The original GPPN is not directly applicable to our experiments.
Inspired by Tamar et al.
, we add a fully-connected MLP to its ﬁnal layers, so that the improved architecture can be applied to high-dimensional continuous domain.
As shown in Figure 14, the GPPN ﬁrst processes the discretized
[Figure]
workspace locations.
Its output and the full robot conﬁgurations are processed together by the MLP, which then produces the current value and action estimates.
The improved GPPN is trained using supervisions from the near-optimal paths produced by RRT*.
F E XPERIMENT R ESULTS F. S OLUTION P ATH I LLUSTRATION
F. D ETAILS OF Q UANTITATIVE E VALUATION
More detailed results are shown in Table 1, 2, 3, including learning-based and non-learning-based ones, on the last 1000 problems in each experiment.
We normalized the number of collision checks and the cost of paths based on the solution of RRT ∗ .
The success rate result is not normalized.
The best planners in each experiment are in bold .
NEXT-KS and NEXT-GP outperform the current state-of-the-art planning algorithm with large margins.
[Table]
[Table] [Table]
We demonstrated the performance improvement curves for 2d workspace planning, 3d rigid body navigation in Figure 17.
As we can see, similar to the performances on 5d 3 -link snake planning task in Figure
[Table]
in these tasks [Table] the NEXT-KS and NEXT-GP improve the performances along with more and more experiences collected, justiﬁed the self-improvement ability by learning ˜ V ∗ and ˜ π ∗ .
F. S EARCH T REES C OMPARISON
We illustrate the search trees generated by RRT* and the proposed NEXT algorithms withsamples in Figure 18, Figure 19, Figure 20 and Figure 21 on several 2d, 3d, 5d and 7d planning tasks, respectively.
To help readers better understand how the trees were expanded, we actually visualize the RRT* search trees without edge rewiring, which is equivalent to the RRT search trees, however the vertex set is the same.
Comparing to the search trees generated by RRT ∗ side by side, we can clearly see the advantages and the efﬁciency of the proposed NEXT algorithms.
In all the tasks, even in 2d workspace planning task, the RRT ∗ indeed randomly searches without realizing the goals, and thus cannot complete the missions, while the NEXT algorithms search towards the goals with the guidance from ˜ V ∗ and ˜ π ∗ , therefore, successfully provides high-quality solutions.
[Figure]
S u cc e ss r a t e
RRT*
RRT*-10k
NEXT-KS
NEXT-GP      6 Training phase
A v e r a g e c
o lli s i
o
n c
h e c k s
RRT*
RRT*-10k
[Figure]
-KS
[Figure]
-GP
[Figure]
A v e r a g e p a t h c o s t
RRT* RRT*
[Figure]
10k
[Figure] [Figure]
S u cc e ss r a t e
RRT* RRT*
[Figure] [Figure]
[Figure]
A v e r a g e c
o lli s i
o
n c
h e c k s
RRT*
[Figure] [Figure]
[Figure]
A v e r a g e p a t h c o s t
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
F. C ASE S TUDY D ETAILS
We conduct a real-world case study on controlling robot arms to move objects on a shelf.
This is a representative of common scenarios in practice where the robot needs to plan its motion in real-time to reach the inside of some narrow space.
For this case study, we focus more on the practical aspect to evaluate how much can we improve on the wall-clock time by learning from similar planning problems.
F.4. T ASK D ESCRIPTION
We generate planning problems randomly to form the training set and test set.
In each planning task, there is a shelf of multiple levels, with each level horizontally divided into multiple bins.
The heights of levels and widths of bins are randomly drawn from some ﬁxed distribution.
Samples of shelves are shown in Figure 22.
Both the start and goal conﬁgurations are randomly sampled from a distribution within the reachable region of the robot arm.
The planning environment is created with the OpenRave simulator (Diankov, 2010).
The task is to ﬁnd a path for the 7 DoF robot arm to move from a location in one bin to another, i.e. , the end effectors of the start and goal conﬁgurations are in different bins, as illustrated in Figure 23.
In this case, the base of the robot is ﬁxed and we are planning the movement of arm.
We generatedproblems for training and 1000 problems for testing.
F.4. B ASELINES AND T RAINING
For traditional planners, we include C++ OMPL (¸Sucan et al., 2012) implementation of BIT* (Gammell et al., 2015) and RRT* (Karaman & Frazzoli, 2011) as baselines.
The hyperparameters of
RRT* and BIT* are specially tuned for this experiment.
We also compare with learning-based planners CVAE-plan (Ichter et al., 2018) and Reinforce-plan (Zhang et al., 2018).
The supervisions for CVAE-plan are produced by the well-tuned BIT* on the training set.
To train NEXT, we consider the BIT* instead of RRT* as the imperfect expert in training problems.
F.4. R ESULTS
We evaluate the algorithms on the separated testing problems, and record the success rate usingdifferent time limits.
The success rate and average path quality are plot in Figure 8 and recorded in
Table 4 and Table 5.
The solution paths found by NEXT and BIT* are illustrated in Figure 23.
In terms of both success rate and solution path quality, NEXT dominates all the planners under all time limits.
[Table] [Table]