Learning to Select Nodes in Bounded Suboptimal Conﬂict-Based Search for Multi-Agent Path Finding
Abstract
Multi-Agent Path Finding is an NP-hard problem that is difﬁcult for current approaches to solve optimally.
Research has shown that bounded suboptimal solvers, such as Enhanced Conﬂict-Based Search (ECBS), are more efﬁcient than optimal solvers in ﬁnding a feasible solution with suboptimality guarantees.
ECBS is a tree search algorithm that repeatedly selects nodes from a focal list to expand the tree.
In this work, we propose to use imitation learning and curriculum learning to learn node-selection strategies for different grid maps and agent sizes.
We then deploy the learned models in ECBS and test their solving performance on unseen instances drawn from the same distribution as the one used in training.
Our approach shows substantial improvement over the baselines on different grid maps.
Introduction
Multi-Agent Path Finding (MAPF) is the problem of ﬁnding a set of conﬂict-free paths for a team of moving agents on a given graph that minimizes the sum of path costs or the makespan.
It has practical applications in video games [18], distribution centers [17, 11] and trafﬁc controls [7].
MAPF is NP-hard [25] and optimal solvers, such as Conﬂict-Based Search (CBS) [22], do not scale well.
Research has shown that Enhanced Conﬂict-Based Search (ECBS) [1] and its variants [4, 5] can ﬁnd bounded suboptimal solutions fast.
Based on CBS, ECBS uses focal searches [19] instead of best-ﬁrst searches to guarantee bounded suboptimality.
The high-level search of ECBS maintains a so-called focal list that contains a subset of nodes on the open list whose costs are at most w ≥ 1 times the current lower bound of the optimal cost.
From the focal list, ECBS could select an arbitrary node to expand but the common practice is to select one with the minimum d -value, which is a heuristic value computed for each node when it is generated and typically, a hard-coded function in the algorithm.
Instead of manually deﬁning d -values, we apply machine learning techniques and propose a novel data-driven framework for learning node-selection strategies for the high-level focal search.
We use imitation learning [6, 20, 21] and curriculum learning [2] to learn ranking functions from solving instances that can differentiate nodes that are closer to a solution within our desired suboptimality bound from those nodes that are far away from one.
During the search, the ranking function takes a node’s features as input and produces a real value as its d -value.
By using our d -values, we can get closer to a desired solution every time we expand a node and, therefore, ﬁnd the solution more quickly.
In experiment, we show that our approach can scale to large problem instances beyond those that are solvable by ECBS, the state-of-the-art bounded suboptimal MAPF algorithm.
MAPF and Related Work
Given an undirected unweighted graph G =
( V, E ) , the Multi-Agent Path Finding (MAPF) problem is to ﬁnd a set of conﬂict-free paths for a set of agents { a 1 , . .
, a k } .
Each agent a
i has a start vertex s i ∈ V and a goal vertex t i ∈ V .
Time is discretized into time steps, and, at each time step, every agent can either move to an adjacent vertex or wait at its current vertex.
A conﬂict occurs when two agents are at the same vertex at the same time or they traverse the same edge in opposite directions between two consecutive time steps.
The cost of an agent is deﬁned as the number of time steps until it reaches its goal and no longer moves.
Given w ≥ 1 , our goal is to ﬁnd a conﬂict-free solution with the sum of costs that is at most w times the minimum sum of agents’ costs.
CBS is a complete and optimal bi-level tree search on a binary search tree called constraint tree (CT).
On the high level, CBS starts with a root node with an empty set of constraints and expands the CT by always expanding a node N with the lowest cost N Cost .
After choosing N , CBS identiﬁes the set of conﬂicts N Conf in N .
If there are no conﬂicts, CBS returns the solution.
Otherwise, it picks one conﬂict to resolve and adds two child nodes of N to the CT by adding to the set of constraints at N a new constraint in one of two conﬂicting agents to one of the child nodes and the other conﬂicting agent in the other child node.
Then, we apply the low-level search to each child to re-plan the optimal path for the affected agent, and record the respective solution and its cost.
ECBS is a bounded-suboptimal version of CBS [1] that is guaranteed to ﬁnd a w -approximate solution for given w ≥ 1 .
ECBS uses focal searches [19] instead of best-ﬁrst searches in both levels.
Consider a node N .
On the low level, ECBS runs a focal search for each agent such that the cost of the path found is at most w times the cost of its individually cost-minimal path w.r.t.
the constraints at N .
Let N LB ,i be the lower bound on agent a i ’s cost given by the low-level search for a i when node
N is generated and let N LB = (cid:80) ki
=1 N LB ,i .
On the high level, ECBS performs a focal search that maintains a focal list that contains all nodes N in N such that N Cost ≤ w LB , where N is the open list and LB = min N ∈N N LB .
Once a solution is found by always expanding a node in the focal list, it is guaranteed to be a w -approximate solution.
Related Work Optimal MAPF solvers, such as mixed-integer programming(MIP)-based approaches [14, 15], and CBS [22] and its variants [3, 9, 16], have been developed.
Our work is more related to research on bounded suboptimal solvers.
ECBS [1] is the current state-of-the-art bounded suboptimal solver.
Variants of ECBS [5, 4] have been proposed to speed up ECBS to solve
MAPF in environments with certain structures such as warehouses.
Machine learning techniques have been applied to speed up branch-and-bound tree search.
[10] is a closely related work that uses imitation learning to learn node-selection and node-pruning strategies for solving MIP.
[23] scales up [10] by progressively increasing the instance sizes in the form of curriculum learning.
Other related work includes learning to branch [13] and run primal heuristics [12] in tree search for solving MIP.
Learning Node-Selection Strategies for ECBS
We introduce our framework for learning node-selection strategies for the high-level focal search of
ECBS.
Our framework consists of two phases, model learning and ML-guided search.
3.1 Model Learning
We focus on solving instances with ﬁxed graph G and w .
We ﬁrst learn node-selection strategies to solve instances with different numbers of agents { k 1 , . . .
k m } where k 1 < . . .
< k m .
For each k i , we learn a node-selection strategy that assigns π i ( φ ( N )) to node N as its d -value, where φ ( N ) is the feature vector of N and π
i is a learned ranking function.
Therefore, a desired ranking function would be one that assigns smaller d -values for nodes that are closer to a solution.
Our training algorithm is a curriculum learning algorithm, as shown in Algorithm 1, that takes { k 1 , . . .
k m } and m sets of training instances {I 1 , . . .
, I m } as input and outputs { π 1 , . .
π m } .
Each instance in I i includes k i agents and is i.i.d.
sampled from a given distribution.
Initially, π 0 is set to an initial strategy π ∗ (e.g., strategies used in previous work [1]) (line 2).
To obtain π 1 for k 1 agents, we deploy
DAgger ( π ∗ , I i ) [21] that learns a ranking function from instances in I i using π ∗ as a starting point.
To obtain π i ( i > 1) , instead of starting from π ∗ again, we start learning from π
i − 1 .
We obtain π i ( 1 < i ≤ m ) iteratively by calling DAgger ( π i − 1 , I i ) that learns a ranking function using π i − 1 as the initial node-selection strategy (line 3-4) until a stopping criterion is met (line 5-7) or i = m .
If the stopping criterion is met before i reaches m , we terminate training (line 7) and simply set π j to π i for all i < j ≤ m (line 6).
DAgger ( π (0) , I ) is shown in Algorithm 2.
The input π (0) and I are the initial ranking function and the set of training instances.
Initially, the dataset D is set to ∅ (line 1).
The algorithm runs for R
1: D = ∅ 2: for i = 1 to R do 3: for I instance set I do 4:
D ← D ∪ CollectData (
−
←
D 6: return iterations (line 2).
In iteration i , it collects training data from solving instances in I using the ranking function π
( i − 1) obtained in iteration i − 1 , aggregates it with D (line 4), and learns a new ranking function π ( i ) from D that minimizes a loss function over D (line 5).
Finally, the algorithm returns the best ranking function (line 6) (returns π (0) if it cannot ﬁnd a better ranking function than π (0) ).
Data Collection Given an instance I and a ranking function π , we run ECBS using ranking function π on I and record the entire tree T .
For each node N ∈ T , we compute a set of nine atomic features { f 1 , . . .
, f 9 } .
The full list of features is deferred to Appendix A. We also obtain the set of interaction features { f i f
j : i ≤ j } and the ﬁnal feature vector φ ( N ) ∈
R p is obtained by concatenating all atomic features and interaction features.
During data collection, we run ECBS with a runtime limit to collect T solutions.
If the search exceeds the runtime limit without any solution, we return an empty set of data.
Otherwise, for each node N , we assign a label y N to it based on the minimum distance
N d between N and any solution found within the subtree rooted at N ( N d = ∞ if no solution found) as follows:
if N d < 10 ; 1 if 10 ≤ N d < 30 ; 2 if 30 ≤ N d < 60 ;
if 60 ≤ N d < ∞ ; 4 otherwise.
Learning a Ranking Function We focus on learning a linear ranking function π : R p
→ R : π ( φ ( N ))
=
w T φ ( N ) with parameter w ∈ R p that minimizes the loss function L ( w ) =
(cid:80)
T ∈
D l ( y T , ˆ y T )
+ C 2 ||
w || 22 over training data D , where y T is the ground-truth label vector of all nodes in T , ˆ y T is the vector of predicted values resulting from applying π to the feature vector φ ( N ) of every node N ∈ T , l ( · , · ) is a loss function measuring the difference between y T and ˆ y T , and
C > 0 is a regularization parameter.
The loss function l ( · , · ) is based on a weighted pairwise loss.
Speciﬁcally, we consider the set of ordered node pairs P T
= { ( N i , N j
) ∈ Q T : y N i > y N j } where
Q T is the set of node pairs ( N i , N j ) such that neither N i nor N j is an ancestor of the other in T .
The weight w N i ,N j of each pair ( N i ,
N j ) ∈
P T is set to e − ( d i + d j ) /rd max , where d i , d j are the depths of N i , N j in T respectively, d max is the maximum depth of T and r is a damping factor.
The loss function l ( · , · ) is the weighted fraction of swapped pairs, deﬁned as
[Equation]
3.2 ML-Guided Search
After learning ranking functions { π i : i ∈ [ m ] } , we deploy them in ECBS.
Given an instance with the same underlying graph as training and k agents, we run ECBS with ranking function π j where j ∈ arg min i ∈ [ m ] {| k − k i |} .
When a node N is generated, we compute the feature vector φ ( N ) and set its d -value to π j ( φ ( N )) .
The overall complexity for computing the d -value is O
( | N Conf | ) .
Experimental Results
In this section, we demonstrate the efﬁciency and effectiveness of our solver, ECBS with ML-guided node selection.
We denote our solver as ECBS+ML.
We deﬁne h 1 , h 2 and h 3 to be the node-selection strategies that use the number of conﬂicts, the number of conﬂicting pairs of agents and the number of conﬂicting agents as the d -values, respectively.
We compare against ECBS using h i and denote them as ECBS+ h
i ( i = 1 , 2 , 3 ).
We evaluate our approach on a random map, a warehouse map and a game map from the MAPF benchmark [24].
We report other experiment setups in Appendix B.
Figure 1 plots the success rates (the percentages of solved instances within the runtime limit ofminutes) on all grid maps.
Overall, ECBS+ML signiﬁcantly outperforms the three baselines in all grid maps.
On the game map, in particular, the success rates of the baselines drop below 20% when the number of agents increases to 170 and the baselines could hardly solve instances with more than 245 agents, while the success rates of ECBS+ML stay above 76% for as many as 245 agents and ECBS+ML can solve 16% of the instances with 305 agents.
To demonstrate the efﬁciency of
ECBS+ML further, we show the success rates for different runtime limits on each map in Figure 2.
We show one ﬁgure for each map with a ﬁxed number of agents and defer the rest to Appendix C.
Typically, the baselines and ECBS+ML tie on the easy instances but ECBS+ML gain an advantage on the hard ones, even more for instances with larger numbers of agents (as shown in Appendix C).
To assess the effect of curriculum learning, we perform two ablation analyses.
First, we experiment with ECBS+ML with early stopping, denoted as ECBS+ML(ES).
The number of agents in the last iteration of training in Algorithm 1 is set to the one where the success rate of ECBS+ h 1 ﬁrst drops below 60%.
The success rates of ECBS+ML(ES) are shown in Figure 1.
ECBS+ML(ES) is competitive with ECBS+ML and outperform all baselines on the random and warehouse maps, but its success rates on the game map drop dramatically after early stopping.
The results imply that the learned strategies do not generalize well to larger numbers of agents in some maps and curriculum learning can help ﬁnd improved strategies for those cases.
Then, we experiment with ECBS+ML using only imitation learning, denoted as ECBS+IL.
ECBS+IL uses the same training algorithm as ECBS+ML except that, for each number of agents, it learns a ranking function starting from the given initial ranking function for each agent size without relying on the previously-learned one.
Overall, ECBS+IL can still outperform the baselines but not as signiﬁcantly as ECBS+ML and its performance drops faster than ECBS+ML when the number of agents increases.
The results show another two advantages of curriculum learning: (1) it enables learning for one to three more iterations than ECBS+IL by enabling the algorithm to collect more data for training since it starts with a higher success rate in DAgger ( · , · ) ; (2) it obtains better node-selection strategies based on previously-learned strategies, as opposed to ECBS+IL that learns from scratch in every iteration.
A Feature List
For each node N ∈ T , we compute the following set of atomic features { f 1 , . . .
, f 9 } :
Features related to the conﬂicts: number of conﬂicts | N Conf | (denoted as f 1 ), number of pairs of agents that have at least one conﬂict (denoted as f 2 ) and number of agents that have at least one conﬂict (denoted as f 3 ); 2.
Features related to N Cost : f 4 := N Cost , f 5 :=
N Cost LB , f 6 :=
N Cost − LB , f 7 := N Cost − S , f 8 := N Cost /S where S is the sum of costs of individually cost-minimal paths over all agents.
The depth of N (denoted as f 9 ).
B Experiment Setup
We implement ECBS in C++.
The experiments are conducted on 2.4 GHz Intel Core i7 CPUs with 16GB RAM.
For each instance, the runtime limit is set to 5 minutes in both data collection and testing.
The number of solutions T collected during data collection is set to 10 .
The number of iterations R for DAgger is set to 10.
The damping factor r for weight w N i ,N j is set to 0 .
r is chosen such that the root node has weight 1 and a node at depth 0 .
6 d max has weight e − 0 .
6 /r = 0 .
Since we are using a pairwise loss, we suffer quadratic complexity ( O ( |T | 2 ) ) of the loss computation.
To tackle this issue, we record only the ﬁrst 10,000 nodes generated for each instance during data collection.
We use an open-source ML package LIBLINEAR [8] in the step of training a ranking function in Algorithm 2 and use the default values for all parameters in LIBLINEAR where
C = 1 .
The performance of our algorithms is not sensitive to hyperparameters and all these parameters are chosen without ﬁne-tuning.
We evaluate our algorithms on three grid maps of different sizes and structures from the MAPF benchmarks 1 [24], including: (1) a random map “random-32-32-20”, a 32 × 32 grid map with 20% randomly blocked cells; (2) a warehouse map “warehouse-10-20-10-2-1”, a 163 × 63 grid map with 200 10 × 2 rectangle obstacles; (3) a game map “den520d”, a 257 × 256 grid map from the video game Dragon Age: Origins .
For testing, we use the “random” scenarios in the benchmarks, yielding 25 instances for each number of agents on each grid map.
For training, we generate anotherinstances drawn from the same distribution as the “random” scenarios for each I i in Algorithm 1.
Parameters related to each individual grid map are listed in Table 2.
For each grid map, we ﬁx the suboptimality factor w , following similar reasonings in previous work [1] where small w values are chosen for large grid maps and vice versa.
Our goal is to obtain a ranking function π i ( i ∈ [ m ]) for each number of agents in { k 1 , . . . , k m } .
The training loss of the learned ranking functions are shown in Table 2, showing good ML performance of the learned ranking functions.
We test the learned strategies on unseen instances with numbers of agents k 1 , . . .
, k m , and report the improvement in solution performance, which is the end goal of this work.
C Additional Experimental Results
The success rates for different runtime limits varying the number of agents on the random, warehouse and game maps are shown in Figure 3.
[Table] [Table]
[Table] [Table]