L EARNING TO S OLVE M ULTI -R OBOT T ASK A LLOCATION WITH A C OVARIANT -A TTENTION BASED N EURAL A RCHITECTURE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper demonstrates how time-constrained multi-robot task allocation (MRTA) problems can be modeled as a Markov Decision Process (MDP) over graphs, such that approximate solutions can be modeled as a policy using Reinforcement
Learning (RL) methods.
Inspired by emerging approaches for learning to solve related combinatorial optimization (CO) problems such as multi-traveling salesman (mTSP) problems, a graph neural architecture is conceived in this paper to model the MRTA policy.
The generalizability and scalability needs of the complex CO problem presented by MRTA are addressed by innovatively using the concept of
Covariant Compositional Networks (CCN) to learn the local structures of graphs.
The resulting learning architecture is called Covariant Attention-based Mechanism or CAM, which comprises: 1) an encoder: CCN-based embedding model to represent the task space as learnable feature vectors, 2) a decoder: an attentionbased model to facilitate sequential decision outputs, and 3) context: to represent the state of the mission and the robots.
To learn the feature vectors, a policy-gradient method is used.
The CAM architecture is found to generally outperform a stateof-the-art encoder-decoder method that is purely based on Multi-head Attention (MHA) mechanism in terms of task completion and cost function, when applied to a class of MRTA problems with time deadlines, robot ferry range constraints, and multi-tour allowance.
CAM also demonstrated signiﬁcantly better scalability in terms of cost function over unseen scenarios with larger task/robot spaces than those used for training.
Lastly, evidence regarding the unique potential of learningbased approaches in delivering highly time-efﬁcient solutions is provided for a benchmark vehicle routing problem – where solutions are achieved 100times faster compared to a non-learning baseline, and for a benchmark MRTA problem with time and capacity constraints – where solutions for larger problems are achieved 10 times faster compared to non-learning baselines.
I NTRODUCTION
In multi-robot task allocation (MRTA) problems, we study how to coordinate tasks among a team of cooperative robotic systems such that the decisions are free of conﬂict and optimize a quantity of interest (Gerkey & Matari ´ c, 2004).
The potential real-world applications of MRTA are immense, considering that multi-robotics is one of the most important emerging directions of robotics research and development, and task allocation is fundamental to most multi-robotic or swarm-robotic operations.
Example applications include disaster response (Ghassemi & Chowdhury, 2018), last-mile delivery (Aurambout et al., 2019), environment monitoring (Espina et al., 2011), reconnaissance (Olson et al., 2012) and combat Behjat et al.
Although various approaches (e.g., graph-based methods (Ghassemi et al., 2019), integer-linear programming (ILP) approaches (Nallusamy et al., 2009; Toth & Vigo, 2014), and auction-based methods (Dias et al., 2006; Schneider et al., 2015)) have been proposed to solve the combinatorial optimization problem underlying MRTA operations, they usually do not scale well with number of robots and/or tasks, and do not readily adapt to complex problem characteristics without tedious hand-crafting of the underlying heuristics.
In the recent years, a rich body of work has emerged on using learning-based techniques to model solutions or intelligent heuristics for combinatorial optimization (CO) problems over graphs.
The existing methods are mostly limited to classical CO problems, such as multi-traveling salesman (mTSP), vehicle routing (VRP), and max-cut type of problems.
We speciﬁcally focus on a class of MRTA problems that falls into the Single-task Robots, and Single-robot Tasks (SR-ST) class deﬁned in (Gerkey & Matari´c, 2004; Nunes et al., 2017).
Based on iTax taxonomoy as deﬁned in Gerkey & Matari´c (2004), these problems fall into the In-schedule Dependencies (ID) category.
Here, a feasible and conﬂict-free task allocation is deﬁned as assigning any task to only one robot (Ghassemi et al., 2019).
For solving these problems, we propose a new covariant attention-based model (aka CAM ), a neural architecture for learning over graphs to construct the MRTA policies.
This architecture builds upon the attention mechanism concept and innovatively integrates an equivariant embedding of the graph to capture graph structure while remaining agnostic to node ordering.
We implement CAM on an original
MRTA problem, a suite of benchmark MRTA problems and benchmark VRP problems to perform generalizability, scalability and comparative analyses of the new method.
We also perform an analysis on the impact of the neighborhood size on the performance on the benchmark MRTA problem.
1. M ULTI -R OBOT T ASK A LLOCATION
In recent years, learning approaches based on Graph Neural Networks or GNN are being increasingly used to solve planning problems with a CO formulation, e.g., TSP, VRP, Max-Cut, Min-Vertex, and
MRTA Kool et al.
; Barrett et al.
; Khalil et al.
; Kaempfer & Wolf (2018); Mittal et al.
; Li et al.
; Nowak et al.
; Wang & Gombolay (2020);
Tolstaya et al.
Sykora et al.
Dai et al.
Further details on these related learning-based studies can be found in Appendix A. Some of the conventional ILP, MILP, and INLP based methods for MRTA have been discussed in Appendix D.1.
GNNs provide the advantage of being able to capture both
Euclidean features (e.g., task location), as well as non-Euclidean features such as task capacity, task deadline and the local structure of task neighborhoods.
The latter serves as higher-level meaningful features that assist in generalized decision-making These existing studies are however limited in three key aspects: 1) They address simpliﬁed problems that often exclude common real-world factors such as resource and capacity constraints Kool et al.
; Kaempfer & Wolf (2018); Khalil et al.
(2017); Tolstaya et al.
They are mostly focused on smaller sized problems ( ≤tasks and 10 robots)
Paleja et al.
; Strens & Windelinckx (2005); Wang & Gombolay (2020);
Sykora et al.
(2020), with their scalability remaining unclear.
They rarely provide evidence of generalizing to problem scenarios that are larger in size than those used for training.
This capability would be particularly critical since real-world MRTA problems often involve simulating episodes whose costs scale with the number of tasks and robots, making re-training efforts burdensome.
To address these gaps, here we propose a new learning framework that can solve large-sized MRTA problems (SR-ST) with commonly considered constraints – involving up to 1000+ tasks and 200+ robots – and generalize across even larger problem scenarios without the need to re-train.
For most practical scenarios with larger number of locations, a highly optimal solution is not always desired, while a good feasible solution is the priority as pointed out by Cappart et al.
Therefore, to enable scalable policies, we design a novel encoder based on the concept of Covariant Compositional
Networks (CCN)
Hy et al.
(2018), which is hypothesized to effectively combine local structural information with permutation invariance.
The encoder is followed by a decoder based on a Multi-head
Attention mechanism Kool et al.
; Vaswani et al.
(2017) which fuses the encoded information and problem/mission-speciﬁc information (Context) using simple matrix multiplication, in order to enable decentralized sequential decision-making.
1. C ONTRIBUTIONS OF THIS P APER
The primary contributions of this paper can thus be stated as follows: 1) We formulate the general
SR-ST class of MRTA problems as a Markov Decision Process or MDP over graphs with the multirobot state information embedded as the context portion of the policy model, such that the (task allocation) policy can be learned using an RL approach.
We design the GNN that acts as the policy network as an encoder-decoder architecture, where the encoder is innovatively based on covariant compositional networks (CCN), whose embedding capabilities signiﬁcantly improve generalizability and scalability to larger task graphs and multi-robot teams.
We implement an attention based decoder (inspired by Kool et al.
to enable sequential decision-making, and speciﬁcally extend it to a multi-agent combinatorial optimization setting.
The proposed CAM architecture is evaluated on a representative MRTA problem that involves coordinating a team of unmanned aerial vehicles (UAVs) to time-efﬁciently deliver ﬂood relief.
The results of this case study demonstrate how CAM clearly outperforms the state-of-the-art attention based method AM (Kool et al., 2019), in terms of scalabilty and convergence, thereby emphasizing the effectiveness of the new encoder.
Further case studies show that CAM continues to compare favorably to AM over benchmark MRTA problems (with time and capacity constraints) and CVRP problems.
Comparisons to non-learning baselines for these benchmark problems demonstrate the signiﬁcant online computation advantages of learnt policies, with the latter being 10-100 times faster.
The remainder of the paper is organized as follows:
Section 2 deﬁnes the MRTA problem and its formulation as an MDP over graphs.
Section 3 presents our proposed new GNN architecture.
Section 4 describes simulation settings and different case studies.
Results are discussed in Section 5.
MRTA: P ROBLEM D EFINITION AND F ORMULATIONS
The multi-robot task allocation (MRTA) problem is deﬁned as the allocation of tasks and resources among several robots that act together without conﬂict in the same environment to accomplish a common mission.
The optimum solution (decision) of the MRTA problem is a sequence of tasks for each robot (conﬂict-free allocation) that maximizes the mission outcome (e.g., fraction of tasks completed) or minimize the mission cost (e.g., total distance travelled) subject to the robots’ range constraints.
Here, the following assumptions are made: 1) All robots are identical and start/end at the same depot; 2) There are no environmental uncertainties; 3)
The location ( x i , y i ) of task- i and its time deadline τ i are known to all robots; 4) Each robot can share its state and its world view with other robots; and 5) There is a depot (Task-0), where each robot starts from and visits if no other tasks are feasible to be undertaken due to the lack of available range.
Each tour is deﬁned as departing from the depot, undertaking at least one task, and returning to the depot.
Motivated by the multi-UAV relief delivery problem, tasks are considered to be instantaneous, which means that reaching the waypoint associated with a task completes that task.
This MRTA problem is a class of combinatorial optimization problems, which can be modeled in graph space.
In order to learn policies that yield solutions to this CO problem, we express the MRTA problem as a Markov Decision Process (MDP) over a graph, described next.
The optimization formulation of MRTA is then given in Section 2.2.
2. MDP OVER A G RAPH
The MRTA problem involves a set of nodes/vertices ( V ) and a set of edges ( E ) that connect the vertices to each other, which can be represented as a complete graph G =
( V, E ) .
Each node represents a task, and each edge connects a pair of nodes.
Let Ω be a weight matrix where the weight of the edge ( ω ij ∈ Ω ) represents the cost (e.g., distance) incurred by a robot to take task- j after achieving task- i .
For MRTA with N tasks, the number of vertices and the number of edges are N and N ( N − 1) / 2 , respectively.
Node i is assigned a 3-dimensional feature vector denoting the task location and time deadline, i.e., d i =
[ x i , y i , τ i ] where i ∈ [1 , N ] .
Here, ω ij can be computed as ω ij = (cid:112) ( x i − x j )
+ ( y i − y j ) 2 , where i, j ∈ [1 , N ] .
The MDP deﬁned in a decentralized manner for each individual robot (to capture its task selection process) can be expressed as a tuple < S , A , P a , R > .
The components of the MDP can be deﬁned as: State
Space ( S ):
A
robot
at its decision-making instance uses a state s ∈ S , which contains the following information: the
current mission time,
its current location,
its remaining ferry-range
battery state), 4) the planned (allocated) tasks of its peers,
the
remaining ferry-range
of its
peers,
and 6) the states of tasks.
The states of tasks contain the location, the time deadline, and the task
status – active, completed, and missed (i
.e.
, deadline is passed)
Here we assume that each robot can broadcast its information to its peers without the need for a centralized system for communication, as
aligned
with modern communication capabilities Sykora et al
Action Space ( A
The set of actions is
represented
as A , where each action a is deﬁned as the index of the selected task, {
, N
} with
the index of the
depot
as 0 .
The task 0 (the depot) can be selected by multiple robots, but the other tasks are allowed to be chosen once if they are active (not completed or missed tasks).
P a ( s (cid:48) | s, a ) :
A robot by taking action a at state s reaches the next state s (cid:48)
in a deterministic manner (i.e., deterministic transition model is deﬁned).
Reward ( R ): The reward function is deﬁned as − f cost , and is calculated when there is no more active tasks (all tasks has been visited once irrespective of it being completed or missed).
Transition:
The transition is an event-based trigger.
An event is deﬁned as the condition that a robot reaches its selected task or visits the depot location.
2. MRTA AS O PTIMIZATION P ROBLEM
This MRTA problem is adopted from (Ghassemi et al., 2019; Ghassemi & Chowdhury, 2021) with the following modiﬁcation – payload constraints are not imposed on the robot.
The exact solution of the MRTA problem can be obtained by formulating it as an integer nonlinear programming problem, which can be summarily expressed as:
[Equation]
where r ∈ [0 , 1] and u ( r ) = (cid:26) 1 if r =0 otherwise (
[Equation]
subject to
[Equation]
Here τ fi is the time at which task i was completed, ∆ k is the available range for robot k at any point of time, and δ ij is the distance between nodes i and j .
A detailed formulation of the exact
ILP constraints that describe the MRTA problem with range restrictions, multi-tours per robot and tasks with deadlines, can be found (Ghassemi & Chowdhury, 2021).
Note that, here we use a slightly different objective/cost function to better reﬂect the generalized setting for the class of MRTA problems with ferry-range and task-deadline constraints For compactness of representation, only the main constraints involved in the studied MRTA problem are shown in the above set of equations.
We however consider all of the constraints, except the one related to payload capacity, as deﬁned in the work by (Ghassemi & Chowdhury, 2021); for detailed formulation of the MRTA problem, please refer to (Ghassemi & Chowdhury, 2021).
Here, we craft the objective function (Eq.
equation 1)
such that it emphasises maximizing the completion rate (i.e., the number of completed tasks divided by the total number of tasks); and if perfect completion rate (100%) is feasible, then the travelled cost is also considered.
The term of 1 − r is deﬁned as task completion rate; i.e., the number of completed tasks ( N success ) divided by the total tasks ( N ) or r = ( N − N success ) /N .
Here, d r is a normalized value of the total distance travelled by all robots in the team.
The term d r is the average travelled distance over all robots (i.e., d r = (cid:80) N r i =
d total i / ( √ 2 N ) ).
The terms N r and d total i represent respectively the number of robots and the total traveled distance by robot- i during the entire mission.
The above objective function (Eq.
equation 1) gives a positive value if the completion rate is lower than 100%, otherwise it gives a negative value.
Equation 1 ensures that the objective function is bounded in the range ( − 1 , 1] .
C OVARIANT A TTENTION - BASED N EURAL A RCHITECTURE
For learning to work on the MDP deﬁned over graphs in Section 2.1, we need to represent each node as a continuous vector, preserving its properties as well as the the structural information of the neighborhood of that node.
[Figure]
Before describing the technical components of our proposed Covariant
Attention Mechanism, the so-called
CAM neural architecture, we provide an illustration and summary description here of how this policy architecture is used by robots or agents during an SR-ST operation.
The CAM model for task allocation is called by/for each robot right when it reaches its current destination (task location or depot), in order to decide its next task or destination.
While robots are moving toward their selected task locations, they can check if their decision is conﬂicting with another robot based on recent information.
If there is conﬂict, the robot with the worst time can cancel its current task and select a new task.
Since full-observability is assumed across the multi-robot team and the policy-model execution-time is almost negligible, the current setup is agnostic to whether the online CAM model is executed centrally off-board or on-board each robot.
As an example, Figure 1 illustrates how robot-1 and robot-2 uses the CAM policy model to choose a task at two different decision-making instances ( t = t 0 and t = t 1 ).
Here, the inputs to the CAM model includes 1) the task graph information, i.e., the properties of all the tasks/nodes d i and the computed weight matrix Ω , 2) the current mission time, 3) the state of robot- r , and 4) the states of robot- r ’s peers.
The CAM model then generates the probability of selecting each task as its output.
A greedy strategy of choosing the task with the highest probability is used here, which thus provides the next destination to be visited by that robot.
It should be noted that the probability values for completed tasks and missed tasks (i.e., missed deadline) are set at 0.
Figure 2 shows the detailed architecture of CAM.
As shown in this ﬁgure, the CAM model consists of three key components, namely: Context , Encoder , and Decoder .
The context includes the current mission time, the states of robot- r , and the states of robot- r ’s peers.
The state of a robot consists of it’s destination x, y coordinates and the available range ρ .
The encoder and decoder components are described below.
[Figure]
For learning over graphs, the performance of the trained model depends mostly on the ability of the Graph Neural Network (GNN) to transform all the required node information into a feature vector or tensor.
For our case, apart from the node properties, some of the features that is essential include a node’s local neighborhood information, and permutation invariance.
Using a node’s local neighborhood information which consists of its association with its local neighbors during training is more beneﬁcial than considering the association with the entire graph, for generalizing to unseen nodes as demonstrated by frameworks like GraphSAGE (Hamilton et al., 2017), thus enabling the
GNN to generalize for problems with larger number of nodes without the need to re-train.
The encoder represent the properties of each graph node (preserving its structural information) into a continuous feature vector of dimension d embed , which is fed to the decoder.
Each node i , has three properties which are the x-coordinate ( x ), y-coordinate ( y ), and the time deadline ( τ ) of the task.
Note that our encoding mechanism can also be extended to a probabilistic scenario, for example where an estimated deadline τ follows a probability distribution (common in disaster response type operations).
The encoding for each node should include its properties and the its positional association with its neighboring nodes.
We implement a variation of CCN (Hy et al., 2018).
We determine the nearest k neighbors of a node ( Nb i ) based on the positional coordinates ( x and y ).
The ﬁrst step is to compute a feature vector by linear transformation for each node i .
To encode the node properties, we do a linear transformation of d i to get a feature vector F d i for all i ∈ [1 , N ] , i.e.,
F d i =
[Equation]
Here W d ε R d embed × 3 , b d ε R d embed × 1 , d i =
[ x i , y i , τ i ] .
For effective decision making, we also need to preserve the structural information.
Therefore we deﬁne a matrix F Nbd i as in equation 5.
[Equation]
We compute a matrix F Nbi (as shown in Eq.
equation 6), which we believe captures the association of a node with its local neighbours (one-hop neighbors) in terms of the node properties.
[Equation]
where W Nb ε R d embed × d embed , b Nb ε R d embed × 1 .
F Nbi captures the information about how close the node properties of neighbor nodes of node i is to itself, which shows a representation of how important node i is to its neighbors.
Further explanation regarding this design choice is discussed in Appendix
B and we strongly encourage the reader to go over the section.
We compute the ﬁnal embedding for each node using Eq.
equation 7.
[Equation]
Here, W f ε R d embed × d embed , b f ε R d embed × 1 .
Thus ﬁnally we get an embedding F i for each node, where
F i ε R d embed × 1 .
W d , b d , W Nb , b Nb , W f , and b f are learnable weights and biases.
The Aggregate function is the summation across all the columns of a matrix.
This summation along with the relative difference in node properties, as in Eq.
equation 6, preserves permutation-invariance and the structural properties (cognizance of inter-node distances for example) of the graph.
Note that, these operations make the encoded state w.r.t.
a given node insensitive to the order of the neighboring nodes, and thus the overall state space becomes independent of the indexing of tasks or to graph rotations.
Equations 5, 6, and 7 represents a single layer of encoding.
Multiple layers of encoding can be performed with the output of the previous layer being the inputs to equations 5 and 6 in the next layer.
3. A TTENTION - BASED D ECODER AND C ONTEXT
The main objective of the decoder is to use the information from the encoder and the current state as context or query, and thereof choose the best task by calculating the probability value of getting selected for each (task) node.
In this case, the ﬁrst step is to feed the embedding for each node (from the encoder)
as key-values ( K , V ).
The key K and value V for each node is computed by two separate linear transformations of the node embedding obtained from the encoder.
The next step is to compute a vector representing the current state, also known as the context (as shown in bottom left of Fig. 2).
The context for the MHA layer in this experiment consist of the following seven features: 1) Current time; 2) Available range of the robot taking decision; 3) Current location of robot taking decision; 4)
Current destination of other robots; 5) Available range for other robots all concatenated to a single vector of length h q , which then undergoes a linear transformation to get a vector of length d embed also called the query Q .
Figure 2 illustrates the structure of the decoder.
Now the attention mechanism can be described as mapping the query ( Q ) to a set of key-value ( K , V ) pairs.
The inputs, which are the query ( Q ) is a vector, while K and V are matrices of size d embed × N (since there are N nodes).
The output is a weighted sum of the values V , with the weight vector computed using the compatibility function expressed as:
[Equation]
Here h l is the dimension of the key of any node i ( k i ∈ K ).
In this work, we implement a multi-head attention (MHA) layer in order to determine the compatibility of Q with K and V .
The MHA implemented in this work is similar to the decoder implemented in Kool et al.
(2019) and Vaswani et al.
As shown in (Vaswani et al., 2017) the MHA layer can be deﬁned as:
[Equation]
Here head i =
Attention ( K , V , Q ) and h e (taken as 8 here) is the number of heads.
The feed-forward layer is implemented to further process the mapping that results from the MHA layer, and transform it to a dimension that is coherent with the number of nodes in the task-graph ( N ).
The interjecting batch normalization layers serve to bound values of a speciﬁc batch using the mean and variance of the batch.
The ﬁnal softmax layer outputs the probability values for all the nodes.
Here, the next task to be done is then chosen based on a greedy approach, which means that the node with the highest probability will be chosen.
The nodes which are already visited will be masked (by setting its probability as 0 ) so that these nodes are not available for selection in the future time steps of the simulation of the multi-robot operation.
C ASE S TUDIES
We design and execute a set of numerical experiments described in Appendix D.3, to investigate the performance of our proposed learning-based algorithm over graph space ( CAM ) and compare it with 1) an extended version of a state-of-the-art graph learning-based algorithm proposed by (Kool et al., 2019), so called attention-based mechanism ( AM ) approach; 2) a recent bipartite graph matching method BiG-MRTA
(Ghassemi & Chowdhury, 2021); 3) a myopic baseline called Feasibilitypreserving Random-Walk (Feas-RND) that takes randomized but feasible actions (avoiding conﬂicts and satisfying other problem constraints).
Since the BiG-MRTA method has been shown to provide near-optimal solutions in comparison to ILP (and competitive performance w.r.t.
state-of-the-art online MRTA methods)
Ghassemi & Chowdhury (2021), it is used here to comparatively gauge performance of AM and CAM.
The Feas-RND method on the other hand provides a baseline that
AM and CAM should clearly surpass in performance (cost function), in order to demonstrate that meaningful MRTA policies are being learnt as opposed to simply mapping random feasible actions.
R ESULTS AND D ISCUSSION 5. G ENERALIZABILITY AND S CALABILITY ANALYSIS OF CAM
The CAM model has been trained on scenarios with 200 tasks and varying robot size (randomly a robot size between 10 and 50 has been selected).
Then, 100 test scenarios have been generated per robot-task size from the same distribution of training scenarios.
In this paper, generalizability refers to the performance of the trained model on unseen test scenarios that involve the same (or lower) number of tasks as in the scenarios used for training; and where the test and training scenarios are drawn from the same probability distribution over task locations and deadlines.
In this work, generalizability was analysed on test scenarios with the number of tasks ﬁxed at 50 and 200, drawn from the same distribution over a 2D space, and number of robots ﬁxed at 5 and 40.
In this paper, scalability refers to the performance of the trained model over test scenarios with higher (and increasing) numbers of tasks and robots than that encountered in the scenarios used for training.
Here, we analyze scalability by evaluating the CAM model on test scenarios with the number of tasks ﬁxed at 500 and 1000, and the number of robots ﬁxed at 50 and 1000.
The task-to-robot ratio is however kept the same across the generalizability and scalability analysis cases, in order not to introduce another control factor affecting the numerical experiments.
To measure and compare performance, we use two metrics: 1) Average cost function (Eq. 1): This metric accounts for the completion rate of tasks and the total travelled distance, averaged over the set of test scenarios; and 2) Average computing time:
This measures how long each method takes to compute the entire solution, averaged over the test set.
The latter is particularly important to note in scalability analysis, since state-of-the-art non-learning based MRTA methods scale poorly in terms of computing efﬁciency as numbers of tasks and robots increases.
[Table]
Table 1 shows the cost function (the lower the better) for the unseen test scenarios for both
AM and CAM.
Note that the cost function (as deﬁned by Eq.1) for MRTA produces a negative value when the completion rate is 100%, while a positive value indicates a task completion rate below 100%.
As it can be seen from
Table.
1 that the proposed CAM approach outperforms the AM approach in all the test cases by achieving better mean values of the cost function, respectively.
The CAM approach performs signiﬁcantly better than AM in terms of the cost function for the lower task-to-robot ratio (here, 5).
Based on table 1, the proposed CAM approach achieved a perfect completion rate for most of scenarios with the task-to-robot ratio of 5.
The local structure of the graph is not only important for effective decision-making, but also expected to be shared across various problems settings drawn from the same distribution, thereby promoting generalizability of policies when adequately captured.
[Table]
[Table]
To investigate the scalability of the learnt model, we use a new set of unseen test scenarios with numbers of tasks and robots much larger than those of the scenarios used in training (the latter involved 200 tasks).
Table 2 shows the performance of the trained model of CAM and AM in terms of the cost function (lower the better) for four large case studies, involving 500-tasks-50-robots, 500-tasks-100-robots, 1000-tasks-100-robots, and 1000-tasks-200-robots; for each case, 100 randomly generated scenarios (i.e., task locations and deadlines randomized) are used.
As shown in Table 2, the proposed CAM method outperforms the AM method in all cases, with a signiﬁcant difference in the cost function for the case studies with the task-to-robot ratio of 5.
In the largest case (i.e., 1000-task-200-robot), the learnt model by CAM achieved a perfect completion rate for most of the scenarios (Appendix E).
It can be argued that, for time-critical problems such as MRTA in disaster response, generating an optimal solution is less of a priority, compared to generating a feasible near-optimal solution as quickly as possible; this capability of the learnt models in evident from the results in Table 7 in Appendix E. BiG-MRTA requires a signiﬁcantly larger computing time for larger sized problem (see Fig.
5 in Appendix E.1).
Comparison with the BiG-MRTA solutions also indicate remaining scope of improvement for learning methods, in terms of distance travelled.
The comparison with Feas-RND shows how the learning method is markedly better than random feasible myopic decisions, thereby indicating that meaningful MRTA policies have been learnt here, as opposed to producing random feasible solution by virtue of the masked policy network design.
In the Feas-RND method, a robot takes a task-selection decision randomly from a feasible set of choices that abide by all the constraints in the problems, e.g., related to inter-robot conﬂicts, task deadline and robot range.
As shown in Table 7, CAM achieves more than 92% task completion rate for all the scenarios, which is found to be a X%-Y% gain in task completion rate over that achieved by Feas-RND across these scenarios.
The performance of CAM in terms of the task completion rate is comparable to that of BiG-MRTA (which is slightly better), with the highest difference being 6.9 % for the 500-tasks-50-robots case.
Note that CAM continues to be slightly better than AM is all these cases as well.
Later in Section 5.2, we also compare the performance of CAM, AM, and BiG-MRTA on a benchmark MRTA problem where the objective is just to maximize the task completion rate (hence the cost function is not affected by distance travelled), where CAM proves to be highly competitive.
Ablation study: We performed two ablation studies (Appendix E.2) on CAM to understand the importance of the novel encoder and the decoder (adopted from attention mechanism).
In the ﬁrst ablation study, the CCN based encoding is replaced with simple feedforward layers (as explained in
Appendix E.2), with the decoder remaining the same.
It should be noted that the node embedding length ( d embed ), is the same for all the cases.
In the second ablation study, the MHA based decoding was replaced with simple feedforward layers and a softmax layer (as explained in Appendix E.2), with the encoder remaining the same.
As shown by Table 8, in both of the cases (i.e., with the encoder and decoder respectively ablated), we observe a signiﬁcant decrease in the performance across all scenarios, with the maximum dip in the completion rate being 19.8% and 13.4% for the ﬁrst and the second study, respectively.
Compiling the results from the ﬁrst ablation study for the encoder and the comparison of CAM with AM, we posit that the CCN based encoding, which is able to aggregate local node neighborhoods while remaining agnostic to node ordering, clearly aids in providing better policies.
Similarly, from the second case of the ablation study, we can conclude the MHA based decoding, which computes a compatibility of the current state information with the node information, aid in learning better policies.
5. C OMPARATIVE A NALYSIS ON B ENCHMARK MRTA P ROBLEMS AND CVRP P ROBLEMS
MRTA - Task Allocation Problem with Time and Capacity (TAPTC) :
The CAM architecture is implemented and tested on a well-known class of (NP hard)
MRTA problems known as Task
Allocation Problem with Time and Capacity constraints or TAPTC, as described in Mitiche et al.
In TAPTC, each task i has a time deadline ( t i ) and workload ( w i ), and each robot has a work capacity ( c j ).
The time to ﬁnish task i by robot j is deﬁned as w i /c
j .
We compare the results of CAM on TAPTC with that of AM (Kool et al., 2019) and with those of the three non-learning baseline methods, namely 1) Iterated Local Search (ILS) (Vansteenwegen et al., 2009), which uses a meta heuristic approach, 2) Enhanced Iterated Local Search (EILS) (Vansteenwegen et al., 2009), which has controlled runtime and perturbations (compared to ILS), and 3) Bi-Graph MRTA (BiGMRTA) (
Ghassemi & Chowdhury, 2021)), which uses a bigraph construction and maximum weighted matching approach.
Further details of the TAPTC benchmark, the baselines used here, and the changes made to CAM and AM for this case study are discussed in Appendix F.
For the results shown here, the CAM model is implemented such that k = 9 nearest neighbors are considered when computing the embedding of each node of the task graph.
The testing is performed for different scenarios that are characterized by the number of robots and % of tasks having slack time deadline.
Mitiche et al.
(2019) categorized the TAPTC problems into 2 groups based on the value of the high deadline for the tasks.
Table 3 here presents the results for group 2, with the group 1 results given in
Table 10 in Appendix F.4.
In these tables, the scenario nomenclature is deﬁned as: R75A5 denotes
R = 75 % tasks have a normally-distributed time deadline and a team of A = 5 robots.
Results on the impact of neighborhood size ( k ) on CAM performance is also discussed in Appendix F.4.
Table 3 shows that for all scenarios with A = { 5 , 7 } , CAM performs better than both AM and the non-learning baselines in terms of task completion rate; and CAM achieves top performance for 50% of the scenarios with A = { 2 , 3 } .
[Table] [Table]
A comparison of the computation time to generate the entire MRTA solution, shown in Table 4, demonstrates the advantage of learning based methods over non-learning baselines as the problem size increases w.r.t.
the number of robots and tasks.
Capacitated VRP :
To demonstrate the versatility of the proposed CAM architecture, we train and test the CAM and the AM architectures (for comparison) on a benchmark variation of the Vehicle Routing Problem (VRP), known as the Capacitated VRP (CVRP).
The CVRP benchmark consists of N task locations, where a vehicle is required to visit the locations and deliver packages in a manner that minimizes a cost function.
We also use the LinKernighan heuristics (LKH3) solver (Helsgaun, 2017) and the Simulated Annealing (SA) algorithm implementation provided by Google Operations Research (OR) tools as well-regarded non-learning based baselines for comparing the results obtained by CAM and AM.
Further details regarding the CVRP problem, the baseline method, and settings changes for CAM and AM are presented in
Appendix G.1.
Table 5 summarizes the results of all four approaches (i.e., AM, CAM, and Google
OR) on unseen scenarios for varying task size (# of locations) ranging from 50 to 1,000, in terms of the cumulative computing time and the cost function (both as average values across scenarios of a given size).
LKH3 is a well known state-of-the-art method for solving CVRP problems and also has the best performance (considering only cost and not run time).
As shown in Table 5, the performance of all methods except LKH3 are comparable for small sized problems (50, 100, and 200 locations or tasks).
As expected, the main advantage of the proposed CAM approach is apparent for the problems with larger number of tasks (i.e., 500 and 1000 tasks), where the average cost function of the solutions obtained by the CAM is signiﬁcantly better (less than half of AM’s and one-third of Google OR’s).
In these larger-sized scenarios, the computing time performance of CAM is slightly better than that of AM, and together AM and CAM are two orders of magnitude faster than SA (Google OR) and
LKH3 in generating the entire solution (sequence of tasks to be undertaken).
C ONCLUSION
[Table]
In this paper
[Table]
we proposed a new GNN architecture, called
CAM, for a multirobot task allocation problem with a set of complexities, including tasks with time deadline and robots with constrained range.
This new architecture incorporates an encoder based on covariant node-based embedding and a decoder based on attention mechanism.
A simple RL algorithm has been implemented for learning the features of the encoder and decoder.
In addition, to compare the performance of the proposed CAM method, an attention-based mechanism approach (aka AM) has been extended to be able to handle a multi-agent CO setting problem, along with a recent state-of-the-art method BiG-MRTA
[Table]
and a mypoic baseline method Feas-RND.
To evaluate the performance of the proposed CAM architecture and the extended version of AM, they are trained with the same settings.
All the methods were tested on 100 unseen case studies.
Performance was analyzed in terms of the cost value and the completion rate.
Our primary proposition, CAM, outperformed AM and Feas-RND on test scenarios by achieving better cost function value, and was also able to achieve high task completion rate ( > 92% ) for even larger sized problems without the need to retrain, which is comparable to the near-optimal (but O (10 1 − 10 2 ) more expensively computed) solutions by BiG-MRTA, thereby demonstrating the favorable scalability of CAM.
The computational cost analysis showed that the proposed CAM model takes a few milliseconds to compute a decision, thereby providing clear advantage over non-learning based approaches to
MRTA in the context of online (time-sensitive) planning.
Moreover, the advantage of using local neighborhood information for node encoding can be seen in the scalability analysis on the MRTA and CVRP, where CAM demonstrates superior performance when applied to graphs with larger number of tasks/nodes.
The ablation studies in Appendix E.2 showed the importance of the CCN based encoding and the MHA based decoding used in CAM.
A comparisons over a different suite of benchmark MRTA problems showed that CAM was competitive w.r.t.
standard non-learning baselines including BiG-MRTA, in terms of task completion rate, while providing substantially faster solutions compared to the latter.
Lastly, a comparative analysis on different CAM models with varying neighborhood size (while encoding) was performed (appendix F.6) to study the impact of the neighborhood size k in the encoder.
Based on Appendix F.6 (for MRTA-TAPTC), k is a trade off between accuracy and computation time.
Ethics statement: The work in this paper does not have any direct negative societal consequences.
Reproducibility statement: The CAM architecture which includes the encoder, decoder and the context part, can be coded in any programming language by following the equations in sectionand ﬁgure 2.
The dataset for training the MRTA-Multi-UAV Flood response problem can be generated using the information in Appendix D.3.
The codes for the AM method can be obtained from https://github.com/wouterkool/attention-learn-to-route .
The AM method can be modiﬁed for solving the MRTA-Multi-UAV Flood Response problem using the information from Appendix D.5.
Both CAM and AM method can be trained using the settings in table 6.
The codes for the BiG-MRTA method and Feas-RND can be obtained from https://github.com/ adamslab-ub/BiG-MRTA .
The training data for MRTA-TAPTC can be generated using the information in Appendix F.5.
Both CAM and AM can be modiﬁed using the information in Appendix F.2.
Details on running BiG-MRTA for MRTA- TAPTC can be obtained from Ghassemi et al.
(2019) and the corresponding code from https://github.com/adamslab-ub/BiG-MRTA .
The EILS can be coded using the inforation in Mitiche et al.
The test dataset can be obtained from https://tinyurl.com/taptc15in .
The CAM architecture can be modiﬁed for CVRP using the information in Appendix G.1.1 and G.1.2.
The codes for the AM method and implementation for LKH3 can be obtained from https://github.com/wouterkool/ attention-learn-to-route .
The google OR tools implementation can be done using the example in https://developers.google.com/optimization/routing/vrp with the dataset generated using the information in Appendix G.4.
A L EARNING OVER G RAPHS
Neural network based methods for learning CO can be broadly classiﬁed into: (i)
Reinforcement
Learning (RL) methods Kool et al.
; Barrett et al.
; Khalil et al.
; Strens &
Windelinckx (2005); and (ii) Supervised learning (often combined with RL) methods Kaempfer &
Wolf (2018); Mittal et al.
; Li et al.
; Nowak et al.
The supervised learning approaches typically address problem scenarios where samples are abundant (e.g., inﬂuence maximization in social networks Mittal et al.
or inexpensive to evaluate (e.g., TSP Kaempfer &
Wolf (2018)), and are thus unlikely to be readily applicable to solve complex problems over real-world graphs.
RL based techniques to learn on graphs include attention models with REINFORCE Kool et al.
and deep Q-learning Khalil et al.
; Barrett et al.
(2019), among others, with some extending solutions to multi-agent settings Jiang et al.
In this work, we are interested in the ﬁrst class of the methods (i.e., RL methods over graph space).
Dai et al. Dai et al.
(2017) showed that a combination of graph embedding and RL methods can be used to approximate optimal solutions for combinatorial optimization problems, as long as the training and test samples are drawn from the same distribution.
Mittal et al.
(2019) presented a new framework to solve a combinatorial optimization problem.
In this framework, Graph Convolutional Network (GCN) performs the graph embedding and Q-Learning learns the policy.
The results demonstrated that the proposed framework can learn to solve unseen test problems that have been drawn from the same distribution as that of the training data set.
More importantly, it has been shown that using a learned network policy instead of tree search, both methods are using the same embedding
GCN, showed a speedup of 5.5 for a problem size of 20,000 for inﬂuence maximization.
Similarly, the effectiveness of learning a network policy using Q-Learning to solve the Max Cut problem (a combinatorial problem) has been demonstrated by Barrett et al.
Recently, there has been a growing interest in using sequence-to-sequence models (e.g., pointer networks and attention mechanism) to encode and learn the combinatorial optimization problems in graph space Kaempfer & Wolf (2018); Kool et al.
Kool et al. Kool et al.
(2019) implemented a framework using an encoder/decoder based on attention mechanism and REINFORCE algorithm for solving a wide variety of combinatorial optimization problems as graphs, with the main contribution being ﬂexibility of the approach on multiple problems with the same hyperparameters.
Wang et al.
Wang & Gombolay (2020) showed how learning can lead to generate faster solutions than standard exact methods for multi-robot scheduling problems.
However, the size of the problem that has been studied in that work and other similar studies Paleja et al.
; Sykora et al.
(2020) is limited torobots and 100 tasks, and only temporal constraints were considered.
In this paper, we study larger problems (up to 1000 tasks/200 robots) as well as include complexities such as time deadlines for tasks, robot ferry range constraints, capacity constraints, multiple routes, etc.
Graph learning has been implemented for a multi-robot coverage problem in Tolstaya et al.
(2020), which demonstrates good scalability.
However, this work is addressing a multi-robot exploration problem (and not MRTA), and it is not clear how this proposed method can be applied to an MRTA problem with complexities such as range constrain, capacity constraints, time extended tasks, multiple routes, etc.
Apart from
CO problems, graph learning can also be used to perform path planning as demonstrated in Li et al.
In this paper, a new neural architecture is proposed that combines the attention mechanism with an enhanced encoding network (embedding layers), where the latter is particularly designed to capture local structural features of the graph in an equivariant manner.
The embedding layer is a variation of Covariant Compositional Networks (CCN), introduced by Hy et al.
CCN was originally implemented for predicting molecular properties by learning local structural information of the molecules.
This node-based embedding has been chosen since it: i) operates on an undirected graph; ii) uses receptive ﬁeld and aggregation functions based on tensor product and contraction operations (which are equivariant), which leads to a permutation- and rotation-invariant embedding; and iii) provides an extendable representation ( n -th order tensor representation can be useful to extend the work to multi-level networks, e.g., involving multiple node properties).
We found an exact implementation of the CCN to be computationally burdensome for learning policies in large
MRTA problems, and hence a variation of the CCN is proposed in this work.
An attention mechanism has been successfully implemented for problems with sequential processes, e.g., Natural Language
Processing (NLP) Vaswani et al.
Here we are interested in the attention mechanisms since they involve simple matrix multiplications, which make them not only computationally inexpensive
(by utilizing modern GPUs), but also programmatically easy to be implemented.
In this work, we implement an attention-based decoder for CAM as proposed in Kool et al.
; Vaswani et al.
B E MBEDDING LOCAL STRUCTURAL INFORMATION
Local structural or neighborhood information of a graph node/ task refers to the association of the node to its neighboring nodes.
These information includes how the properties of the neighborhood nodes differ from that of the node under consideration.
It is this information that is being encoded in the node embedding along with the information of the node properties d i .
The encoder should be able to use this local structural information in order to scale to larger-sized problems as pointed out by (Cappart et al., 2021).
Here, we explain how local structural information is being encoded with the help of two graphs, one which is small and the other is a larger graph but having some nodes with similar neighborhood as shown in ﬁgure 3.
Node A in graph G 1 , and nodes B , and C in graph
G 2 have almost similar neighborhood.
Here the local neighborhood information encoding of nodes
A , B , and C (which are F NbA , F NbB , and F NbC respectively) will be almost same, irrespective of the size of the graph.
Therefore by equation 6, (( F Nbd A − F d A )
≈ ( F Nbd B − F d B ) ≈
( F Nbd C − F d C ))
=
⇒ ( F NbA ≈ F NbB ≈ F NbC ) .
Even though the actual locations of nodes A , B , and C are different, their association with it’s neighboring are almost the same, which is being captured by equation 6.
C L EARNING F RAMEWORK
Both the CCN-inspired encoder and the attention-based decoder consist of learnable weight matrices as explained in Sections 3.1 and 3.2.
In order to learn these weight matrices, both supervised and unsupervised learning methods can be used.
However, supervised learning methods are not tractable since the computational complexity of the exact I(N)LP solution process required to generate labels.
The complexity of the ILP formulation of the MRTA problem scales with O ( n 3 m 2 h 2 ) , where n , m , and h represent the number of tasks, the number of robots, and the maximum number of tours per robot, respectively
Therefore, we use a reinforcement learning algorithm to conduct the learning.
Learning Method: In this work we implement a simple policy gradient method (REINFORCE) as the learning algorithm with greedy rollout baseline, which also enables us to compare the effectiveness of our method with that of (Kool et al., 2019).
For each epoch, two sets of data are used which are the training set and the validation set.
The training data set is used to train the training model ( θ CAM ) while the validation data set is used to update the baseline model ( θ BL CAM ).
The size of the training data and the validation data used for this paper is mentioned in section D.3.
Each sample data from the training and validation data-set consist of a graph as deﬁned in Section 2.1.
The pseudo code of the training algorithm for our architecture is shown in Alg. 1 in Appendix E.
It should be noted that the policy gradient method requires the evaluation of a cost function, which is deﬁned to be same as in
Eq.
equation 1.
Policy: We deﬁne the policy such that if the robot r does not satisfy the constraints in
Eqs.
equation 3, it returns to depot (i.e., a = 0 ).
Otherwise the robot r runs the learnt CAM network and chooses the output (task) based on a greedy approach (selects a task with the highest probability value), as shown in Fig.
[Table] [Table]
9: a
←
10: ∇L
[Table] [Table]
[Table]
DAM
13: a
14: a
15: if
N cost
Test i ) 22: a ← a ∪ a i 23: f cost ← f cost ∪ f cost,i 24: end for 25: return a , f cost
C. S IMULATION AND F RAMEWORK S ETTINGS
The ”Python” 3.7 and the 64-bit distribution of ”Anaconda 2020.02” are used to implement the
MRTA approaches.
The environment, training algorithm, and the evaluation of the trained model, are all implemented in Pytorch-1.5 for CAM and AM.
The training, based on Pytorch, is deployed on two GPUs (NVIDIA Tesla V100) with 16GB RAM.
D M ORE D ETAILS ON MRTA D. C ONVENTIONAL METHODS FOR MRTA
The MRTA problem can be formulated as an Integer Linear Programming (ILP), mixed ILP or
Integer Non-Linear Programming (INLP) problem depending on the application.
When tasks are deﬁned in terms of location, the MRTA problem becomes analogical to the Multi-Traveling Salesmen
Problem (mTSP) (Khamis et al., 2015) and its generalized version, the Vehicle Route Planning (VRP) problem (Dantzig & Ramser, 1959).
Existing solutions to mTSP and VRP problems in the literature (Bektas, 2006; Braekers et al., 2016) have addressed analogical problem characteristics of interest to MRTA, albeit in a disparate manner; these characteristics include tasks with time deadlines, and multiple tours per vehicle, with applications in the operations research and logistics communities (Azi et al., 2010; Wang et al., 2018).
ILP-based mTSP-type formulations and solution methods have also been extended to task allocation problems in the multi-robotic domain (Jose &
Pratihar, 2016).
Although the ILP-based approaches can in theory provide optimal solutions, due to the NP-hard time complexity of the SR-ST problems (Mazyavkina et al., 2021; Archetti et al., 2011), they are characterized by exploding computational effort as the number of robots and tasks increases (Toth & Vigo, 2014; Cattaruzza et al., 2016).
For example, for the studied SR-ST problem, the cost of solving the exact integer programming formulation of the problem scales with O ( n 3 m 2 h 2 ) , where n , m , and h represent the number of tasks, the number of robots, and the maximum number of tours per robot, respectively (Ghassemi et al., 2019).
As a result, most practical online MRTA methods, e.g., auction-based methods (Dias et al., 2006) and bi-graph matching methods (Ghassemi & Chowdhury, 2018; Ismail & Sun, 2017), use some sort of heuristics, and often report the optimality gap at least for smaller test cases compared to the exact ILP or INLP solutions (where tractable).
D. M ORE DETAILS ON ENCODING THE CONTEXT
As discussed in section 3.2, the context portion of while a robot r makes a decision, consists of 1)
Current time t ; 2) Available range of the robot taking decision ρ r ; 3) Current location of robot taking decision ( x r , y r ); 4) Current destination of other robots ( x p , y p , ∀ p ∈ P r ); 5) Available range for other robots ( ρ p , ∀ p ∈ P r ), where P r represents the peers of robot r .
Here, features 2 and 3 represent the current state of the robot taking the decision, while features 4 and 4 represents the states of the peer robots.
The context feature vector can be computed as shown in Eq.10.
[Equation]
where,
[Equation]
(11) and
[Equation]
The dimensions of Q r , and Q P r is d embed , and the length of the ﬁnal feature vector Q is also considered d embed .
The summation aggregation operation in Eq.12, makes the context vector agnostic to the number of robots.
D. D ESIGN OF E XPERIMENTS & L EARNING P ROCEDURES
To evaluate the proposed CAM method, we deﬁne an MRTA case study with varying number of
UAVs and 200 task (ﬂood victims) locations.
A 2D environment with 1 sq.
km area is used for this purpose, with the time deadline of tasks varied from 0.1 to 1 hour.
The UAVs are assumed to have a range of 4 km, and a nominal speed of 10 km/h.
We assume instantaneous battery swap is provided at depot location, which is used when UAVs return to depot since they were running low on battery.
It is important to note that the ﬂood victim application is used here merely for motivation, and the
CAM architecture is in no way restricted to this application, but can rather solve problems in the broad (important) class of capacity/range-constrained and timed task-constrained SR-ST problems.
Moreover, even the policies learnt here for CAM demonstration on the described case settings can generalize to related SR-ST problems with up to 1,000 tasks, which represents a fairly large MRTA problem in reference to the existing literature in the MR domain.
To perform learning and testing of the learned model, we proceed as follows: Learning Phase: We use a policy gradient reinforcement learning algorithm (REINFORCE with rollout baselines in this case) for learning the optimal policy.
The learnable parameters in this architecture includes all the weights in the encoder and the decoder.
The training is carried out for a total of 100 epochs.
Each epoch consists of 10,000 random training samples, which are evaluated and trained in batches ofsamples.
Testing Phase:
In order to provide a statistically insightful evaluation and comparison, the models are tested for different cases of varying number of tasks and varying number of robots with each case having 100 random test scenarios from training data distribution.
Here, for each task in a sample scenario, the location of the tasks, time deadline, location of the depot are all generated from a random uniform distribution.
More details on the learning framework is given in Appendix C.
The training and testing settings and the modiﬁcations to AM for MRTA are given in Appendix D.
D. B ASELINES
BiG-MRTA:
The BiG-MRTA algorithm Ghassemi & Chowdhury (2021); Ghassemi et al.
(2019) is an online method based on the construction and maximum weighted matching of a bipartite graph.
BiG-MRTA (Ghassemi & Chowdhury, 2021) uses a novel combination of a bipartite graph construction, an incentive model to assign edge weights in the bigraph, and maximum weighted matching (based on the Karp algorithm (Karp, 1980)) over the bigraph to allocate tasks to robots.
This method has been developed as an online solver for SR-ST type MRTA problems, where tasks have deadlines, new tasks could appear during the mission, and robots are subject to range and payload constraints.
AM : The number of attention heads for the encoder is 8 , with 3 layers of encoding.
The node embedding length is 128.
Feas-RND:
In the Feas-RND approach, each robot randomly chooses available tasks that are feasible to be undertaken by the UAV, satisfying all the constraints in section 2.2.
The algorithm used for
Feas-RND can be found in (Ghassemi & Chowdhury, 2021).
D. M ORE D ETAILS ON T RAINING FOR MRTA
Learning Curve :
In order to compare the convergence of the proposed CAM method with that of the
AM approach, we run both methods with similar settings and plot their learning curve (convergence history), as shown in Fig.
4 in appendix D. As seen from this ﬁgure, the AM method took 3 epochs to reach its best cost value and stagnated.
On the other hand, the CAM method took 20 epochs to reach the best cost value of AM and continued to improve up to ∼ 24 epochs, leading to a signiﬁcantly better cost function value ( f ∗ cost,CAM = − 0 .
compared to AM ( f ∗ cost,AM = − 0 . 009 ).
The stagnation of AM could be attributed to direct implementation of the transformer network (Vaswani et al., 2017), which was designed for machine translation and thus consists of multiple layers of Multi-head attention.
In contrast, our CAM model uses simple linear transformations of the node properties and its relative differences in local neighborhoods to capture structural information.
[Figure]
Modiﬁcations to AM : The attention-based mechanism (AM) reported by (Kool et al., 2019) has been shown to solve a few different classes of single-agent/robot combinatorial optimization problems.
To be able to implement the AM method for our problem (for comparison with our CAM method), the AM method is adapted to a multi-robot setting.
For this purpose, we make the following three changes to the AM method: (i)
The node properties that are deﬁned in Section 2.1 are used in AM; (ii) The context for the attention mechanism is modiﬁed to be the same as that used for CAM; and (iii)
The cost function used for training is changed to that in Eq.
equation 1.
We want to compare the structural representational quality of CAM and AM for generalizability and scalabilty.
Hence, for a fair comparison, both CAM and AM models were trained with the same settings (Table 6 and14).
D.5. C OMPUTING TIME (T RAINING AND E XECUTION )
Based on the epoch information in Section D.3, the average time to complete a training epoch was found to be 19.50 minutes (i.e., ∼ 11.7 seconds per sample) for CAM and AM.
The average computing time (from Tables 1 and 2) taken by the learnt policies for producing the entire MRTA solution (sequence of tasks assigned to each robot) was found to grow from 0.14 s to 20 s, as the number of tasks grew from 50 to 1000, and thus remaining unprecedentedly attractive for real-time decision-making even for large problems.
E MRTA – M ULTI -UAV F LOOD R ESPONSE : F URTHER R ESULTS E. T ASK C OMPLETION R ATE AND C OMPUTATION T IME
Table 7 shows the completion rate which corresponds to the test cases for generalizability (Table 1) and the test for scalability (Table.2), respectively.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
Figure 5, shows the comparison of the average computation time between CAM and BiG-MRTA.
It can be seen that, the computation time of BiG-MRTA increases exponentially as compared to that of CAM.
Even though the completion rate for BiG-MRTA is slightly greater than that of CAM as can be seen from table 7, this advantage comes with a very high computational cost.
This shows the scalability of CAM to larger-sized problems.
E. A BLATION STUDIES
[Table]
For Encoder: In order to study the true impact of the graph node encoding, we compared the performance of CAM with it’s covariant based encoding removed to get a new model CAM E FF .
In this case the, node encoding is performed using a simple feedforward network, following Eq.14.
The decoder for CAM E FF is the same as that of CAM .
[Equation]
where, d i =
[ x i , y i , τ i ] , ∀ i ∈ V .
W d , W F , b d , and b F are learnable weights and biases, where
W d ε R d embed × 3 , b d ε R d embed × 1 , and W F ε R d embed × d embed , b F ε R d embed ×
For Decoder: In order to study the impact of the MHA based decoder, we compare the performance of CAM, with it’s decoder being replaced by a simple feedforward network to get a new model
CAM D FF , which takes in the node embeddings and the context information, and computes the output probabilities using the following equations.
The encoder for CAM D FF is the same as that of CAM .
[Equation]
V
[Equation]
Here W dec and b dec are learnable weights and biases, where W dec ∈ R N × 2 d embed and b dec ∈ R N × 1 .
F i ∀ i ∈ V are the node embeddings from the encoder, and Q ∈ R d embed is the context vector.
The value of d embed for both the models is the same as that of CAM in section 5.
Both the models were trained on the MRTA-Multi-UAV ﬂood response problem in section 4, using the parameters in table 6, and tested on the same test scenarios as that used for CAM , AM , and BiG - MRTA , for varying number of locations and varying number of robots (shown in table 8).
The third and the fourth column in table 8 shows the average cost function of 100 test scenarios, and it’s corresponding % completion rate, for CAM E FF and CAM D FF ablation study, respectively.
As can be seen from table 8, the performance of both CAM E FF and CAM D FF is signiﬁcantly poor compared to CAM , in terms of both cost function and task completion rate.
The performance drop is more for scenarios with lower number of robots, with the maximum drop in task completion rate being for CAM E FF being 19 .
8% (for 50-tasks-5-robots), and for CAM D FF being 13 .
4% (for 200-tasks-20-robots).
It can also be observed that, the performance drop is greater for CAM E FF than CAM D FF , this indicates that the CCN-based encoder has a slightly more inﬂuence on the performance, than the decoder.
F MRTA - T ASK A LLOCATION P ROBLEM W / T IME & C APACITY (TAPTC) F. P ROBLEM D ESCRIPTION AND F ORMULATION
In order to assess the effectiveness of the CAM architecture for MRTA problems with time-extended assignments, we test and validate our CAM on Task Allocation Problem with Time and Capacity (TAPTC) benchmark problem Mitiche et al.
(2019), which falls into the ST-SR-TA-TW (Single
Task robot, Single Robot task, Time-extended Assignment, TimeWindows).
The TAPTC involves task locations within a 100 × 100 grid map, with each task i having a time deadline t i , and also a workload w i .
Each robot j has a work capacity of c j .
All the robots move with a maximum speed of 1 unit per second.
A task i is considered to be completed only if a robot j visits node i and spends a time of w i /c j .
Each scenario is generated based on a uniform distribution reported in Mitiche et al.
(2019) and the training samples for our models are generated from the same distribution.
For evaluating the performance of the learned model and conducting a comparative analysis, we use the TAPTC dataset (accessible from http://tinyurl.com/taptc15in .
This dataset consists of test cases with 100 tasks and varying number of robots ( A = 2 , 3 , 5 , 7 robots), and the speed of every robot is considered to be the same (1 unit per second).
The test cases can be divided equally into two groups, based on tight deadlines (Group 1) and slack deadlines (Group 2).
Each group can be further divided into 4 sub-categories based on the fraction of tasks ( R = 25% , 50% , 75% , and 100% ) that have normally distributed task deadlines.
For example, 25% indicates that there aretasks with deadlines normally distributed between the limits t low and t high , while the remainingtasks have a deadline of t high .
For group 1 (tight deadline), the value of t high is considered as half as that for group 2 (tight deadline).
In both the groups, for all values of A and R , there will besamples.
Therefore the total number of test cases is given by number of groups × | A | × | R | × 3) = 96, where each group has 48 test cases.
Further description of the test cases can be found in
Mitiche et al.
The exact solution of the MRTA-TAPTC problem can be obtained by formulating it as the following integer non-linear programming (INLP) problem:
[Equation]
subject to
[Equation]
Here, N is the number of tasks/nodes, τ fi is the time at which task i was completed, τ i is the time deadline of task i , and S =
[ s 1 , s 2 , ...
s N ] is the sequence of all the nodes that were visited.
The minimum cost function that can be achieved using Eq. 1 is 0 , which corresponds to the case where all the tasks are successfully completed.
A detailed formulation of the exact ILP constraints that describe the MRTA-TAPTC problem can be found in Mitiche et al.
Note that, in our paper we use a slightly different reward function as compared to the objective function in Mitiche et al.
, but the intention of both the functions are essentially the same, which is to maximize the number of successfully completed tasks.
Here, we craft the objective function (Eq.
equation 17) such that only missed tasks ( τ fi > τ i ) contribute to the cost function.
It is important to note that the objective function can be tailored according to the priority of the problem.
Since the main priority in Mitiche et al.
(2019) is to maximize the number of successfully completed tasks, the objective function (Eq. 17) also prioritizes task completion for a fair comparison with the baseline methods.
The constraints in Eq.
18 and 19 are such that each tasks must be visited exactly once by any robot.
F. M ODIFICATIONS TO CAM AND AM
Change in encoder:
The encoder for MRTA-TAPTC problems consider additional node properties, namely the location of the tasks ( x i , y i ), the time deadline ( τ i ), and the workload w i , i.e., d i =
[ x i , y i , τ i , w i ] .
Change in context:
The context for the MHA layer in the decoder consists of the following ﬁve features: 1) elapsed mission time; 2) Work capacity of the robot taking decision; 3) Current location of robot taking decision; 4) Current destination of peers; and 5) Work capacity of peer.
Cost function: We use Eq.
17 as cost function.
The decoder need no change in this case.
F. B ASELINE METHODS
We consider three non-learning methods, which are i)
Iterated Local Search (ILS), ii) Enhanced
Iterated Local Search (EILS), iii) Bi-Graph MRTA (BiG-MRTA), which are brieﬂy described below.
The learning based baseline method we implemented here is AM with minor modiﬁcations as also discussed below.
i)
Iterated Local Search (ILS) :
This is an online meta heuristic iterated search algorithm Vansteenwegen et al.
(2009), where the output of one iteration is partially used as the input to the next iteration.
During each iteration, the best solution is improved by a perturbation step,
followed by
a local search.
ii) Enhanced Iterated Local Search (EILS) : EILS is also an
online
meta
heuristic iterated
search
method Mitiche et al.
), with
an improved perturbation
step as compared to
Vansteenwegen
et
al
iii)
Bi-Graph MRTA (BiG-MRTA) :
The
BiG-MRTA
algorithm Ghassemi & Chowdhury (2021);
Ghassemi et al.
(2019) is an online method based on the construction and maximum weighted matching of a bipartite graph.
BiG-MRTA (Ghassemi & Chowdhury, 2021) uses a novel combination of a bipartite graph construction, an incentive model to assign edge weights in the bigraph, and maximum weighted matching (based on the Karp alogirthm (Karp, 1980)) over the bigraph to allocate tasks to robots.
This method has been developed as an online solver for SR-ST type MRTA problems, where tasks have deadlines, new tasks could appear during the mission, and robots are subject to range and payload constraints.
iii) AM : The AM implementation is almost the same for the MRTA-TAPTC problem as that implemented for the multi-UAV ﬂood response problem in section D.3.
In addition, the changes implemented in CAM, as explained in Section F.2, are also applied here.
The number of attention heads for the encoder is 8 , with 3 layers of encoding.
The node embedding length is 128.
Both CAM and AM are trained using REINFORCE as described in algorithm 1 using the settings given in Table 9.
[Table]
F. R ESULT AND D ISCUSSIONS
Tables 3 and 10 summarize the performance of CAM alongside the baseline methods, in terms of the average completion rate, i.e, the ratio of the number of successfully completed tasks to the total number of tasks averaged over 3 samples for all the scenarios (denoted by A and R ).
The CAM model here uses k = 9 , where k represents the number of nearest neighbors considered for a node for computing its node embedding.
For group 1, the CAM model was able to generate the best results for 8 out of the 16 different scenarios (Table 10).
From Table 10, it can be inferred that CAM has a superior performance compared to the baselines for cases with larger number of robots (best performance for all cases with A = 7 , and for 3 out of 4 scenarios with A = 5 ), including a maximum margin of 19% for R75A7 compared to the next best solution (EILS).
While for cases with smaller number of robots ( A = { 2 , 3 } ), CAM achieved the best performance for one scenario ( R25A3 ), with the worst performance having a margin of only 7% (scenario R75A3 ) compared to best performer (EILS) in that case.
For group 2, CAM achieves top performance for all scenarios with A = { 5 , 7 } , while achieving top performance for 50% of the scenarios with A = 2 , 3 , as discussed in Section 5.2 of the main text.
Tables 11 and 4 gives the average computation time (in milliseconds) to generate the entire solution for all the methods.
As it can be seen from these tables, for the non-learning methods EILS and
BiG-MRTA, the general trend is an increase in the computation time with increasing number of robots, while for both the learning-based methods (CAM and AM) the computation time increases marginally with increasing number of robots.
[Figure]
F. M ORE DETAILS ON T RAINING DATASET FOR MRTA-TAPTC
Each training sample has 100 tasks, and are located randomly within a 100 × 100 grid map.
Each task i has a time deadline 50 ≤ τ i ≤ 600 , and a workload 10 ≤ w i ≤ 30 .
Each sample has n r number of robots where 2 ≤ n r ≤ 7 .
The initial positions of the robots in a sample is also chosen randomly within the grids.
Each robot j has a work capacity of c j where 1 ≤ c j ≤ 3 .
All the robots move at a speed of 1 unit.
A task i is considered to be completed only if a robot j visits node i and spends a time of w i /c j .
All the training samples are generated such that all the associated variables (mentioned above) follow a uniform distribution within their respective bounds
[Table]
[Table] [Table]
F. I MPACT OF NEIGHBORHOOD SIZE FOR CAM ENCODER
Tables 12 and 13 compare the performance of CAM models with varying neighborhood size ( k ) in the encoder, based on the average completion rate (average of the 3 samples) for all the scenarios (all values of A and R for the two groups).
The impact of the neighborhood size k (during encoding) is more evident on the performance of Group 1 test cases which has tasks with more number of tight deadlines.
As shown in Table 12, the completion rate of CAM (with k = { 6 , 9 , 12 } ) are almost comparable, while the performance of CAM with k = 3 is signiﬁcantly lower than the other models.
Lower the neighborhood size, smaller will be the local structural information learned, which could result in performance loss.
However, increasing neighborhood size beyond a point may not necessarily improve performance.
The average epoch time for training for the models with k = { 3 , 6 , 9 , 12 } are 11 , 13 , 14 .
[Table]
and 15 minutes respectively.
Figure 6 shows the learning curve for training all the CAM models and AM model.
[Table]
[Table]
G F URTHER DETAILS ON C APACITATED V EHICLE R OUTING P ROBLEM G. F ORMULATION OF C APACITATED V EHICLE R OUTING P ROBLEM (CVRP)
The vehicle routing problem here we considered is a capacitated vehicle routing problem (CVRP), where a vehicle is required to deliver packages to a number of locations N .
Each task is designated an index from 1 to N .
We also consider a depot with id as 0 .
Each location has a demand c
i on the number of packages where i ∈ [1 , N ] , and the vehicles has a constraint for the maximum number of packages C it can carry, such that c i < D .
We assume that each package is of the same size.
The vehicle is required to create multiple routes visiting different locations to deliver the packages.
The vehicle starts from a depot, has a maximum capacity for the number of packages, and can have multiple routes to deliver all the packages satisfying the demands in every location.
Here we assume that the vehicle can return to the depot for reﬁlling to maximum capacity before starting a new route.
In this experiment we do not consider split delivery where the demand of a location is fulﬁlled partially during a route, and then completed in another route.
The ILP formulation for CVRP can be represented as:
[Equation]
where R is the number of routes, δ j is total distance travelled in route j , C t is the available capacity at a time t , V is the set of locations visited.
The node encoding and the context encoding are modiﬁed for CVRP (as explained in appendix G.1.1 and G.1.2) for both CAM and AM.
Both CAM and AM are trained for the scenarios with 100 locations and tested on the unseen scenarios with varying number of locations ranging from 50 to 1,000.
The experimental details of this comparative study are given in Appendix G.2.
G.1. E NCODING FOR CVRP
Except for the node properties, all the other steps for computing the node embeddings is same for
CVRP as compared to MRTA for CAM and AM.
The node properties associated with the CVRP includes the x coordinate, y coordinate, and the demand c i for each location.
Therefore each node can be represented as d i =
[ x i , y i , c i ] , where x i , y i , and c i are the x coordinate, y coordinate, and capacity respectively for node i .
For the encoding, d i for CVRP will be used in equation 22 for CAM.
[Equation]
G.1. C ONTEXT ENCODING
The context serves the same purpose for which is represent the current state.
For CVRP the current location and the remaining capacity as the context, for CAM and AM.
G. T RAINING DETAILS FOR CVRP
The training procedure for CVRP follows algorithm 1 in appendix D with the only change being in the calculation of the cost.
Equation 20 is used in the algorithm 1 to compute the cost.
Tableshows the different parametric setting for training CAM and AM for CVRP.
CAM and AM models were trained for 100 tasks and tested on CVRP with 5 different task sizes (50, 100, 200, 500, 1000).
Figure 7 shows the learning curve for training CAM and AM.
[Table]
G. B ASELINE DETAILS FOR CVRP
Lin-Kernighan heuristics (LKH3) :
We performed a single run with a maximum number of trails as 10000.
AM :
Same as that of MRTA-Multi-UAV ﬂood response and MRTA-TAPTC.
Google OR tools: The ﬁrst solution strategy algorithm used was PATH CHEAPEST ARC and the local search algorithm was Simulated Annealing .
G. M ORE DETAILS ON CVRP DATASET
[Table]
[Figure]
The dataset used for training CVRP consists of scenarios with 100 locations and one depot.
The x and y coordinates of the locations (including the depot) are randomly generated from a uniform distribution within the limits [0, 1].
The demand for all task locations will be a random integer from a uniform distribution between [1,9], with depot assigned a 0 demand.
The vehicle capacity ( C ) for a scenario with 100 locations, is considered as 50.
The dataset used for testing (to analyze both generalizability and scalibility) has the same limits as explained above.
The assumed capacity of the vehicle for test scenarios of different number of locations are shown in Table 15.
[Figure]
H L IMITATIONS
This work is implemented for a ﬁxed number of nodes but can be easily extended for cases where nodes are determined dynamically.
The impact of the learning algorithm parameters such as the learning rate, training frequency or training batch size, etc. is not analyzed.
Parametric analysis of the learning algorithm, as well as the implementation of more recent state-of-the-art RL algorithms (e.g., PPO), can be considered as other directions of future work with CAM.
The current learning framework for CAM has been implemented with a greedy approach for decision-making.
The performance can be improved by adopting an epsilon greedy approach.
In real-world settings, it is possible that two robots want to make decisions at the same time, which might cause them to visit the same location.
This is very rare and there are various mitigations to address this issue.
For example, while robots are moving toward their selected task locations, they can check if their decision is conﬂicting with another robot based on recent information.
If there is conﬂict, the robot with the worst time can cancel its current task and select a new task.
I A BBREVIATIONS :
[Table]