Learning NP-Hard Joint-Assignment planning using GNN: Inference on a Random Graph and Provable Auction-Fitted Q-iteration
Abstract
The present paper explores the possibility of near-optimally solving multi-robot,multi-task NP-hard scheduling problems with time-dependent rewards using alearning-based algorithm.
In particular, we consider a class of robot/machinescheduling problems called the multi-robot reward collection problem (MRRC).
Such MRRC problems well model ride-sharing, pickup-and-delivery, and a varietyof related problems.
In representing the MRRC problem as a sequential decisionmaking problem, we observe that each state can be represented as an extension ofprobabilistic graphical models (PGMs), which we refer to as random PGMs.
Wethen develop a mean-ﬁeld inference method for random PGMs.
We then propose(1) an order-transferable Q-function estimator and (2) an order-transferabilityenabled auction to select a joint assignment in polynomial-time.
These result in areinforcement learning framework with at least 1  1 /e optimality.
Experimentalresults on solving MRRC problems highlight the near-optimality and transferabilityof the proposed methods.
We also consider identical parallel machine schedulingproblems (IPMS) and minimax multiple traveling salesman problems (minimaxmTSP).
Introduction
Motivation Consider a set of identical robots seeking to serve a set of spatially distributed tasks.
18 Each task is given an initial age (which then increases linearly in time).
Greater rewards are given toyounger tasks when service is complete according to a predetermined reward rule.
Such problems 20 prevail in operations research, e g., dispatching drivers to transport customers or scheduling machinesin a factory.
Solving such highly structured NP-hard problems with the constraint of ‘no possibility 22 of two robots assigned to one task at once’ using mathematical optimization schemes is infeasibleor ineffective due to the expensive computational cost, especially when the problem size is large.
24 Applying decentralized approach using multi-agent modeling frameworkLong et al.
(2020); Rashidet al.
; Sunehag et al.
(2017) is a possible way to solve a large scale problems.
However, due 26 to the impossibility of inducing consensus among agents in achieving the global objective without effective communication Fischer et al.
(1985), such decentralized approaches are rarely used in industries (e g., factories).
Thus, this study focuses on centralized methods for solving MRRC.
Research Questions.
Many of such NP-hard scheduling problems have time-dependent rewards.
To the best of our knowledge, these problems have not yet been addressed by non-decentralized learning-based methods.
Even if one can, it must also be able to simultaneously address a fundamental 32 challenge: the number of possible robot-task pairs to be considered increases exponentially.
For instance, scheduling 8 robots and 50 tasks involves 10 13 possible joint assignments at each time-step
The main research question that the current study seeks to resolve how to design a computationally effective (i.e., scalability in terms of learning and decision making) a learning based centralized 36 decision making scheme for solving a large-scale NP-hard scheduling problems?
Proposed method and contributions.
The present paper explores the possibility of near-optimally 38 solving multi-robot, multi-task NP-hard scheduling problems with time-dependent rewards using alearning-based algorithm.
The study formulates multi-robot, multi-task NP-hard scheduling problems 40 in a sequential decision-making framework and derives a joint scheduling policy with a theoreticalperformance bound under reasonable assumptions.
The novelties of the current study are as follows:
• The study ﬁrst observes that a state-joint assignment pair can be represented as a random PGM.After developing a theory of random PGM-based mean-ﬁeld inference, we derive random structure2vec , a random PGM-based extension of structure2vec Dai et al.
(2016).• We estimate the Q -function Q ( s k , a k ) using layers of random structure2vec , where ( s k , a k ) isthe state-joint action pair.
Using an interpretation of a layer of structure2vec as a WeisfeilerLehman kernel (as in Dai et al.
, we design the estimator to possess a property we callorder-transferability.
This property enables transferability in problem size.•
We propose a joint assignment rule called order-transferability-enabled auction policy (OTAP)to address exponential growth in joint assignment space.
We propose auction-ﬁtted Q -iteration(AFQI) by substitution of the argmax operation of ﬁtted Q -iteration with OTAP to train theQ -function in a scalable manner.
We prove that AFQI results in a policy with polynomial-timecomputation that achieves at least 1  1 /e performance compared with the optimal policy.
Results and Impacts.
Using simulation experiments, we show that the proposed policy typically 55 achieves 97% optimality for the multi-robot reward collection (MRRC) problem in a deterministic environment with linearly time-varying rewards.
This performance is well extended to experiments 57 with stochastic traveling times.
To the best of our knowledge, this result is the ﬁrst to learn anear-optimal NP-hard multi-robot/machine scheduling policy with time-dependent rewards.
Related studies.
Reinforcement Learning based Vehicle Routing Problems .
Mazyavkina et al. (2020) have catego- 61 rized the RL approaches solving vehicle routing problems into two: (1) the improvement heuristicsthat learn an operator can iteratively improve the entire routing plans until there is no improvement is 63 made (Wu et al., 2020; da Costa et al., 2020; Chen & Tian, 2019; Lu et al., 2020; Kim et al., 2021), (2)the construction heuristics that learn a policy that sequentially make a single routing action given the 65 partial solution (state) (Bello et al., 2016; Nazari et al., 2018; Kool et al., 2018; Khalil et al., 2017),and (3) the hybrid approaches that mix these two approaches (Joshi et al., 2020; Fu et al., 2021; Kool 67 et al., 2021; Ahn et al., 2020). These RL approaches have mainly considered a single-agent routingproblem.
Although these methods solve CVRP with multiple vehicles, they solve this problem from 69 a single-agent perspective.
In addition, most of these approaches consider the static reward functionsetting.
On the contrary, our study explicitly considers multi-vehicle interaction while considering 71 the time-varying reward, which is a more realistic routing problem setting.
Graph Inference Based Approach .
Dai et al. (2017) showed that a graph neural network (GNN) 73 called structure2vec Dai et al. (2016) can construct a solution for the Traveling Salesman Problem(TSP).
structure2vec is a popular GNN derived from mean-ﬁeld inference with a probabilistic 75 graphical model (PGM).
Dai et al.
(2017) formulates the TSP as a Markov decision process (MDP)where a heuristically constructed PGM represents each state-next assignment pair.
They employ 77 structure2vec derived from the heuristic PGM to infer the Q -function, which they use to select thenext assignment.
While their choice of PGM was heuristic, their approach achieved near-optimality 79 and transferability of their trained single-robot scheduling algorithm to new single-robot schedulingproblems with an unseen number of tasks.
Multi-Robot Reward Collection Problem
In the main text, we model a general multi robot/machine scheduling problem as a discrete-time, 83 discrete-state (DTDS) sequential decision-making problem.
In the DTDS model, time advancesin ﬁxed increments  , i.e., t k = t {+
+} +  ⇥ k where t k is the actual time after k decision epochs 85 have passed.
For simplicity, we use k as a time index representing the k th decision epoch.
In this framework, s k denotes a state, and action a k denotes a joint assignment of robots/machines to 87 unﬁnished tasks at the k th epoch.
The objective of the problem is to learn the optimum schedulingpolicy ⇡ ✓ : s k !
a k that maximizes the reward collected or minimizes the total completion time 89 (a.k.a. makespan minimization).
Below is the formulation of MRRC.
We additionally propose acontinuous-time continuous-state (CTCS) problem to identical parallel machine scheduling problems 91 (IPMS) in Appendix A and minimax multiple traveling salesman problems (minimax-mTSP) in Appendix B. 3.1 State
The state s k at epoch k is represented as ( g k , D k ) where a graph g k
= (( R , T k ) , ( E T T k , E RT k )) andassociated feature set D k = ( D R k , D T k , D T T k , D RT k ) .
The elements of graph g k are deﬁned as (See 96 Figure 1):
• R = { 1 , ..., M } is the index set of all robots.
The index i and j will be used to speciﬁcally denoterobots.•
T k = { 1 , ..., N } is the index set of all remaining unserved tasks at decision epoch k .
The index pand q will be used to speciﬁcally denote tasks.• E T T k = { ✏ T T pq | p 2 T k , q 2 T k } is the set of all directed edges from a task in T k to any task inT k .
We consider each edge as a random variable.
The task-to-task edge ✏ T T pq = 1 indicates theevent that a robot that has just completed task p subsequently completes task q .
We denote theprobability p ( ✏ T T pq = 1) 2 [0 , 1] the presence probability of the edge ✏ T T pq .•
E RT k = { ✏ RT ip | i 2 R , p 2 T k } is the set of all directed edges from a robot in R to a task in T k . [Figure] We say the robot-to-task edge ✏ RT ip = 1 indicates the event that robot i is assigned to the taskp .
This edge is deﬁned deterministically depending on the joint assignment action.
If robot i isassigned to task p , then p ( ✏ RT ip ) = 1 , otherwise 0.
The element of feature set D k associated with the graph g k is deﬁned as:
• D R k = { d R i | i 2 R} is the set of node features for the robot nodes in R at epoch k .
In MRRC, d R iis deﬁned as the location of robot i at epoch k (epoch index k is omitted).• D T k
= { d T p | p 2 T k } is the set of node features for the task nodes in T k at epoch k .
In MRRC, d T pis deﬁned as the age of task p at epoch k (epoch index k is omitted).• D T T k
= { d T T pq | p 2 T k , q 2 T k } is the set of task-task edge features at epoch k .
d T T pq denotes theduration for a robot that has just completed task p to subsequently compete task q .
We call thisduration task completion time.
In MRRC, a task completion time is given as a random variable(in practice, our method only requires a set of samples of random variable).• D RT k
= { d RT ip | i 2 R , p 2 T k } is the set of robot-task edge features at epoch k .
d RT ip denotes thetraveling time for robot i to reach task p .
3.2 Action
An action a k , a joint assignment at epoch k , is deﬁned as a maximal bipartite matching of the 122 complete bipartite graph ( R , T k , E RT k ) composed of the robot nodes R , the remaining task nodes T k , 123 and the fully connected edges between them E RT k .
That is, given the current state s k =
( g k , D k ) , a kis a subset of E RT k satisfying (1) no two robots can be assigned to the same tasks, and (ii) a robot may 125 only remain without assignment when the number of robots exceeds the number of remaining tasks.
If ✏ RT ip 2 a k , it means that robot i is assigned with task p at epoch k .
For example, Figure 1 showsthe case where a k = ( ✏ RT 1 , 1 , ✏ RT 2 , 3 ) .
(Note, we equivalently may say this as ✏ RT 1 , 1 = ✏ RT 2 , 3 = 1 and 0 128 otherwise.)
In MRRC, all robots are allowed to change their assignments at each decision epoch.
(In
IPMS and mTSP, only free machines/salesmen are newly assigned.)
3.3 State transition
As the joint assignment a k is executed given the current state s k
= ( g k , D k ) , the next state 132 s k +
=
( g k +1 , D k +1 ) is determined with the updated graph g k +1 and features D k +1 .
Graph update.
When the decision epoch corresponds to the point when task p is completed, the 134 corresponding task node will be removed in the updated task nodes as
{+T k +
+} =
T k / { p } , and the 135 task-task edges and robot-task edges,
E T T k +1 and E RT k +1 , will be accordingly updated.
Feature update.
At decision epoch k + 1 , D k +1 =
( D R k +1 ,
D T k
+1 , D T T k +1 , D RT k +1 ) is determined.
In
MRRC, task locations
D R k +
=
{ d R i | i 2 R} and task ages D T k
+1 =
{ d T p | p 2 T k +1 } are updated.
The robot-task edge features D RT k +1 will be updated according to D R
k +1 as well.
3.4 Reward and objective
At time 0 , each task is given an initial age which increases linearly in time.
A reward r k = r  d T p  isgiven when a task p 2 T k , whose age is d T p , is served at epoch k .
We consider linear and nonlinear 142 reward functions r for MRRC.
The objective is to learn a stationary policy ⇡ , a function that mapscurrent state s into current action a , to maximize expected total collected rewards Q ⇡ ( s, a ) =: 144 E P, ⇡ [
P
[Equation]  Random graph embedding: RandStructure2Vec
We observe that when task completion time is not deterministic, a scheduling problem can be 147 represented as an extension of probabilistic graphical models (PGMs), which we refer to as random
PGMs.
This section proposes a mean-ﬁeld inference method, random struct2vec , for random PGMs 149 to estimate the state-action value Q ( s k , a k ) in solving MRRC.
4.1 Random PGM for representing a state of MRRC
Given random variables X
= { X p } , suppose that we can factor the joint distribution p ( X ) as 152 p ( X )
= 1 Z
Q i  i ( D i ) where  i ( D i ) denotes a marginal distribution or conditional distribution as- 153 sociated with a set of random variables
D i ; Z is a normalizing constant.
Then { X p } is called a 154 probabilistic graphical model (PGM).
In a PGM, D i is called a clique and  i ( D i ) is called the clique 155 potential for D i , and D i is called the scope of  i .
We often write simply  i , suppressing D i .
Starting from a state s k and an action a k , one can conduct a random experiment of “sequential 157 decision making using policy ⇡ ”.
In this random experiment, we can denote the events ‘How robotsserve all remaining tasks in which sequence’ as scenarios .
For example, suppose that at time-step k we are given robots { r 1 , r 2 } , tasks { t 1 , t 2 , t 3 , t 4 , t 5 , t 6 } and we follow the policy ⇡ onward.
One possible scenario is that robot r 1 serves { t 1 ! t 4 } and robot r 2 serves tasks { t 3 ! t 2 !
t 5 ! t 6 } 161 (see Figure 1). Note that the time when t 5 is served depends on the time when t 2 is served (state transition is Markovian); thus the reward from t 5 depends on the reward from t 2 .
As shown in Figure be a
4.2 Mean-ﬁeld inference with random PGM
Let X =
{ X p } be the set of all random variables in the inference problem.
Let G X be the set of all 169 possible PGMs on X .
Let P : G X 7! [0 , 1] be a probability measure on G X .
Deﬁne a random PGM 170 on X as {G X , P} .
Note that the inference of {G X , P} will be difﬁcult; |G X | is too large for inferring
P even using Monte-Carlo sampling approach.
To avoid this difﬁculty, we use the approximated 172 inference using semi-cliques .
Suppose that we are given the set of all possible cliques on X as C X . {+
+}
As a PGM will be realized according to P , only a few of the possible cliques in C X will be actuallyrealized as an element of the PGM and become real cliques.
We call such potential clique elements 175 of C X as semi-cliques .
Note that if we were given P , we could calculate the presence probability p m 176 of the semi-clique D m as p m =
[Equation] G , where 1 denotes the indicator function.
Mean-ﬁeld inference with random PGM.
We start from a speciﬁc inference problem and state the 178 main theorem in more general way.
Consider a random PGM on
X =
( { H i } , { X j } ) where H k is 179 the latent variable corresponding to the observed variable X k .
Our goal is to infer { H i } given { X j } 180 by ﬁnding p ( { H i }|{ x j } ) .
In mean-ﬁeld inference, we instead ﬁnd a set of surrogate distributions{ q { x j } ( H i ) } for which { H i } are independent.
Here q { x j } means that q is a function of { x j } ).
We next state Theorem 1.
in a very general manner.
The statement is the same as that of mean-ﬁeld 183 inference with PGM Koller & Friedman (2009) except that ours has the presence probability terms{ p m } of semi-cliques {D m } .
The implication is that inference of presence probability of each 185 semi-clique is enough to conduct mean-ﬁeld inference, and the inference of {G X , P} is not needed.
Theorem 1 .
Random PGM based mean ﬁeld inference.
Suppose a random PGM on X =
{ X p }is given, and the presence probability { p m } for all semi-cliques D m 2 C X are known.
Then, the surrogate distribution { q p ( x p ) } in mean-ﬁeld inference is optimal only if q p ( x p ) =1 Z p exp
[Equation]
o where Z p is a normalizer and  m cliquepotential for D m .
For the general background and proof, see Appendix C.
RandStructure2Vec.
In Dai et al.
(2016), structure2vec was derived as a vector space embedding of 193 mean-ﬁeld inference with PGM.
For detailed background on vector space embedding, see Appendix
D. From Theorem 1 , we derive random structure2vec as a vector-space embedding of mean-ﬁeld 195 inference with random PGM.
Suppose that once a PGM is realized the PGM has joint distributionproportional to some factorization [Equation]
[Equation] p | [Equation] q ) (as in Dai et al.
Underthis assumption, we can write { q { x j } ( H i ) } as { q x i ( H i ) } .
In Dai et al.
, they suggest that 198 structure2vec is essentially a ﬁxed point iteration ˜ µ p
[Equation] p [Equation] q 6 = p ˜ µ q ) where ˜ µ p is a 199 latent vector for node p and x p is input for node p .
They show that, when we interpret ˜ µ i as a vectorspace injective embedding expressed as ˜ µ i = R H  ( h i )
q x i ( h i ) dh
i for some  , structure2vec’s ﬁxedpoint iteration is the embedding of ﬁxed point iteration of mean-ﬁeld inference with the PGM.
Lemma 202 1 states that we have a similar result for random PGM by including only the { p qp } information.
Lemma 1 .
Structure2vec for random PGM.
Assume that the presence probabilities { p qp } for allpairwise semi-cliques D qp 2 C X are given.
Then embedding the ﬁxed point equation in Theorem 1generates the ﬁxed point equation ˜ µ p  ⇣ W 1 x p + W P q 6 = p p qp ˜ µ q ⌘ .
We refer to this ﬁxed pointiteration as random structure2vec.
(The proof of Lemma 1 can be found in Appendix E.)
Remarks.
Note that inference of {G X , P} is in general a difﬁcult task.
One implication of Theorem 208 1 is that we transformed a difﬁcult inference task into a simple inference task : inferring the presenceprobability of each semi-clique.
(See Appendix F for the algorithm that conducts this task.)
In 210 addition, Lemma 1 provides a theoretical justiﬁcation to ignore the inter-dependencies among edgepresences when embedding a random graph using GNN.
When graph edges are not explicitly given or 212 known to be random, the simplest heuristic one can use is to separately infer the presence probabilitiesof all edges and adjust the weights of GNN’s message propagation.
According to Lemma 1 , possible 214 inter-dependencies among edges would not affect the quality of such heuristic inference.
[Figure] This section describes how the proposed method solves the MRRC problem with the newly developed 217 random structure2vec method.
The total solution (i.e., the sequential assignments of robots/machinesto tasks) is found by iteratively repeating a sequential decision making.
This section speciﬁcally 219 describes how to choose a joint assignment a k given the current state s k =
( g k , D k ) .
The procedure 220 for determining a joint assignment action is composed of (1) represent the state using a random
Bayesian Network, (2) estimate the Q-value using random graph embedding, and (3) select a joint 222 assignment.
Figure 2 depicts the overall procedure.
This section focuses on MRRC; the appendiceswill provide the formulation, solution procedure, and results for IPMS and mTSP problems as well.
5.1 Representing a state using a random PGM
In this section, we describe how the MRRC problem state s k and one possible joint assignment 226 a k can be represented as a Bayesian network.
Assume H p , the hidden random variable for task 227 p , carries information about the beneﬁt of serving task p .
Given a scenario (see section 3.1), H pdepends on H q only if q is served after p by the same robot, and H p depends on X p , the observedfeatures for task p .
For an MRRC problem, X p can be either a task’s assignment information d RT ip ora task’s age d T p .
Based on these deﬁnitions, a Bayesian Network for { H p } and { X p } is constructedas p ( { H p }|{ X p } ) =
Q p  ( H p | X p ) Q p,q  ( H p | H q ) .
This Bayesian Network corresponds to onescenario; there are many possible scenarios that can be realized randomly.
Since random PGM natu- 233 rally models this characteristic, according to Lemma 1, we are justiﬁed to use random structure2vecwith edge (semi-clique
presence probabilities { p ( ✏ T T pq ) } from in section 2.1.
5.2 Estimating state-action value using Order trainability-enabled Q-function
We illustrate how we can estimate a Q-function for MRRC by applying random structure2vec 237 to the random PGM that represents ( s k , a k ).
Lemma 1 provides the ﬁxed point equation ˜ µ p = ⇣ W 1 x p + W P j 6 = k p qp ˜ µ q ⌘ to compute the embeddings ˜ µ p for node task p in a random PGM.
The embeddings ˜ µ p for p 2 T k are computed in an iterative manner using random structure2vec as:˜ µ ( ⌧
[Equation]
) p = [Equation] ⌘ .
We propose a network architecture composed of two-stepsequential random structure2vec , which is composed of action embedding and value embedding .
Action Embedding.
The ﬁrst random structure2vec layer embeds the joint assignment into the task 243 nodes T k
= { 1 , ..., n } .
The action embedding ˜ µ Ap for task node p is deﬁned as the ﬁxed point for theequation ˜ µ
Ap =  [Equation] ˜ [Equation]
⌘ where x Ap = d RT ip (distance from robot i to task p )when task p is assigned to robot i , x Ap = 0 when task p is not assigned.
The action node embeddings
{ ˜ µ Ap | p 2 T k } , computed via iteration simultaneously by employing random structure2vec , provide 247 sufﬁcient information about the relative locations between robots and their assigned tasks.
Value Embedding.
The second random structure2vec layer embeds the task ages into the task 249 nodes.
The value embedding ˜ µ Vp for task node p is deﬁned as the ﬁxed point for the equation˜ µ Vp =  ⇣
W V 1 x Vp + W V P q 6 = p p qp ˜ µ Vq ⌘ where x Vp =
(˜ µ Ap , d T p ) is the concatenation of the actionembedding ˜ µ
Ap computed by the ﬁrst layer and the task age d T p for node p .
The resulting value nodeembeddings { ˜ µ Vp | p 2 T k } , computed in an iterative manner, provide sufﬁcient information about 253 how much value is likely in the local graph around each task by the speciﬁed joint assignment.
Computing Q ✓ ( s k , a k ) .
To derive Q ✓ ( s k , a k ) , we aggregate the embedding vectors for all nodes by 255 ˜ µ V = P p ˜ µ
Vp to obtain one global vector ˜ µ V to embed the value afﬁnity of the global graph.
Wethen use a neural network to map ˜ µ V into Q ✓ ( s k , a k ) .
The overall pseudo-code for estimating steps 257 is provided in Appendix G.
It is essential for the proposed inference method can estimate the state action value Q ✓ ( s k , a k ) 259 for varying graph size g k .
Let us provide the intuition related to scuh problem-size transferability.
For Action Embedding , transferability is trivial; the inference problem is a scale-free task locally 261 around each node .
For Value Embedding , consider the ratio of robots to tasks.
The overall valueafﬁnity embedding will be underestimated if this ratio in the training environment is smaller than 263 this ratio in the testing environment; overestimated overall otherwise.
The intuition is that thisover/under-estimation does not matter in Q-function based policies as discussed in van Hasselt et al. 265 (2015) as long as the order of Q-function value among actions are the same.
That is, as long asthe best assignments chosen are the same, i.e., arg max a k Q ( s k , a k )
=
arg max a k Q ✓ ( s k , a k ) , the 267 magnitude of imprecision | Q ( s k , a k )  Q ✓ ( s k , a k ) | does not matter.
We call this property of an 268 estimator order-transferability (with respect to the max operation).
5.3 Selecting a joint assignment using OTAP
With the previously introduced way to estimate Q ✓ ( s k , a k ) , we illustrate how to compute the joint 271 assignment (action) a ⇤ k , a maximal bipartite matching in the bipartite graph ( R , T k , E RT k ) , given 272 the state s k =
( g k , D k ) .
Speciﬁcally, we propose the order transferability-enabled auction policy(OTAP) that constructs a joint assignment a k through N = max ( |R| , | k | ) iterations of Bidding and 274 Consensus phases.
Each iteration adds one robot-task assignment to construct a full joint assignment.
Bidding-phase.
In the n th iteration of the bidding phase, given M ( n  1) ✓ , the ordered set of n  1robot-task edges in E RTk determined by the previous n  1 iterations, all the unassigned robots bid 277 their the most preferable task to conduct.
Respecting robot-task assignments determined in previousn  1 iterations and ignoring other unassigned robots, each unassigned robot i select the best taskassignment ✏ RT il that maximizes Q n ✓ ( s k , M ( n  1) ✓ [ { ✏ RT ip } ) among all unassigned tasks p 2 T k , 280 where Q n ✓ is the ✓ -parameterized network with superscript n indicating that the state action value isestimated at n th iteration.
Then, robot i bids { ✏ RT i ` , Q n ✓ ( s k , M ( n  1) ✓ [ { ✏ RT i ` } ) } to the centralized 282 auctioneer.
This bidding occurs simultaneously by all the unassigned robots at the n th iteration.
Since the number of ignored robots varies at each iteration, transferability of Q-function inference is 284 crucial.
Consensus-phase.
In the consensus phase of n th iteration, the centralized auctioneer ﬁnds the bidwith the best bid value, say { ✏ RT i ⇤ p
⇤ , Q n ✓ ( s k , M ( n  1) ✓ [ { ✏ RT i ⇤ p ⇤ } ) } (Here i ⇤ and p ⇤ denote the best robottask pair.)
Denote ✏ RT i ⇤ p ⇤ :
= m ( n ) ✓ .
The auctioneer sets everyone’s M ( n ) ✓ as M ( n ) ✓ =
M ( n  1) ✓ [ m ( n ) ✓and initiate bidding phase for the remaining unassigned robots.
These two phases iterate until reaching M ( N ) ✓ = { m (1) ✓ , . . .
, m ( N ) ✓ } .
This M ( N ) ✓ is chosen as the 290 joint assignment a ⇤ k of N -robots at time step k .
That is, ⇡ ✓ ( s k ) = a ⇤ k .
The computational complexityfor computing ⇡ ✓ is O ( | R | | T | ) and is only polynomial; see Appendix L.
5.4 Training Q-function using AFQI
The ﬁtted Q-iteration (FQI) ﬁnds ✓ that minimizes E ( s k ,a k ,r k ,s k +1 )
⇠ D [ Q ✓ ( s k , a k )  [ r ( s k , a k ) + 294  max a Q ✓ ( s k +1 , a )]] where D denotes the distribution of training data.
We propose a new rein- forcement learning method, which we call Auction-ﬁtted Q-iteration (AFQI) , which replaces max a 296 Q ✓ ( s k , a ) used in the conventional FQI with OTAP.
That is, writing OTAP as ⇡ Q ✓ , AFQI ﬁnds ✓ that 297 empirically minimizes E ( s k ,a k ,r k ,s k +1 )
⇠ D [ Q ✓ ( s k , a k )  [ r ( s k , a k ) +  Q ✓ ( s k +1 , ⇡ Q ✓ ( s k +1 ))]] .
In learning the parameters ✓ for Q ✓ ( s k , a k ) , we use the exploration strategy that perturbs the 299 parameters ✓ randomly to actively explore the joint assignment space with OTAP.
While this methodwas originally developed for policy-gradient based methods Plappert et al.
(2017), exploration in 301 parameter space is useful in our auction-ﬁtted Q-iteration since it generates a reasonable combinationof assignments.
Theoretical analysis
We show the proposed AFQI obtains at least 1  1 /e optimality and enables computation of the joint 305 assignment in polynomial time.
This result is achieved by the order-transferability of the proposed
Q -function estimator and its use in selecting the joint assignment.
6.1 Performance bound of OTAP
Recall that Q n denotes the n -robot problem’s true Q -function.
In the same way as we deﬁned M ( N ) ✓above, denote the joint assignment chosen by OTAP as { Q n } Nn =1 as M ( N ) = { m (1) , . . .
, m ( N ) } .
Lemma 2.
If the Q -function approximator has order transferability, then M ( N ) =
M ( N ) ✓ .
For any decision epoch k , let M denote a set of robot-task pairs (a subset of E RT k ).
For any robot-taskpair m 2 E RT k , deﬁne  ( m | M ) :
= Q |M [ { m }| ( s k , M [ { m } )
Q |M| ( s k , M ) as the the marginal 313 value (under the true Q-functions) of adding robot-task pair m [Equation] E RT k .
Lemma 2 enables us to use 314 the result discussed in Nemhauser et al.
and achieve the result of Theorem 2.
Theorem 2.
Suppose that the Q -function approximation with the parameter value ✓ exhibits ordertransferability.
Denote M ( N ) ✓ as the result of OTAP using { Q n ✓ } Nn =1 and let M ⇤ = argmax a k
Q | a k | ( s k , a k ) .
If  ( m | M )  0 , 8 M ⇢ E RTk , 8 m 2 E RT k , and the marginal value of adding one 318 robot diminishes as the number of robots increases, i.e.,  ( m | M )   ( m | N ) , 8 N ⇢ M ⇢ E RTk ,8 m 2 E RTk , then the result of OTAP is at least better than 1  1 /e of an optimal assignment.
That is,
Q N ✓ ( s k , M ( N ) ✓ )  Q |M ⇤ | ( s k , M ⇤ )(1  1 /e ) .
See Appendix H and K for the proofs.
6.2 Performance bound of AFQI
AFQI seeks to ﬁnd ✓ that minimizes E ( s k ,a k ,r k ,s k +1 )
⇠ D [ Q ✓ ( s k , a k )  [ r ( s k , a k ) + 323  Q ✓ ( s k +1 , ⇡ Q ✓ ( s k +1 ))]] .
Here, we use OTAP, denoted as ⇡ Q ✓ , instead of max a Q ✓ ( s k , a ) which is 324 used in general ﬁtted-Q iteration.
As we have seen in section 5.1, OTAP replaces the max operationwith the auction algorithm with a provable performance bound compared with the max operation.
326 Lemma 3 allows us to use this performance bound to obtain a performance assurance on AFQIcompared with FQI.
We only write an abbreviated version of the statement for brevity.
The formal 328 description of conditions and the proof is provided in Appendix I.
Lemma 3.
Kang & Kumar (2021) Suppose that a 1  1 /r approximation algorithm is substituted for 330 the max operation in FQI.
Then, the corresponding new Fitted Q-iteration’s performance is at least1  1 /r optimal.
Corollary 1.
AFQI achieves at least 1  1 /e performance compared with the optimal policy.
6.3 Experiment settings
In the main text, we focus on discrete-time & discrete-state (DTDS)
MRRC problems with deter- 335 ministic and stochastic task completion times.
For CTCS deterministic problems with real-worlddatasets, see IPMS (Appendix A) and mTSP (Appendix B).
Environment.
Since there is no standard dataset for MRRC problems, we used the complex maze-like 338 environment generator of Neller et al.
(2010) (code provided in Appendix 10).
This complex mazemimics the complex road layout of a city and random trafﬁc, inducing nontrivial task completion [Table] [Table] [Table]
times.
See the leftmost image of Figure 2 and the supplementary video.
We randomly generated a 341 new maze for every training and testing experiment with randomly chosen initial task/robot locations [Table]
To generate the task completion times [Table] Dijkstra’s algorithm and dynamic programming were used for 343 deterministic and stochastic environments, respectively.
In the deterministic environment [Table] robots always succeed in their movement.
In the stochastic 345 environment [Table] a robot makes its intended move with a certain probability.
(Cells with a dot: successwith 55%, every other direction with 15% each.
Cells without a dot: 70% and 10%, respectively.)
A 347 task is considered served when a robot reaches it.
We consider two reward rules: linearly decayingrewards f ( age ) =
max { 200  age, 0 } and nonlinearly decaying rewards f ( age ) =
age with  = 349 0.99, where age is the task age when served.
The initial age of tasks are uniformly distributed in
the interval [
Baselines.
For deterministic environments with linear rewards, where the corresponding MRRC can 352 be formulated as a mixed-integer linear program (MILP), we consider the following two baselinealgorithms:
• Optimal : Gurobi Gurobi Optimization (2019), an off-the-self the optimization solver for MILP,was used to solve the problems with 60-min time limit.• Ekici et al.
: Ekici & Retharekar (2013), the most up-to-date heuristic for solving MRRC in theOperations Research community, was used the problems.
For stochastic environments or exponential rewards, to our knowledge, there is no literature addressing 359 MRRC with.
Thus, we construct an indirect baseline:
• Sequential Greedy Algorithm (SGA) : a general-purpose multi-robot task allocation algorithmcalled SGA Han-Lim Choi et al.
The performance measure we used is⇢ = rewards collected by the proposed method reward collected by the baseline
Thus, the value of ⇢ greater than 100% indicates the the proposed method collects more reward than 364 the corresponding baseline algorithm.
Note that ⇢ against Optimal always lower than 100%.
Note that we cannot provide other reinforcement learning-based heuristics as additional baselines 366 since, to the best of our knowledge (for the class of NP-hard multi-robot/machine scheduling problemswith decaying rewards), this paper is the ﬁrst to propose a reinforcement learning-based heuristic.
6.4 Performance test.
We tested the performance under four environments: deterministic/linear rewards, determinis- 370 tic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.
See Table .
Ourmethod achieves near-optimality for linear/deterministic rewards with 3% fewer rewards than optimal 372 on average.
The standard deviation for ⇢ is provided in parentheses.
For others, we see that the%SGA ratio for linear/deterministic is well maintained in stochastic or nonlinear environments.
Due 374 to dataset generation’s dynamic programming computation complexity, we only consider 8 robots/50tasks at maximum.
We considered larger size problems in IPMS experiments discussed in Appendix 376 A.
6.5 Transferability test.
Table 2 provides comprehensive transferability test results.
The rows indicate training conditions, 379 while the columns indicate testing conditions.
The results in the diagonal cells in red (cells withthe same training size and testing size) serve as baselines (direct testing).
The results in the off- 381 diagonal show the results for the transferability testing and demonstrate how the algorithms trainedwith different problem sizes perform well on test problems (zero-shot transfer).
We can see that 383 lower-direction transfer tests (trained with larger size problems and tested with smaller size problems)show only a small loss in performance.
For upper-direction transfer tests (trained with smaller size 385 problems and tested with larger size problems), the loss was up to 4 percent.
6.6 Scalability analysis.
For training complexity, we measured the training time required to achieve 93% optimality considering 388 a deterministic environment with linear rewards.
Table 4 shows that training time may not necessarilyincrease as problem size gets larger, while the performance is fairly maintained.
MRRC can be formulated as a semi-MDP (SMDP) based multi-robot planning problem (e.g., Omid- 391 shaﬁei et al.
This problem’s complexity with R robots and T tasks and maximum H timehorizon is O
( R ! /T !
( R  T )!)
H ) .
In our proposed method, this complexity is addressed by a combi- 393 nation of two complexities: computational complexity and training complexity.
For computationalcomplexity of joint assignment decision at each timestep is O ( | R || T | 3 ) .
See Appendix L for details.
Concluding Remarks
In this paper, we addressed the challenge of developing a near-optimal learning-based method 397 for solving NP-hard multi-robot/machine scheduling problems.
We developed a theory of meanﬁeld inference for scheduling problems and a corresponding theoretically justiﬁed GNN method to 399 precisely infer the Q-function.
We addressed the scalability issue of Fitted Q-Iteration methods formulti-robot/machine scheduling problems by providing a polynomial-time algorithm with a provable 401 performance guarantee.
Simulation results demonstrate the effectiveness of the our methods.
In termsof future directions, we have not yet tried the same problem with heterogeneous agents, where the 403 agent’s speeds are different.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on 491 how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justiﬁcation to your answer , either by referencing 493 the appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .• Did you include the license to the code and datasets? [No] The code and the data areproprietary.• Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the 499 Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]