Learning NP-Hard Multi-Agent Assignment Planning using GNN: Inference on a Random Graph and Provable Auction-Fitted Q-learning
Abstract
This paper explores the possibility of near-optimally solving multi-agent, multi-task
NP-hard planning problems with time-dependent rewards using a learning-based algorithm.
In particular, we consider a class of robot/machine scheduling problems called the multi-robot reward collection problem (MRRC).
Such MRRC problems well model ride-sharing, pickup-and-delivery, and a variety of related problems.
In representing the MRRC problem as a sequential decision-making problem, we observe that each state can be represented as an extension of probabilistic graphical models (PGMs), which we refer to as random PGMs.
We then develop a mean-field inference method for random PGMs.
We then propose (1) an order-transferable
Q-function estimator and (2) an order-transferability-enabled auction to select a joint assignment in polynomial-time.
These result in a reinforcement learning framework with at least 1 − 1 /e optimality.
Experimental results on solving MRRC problems highlight the near-optimality and transferability of the proposed methods.
We also consider identical parallel machine scheduling problems (IPMS) and minimax multiple traveling salesman problems (minimax-mTSP).
Introduction
Motivation Consider a set of identical robots seeking to serve a set of spatially distributed tasks.
Each task is given an initial age (which then increases linearly in time).
Greater rewards are given to younger tasks when service is complete according to a predetermined reward rule.
Such problems prevail in operations research, e.g., dispatching drivers to transport customers or scheduling machines in a factory.
Solving such highly structured NP-hard problems with the constraint of ‘no possibility of two robots assigned to one task at once’ using mathematical optimization schemes is infeasible or ineffective due to the expensive computational cost, especially when the problem size is large.
Applying decentralized approach using multi-agent modeling frameworkLong et al.
Rashid et al.
; Sunehag et al.
(2017) is a possible way to solve a large scale problems.
However, due to the impossibility of inducing consensus among agents in achieving the global objective without effective communication Fischer et al.
(1985), such decentralized approaches are rarely used in industries (e.g., factories).
Thus, this study focuses on centralized methods for solving MRRC.
Research Questions.
Many of such NP-hard scheduling problems have time-dependent rewards.
To the best of our knowledge, these problems have not yet been addressed by non-decentralized learning-based methods.
Even if one can, it must also be able to simultaneously address a fundamental challenge: the number of possible robot-task pairs to be considered increases exponentially.
For instance, scheduling 8 robots and 50 tasks involves 10 13 possible joint assignments at each time-step.
The main research question that the current study seeks to resolve is "how to design a computationally effective (i.e., scalability in terms of learning and decision making) a learning based centralized decision making scheme for solving a large-scale NP-hard scheduling problems?"
Proposed method and contributions.
The present paper explores the possibility of near-optimally solving multi-robot, multi-task NP-hard scheduling problems with time-dependent rewards using a learning-based algorithm.
The study formulates multi-robot, multi-task NP-hard scheduling problems in a sequential decision-making framework and derives a joint scheduling policy with a theoretical performance bound under reasonable assumptions.
The novelties of the current study are as follows:
• The study first observes that a state-joint assignment pair can be represented as a random PGM.
After developing a theory of random PGM-based mean-field inference, we derive random structure2vec , a random PGM-based extension of structure2vec Dai et al.
• We estimate the Q -function Q ( s k , a k ) using layers of random structure2vec , where ( s k , a k ) is the state-joint action pair.
Using an interpretation of a layer of structure2vec as a WeisfeilerLehman kernel (as in Dai et al.
, we design the estimator to possess a property we call order-transferability.
This property enables transferability in problem size.
• We propose a joint assignment rule called order-transferability-enabled auction policy (OTAP) to address exponential growth in joint assignment space.
We propose auction-fitted Q -iteration (AFQI) by substitution of the argmax operation of fitted Q -iteration with OTAP to train the Q -function in a scalable manner.
We prove that AFQI results in a policy with polynomial-time computation that achieves at least 1 − 1 /e performance compared with the optimal policy.
Results and Impacts.
Using simulation experiments, we show that the proposed policy typically achieves 97% optimality for the multi-robot reward collection (MRRC) problem in a deterministic environment with linearly time-varying rewards.
This performance is well extended to experiments with stochastic traveling times.
To the best of our knowledge, this result is the first to learn a near-optimal NP-hard multi-robot/machine scheduling policy with time-dependent rewards.
Related studies.
Reinforcement Learning based Vehicle Routing Problems .
Mazyavkina et al.
(2020) have categorized the RL approaches solving vehicle routing problems into two: (1) the improvement heuristics that learn an operator can iteratively improve the entire routing plans until there is no improvement is made (Wu et al., 2020; da Costa et al., 2020; Chen & Tian, 2019; Lu et al., 2020; Kim et al., 2021), (2) the construction heuristics that learn a policy that sequentially make a single routing action given the partial solution (state)
(Bello et al., 2016; Nazari et al., 2018; Kool et al., 2018; Khalil et al., 2017), and (3) the hybrid approaches that mix these two approaches (Joshi et al., 2020;
Fu et al., 2021; Kool et al., 2021;
Ahn et al., 2020).
These RL approaches have mainly considered a single-agent routing problem.
Although these methods solve CVRP with multiple vehicles, they solve this problem from a single-agent perspective.
In addition, most of these approaches consider the static reward function setting.
On the contrary, our study explicitly considers multi-vehicle interaction while considering the time-varying reward, which is a more realistic routing problem setting.
Graph Inference Based Approach .
Dai et al.
(2017) showed that a graph neural network (GNN) called structure2vec Dai et al.
(2016) can construct a solution for the Traveling Salesman Problem (TSP).
structure2vec is a popular GNN derived from mean-field inference with a probabilistic graphical model (PGM).
Dai et al.
(2017) formulates the TSP as a Markov decision process (MDP) where a heuristically constructed PGM represents each state-next assignment pair.
They employ structure2vec derived from the heuristic PGM to infer the Q -function, which they use to select the next assignment.
While their choice of PGM was heuristic, their approach achieved near-optimality
Multi-Robot Reward Collection Problem
In the main text, we model a general multi robot/machine scheduling problem as a discrete-time, discrete-state (DTDS) sequential decision-making problem.
In the DTDS model, time advances in fixed increments ∆ , i.e., t k = t 0 + ∆ × k where t k is the actual time after k decision epochs have passed.
For simplicity, we use k as a time index representing the k th decision epoch.
In this framework, s k denotes a state, and action a k denotes a joint assignment of robots/machines to unfinished tasks at the k th epoch.
The objective of the problem is to learn the optimum scheduling policy π θ : s k → a k that maximizes the reward collected or minimizes the total completion time (a.k.a. makespan minimization).
Below is the formulation of MRRC.
We additionally propose a continuous-time continuous-state (CTCS) problem to identical parallel machine scheduling problems (IPMS) in Appendix A and minimax multiple traveling salesman problems (minimax-mTSP) in
Appendix B.
3.1 State
The state s k at epoch k is represented as ( g k , D k ) where a graph g k
= (( R , T k ) , ( E T T k , E RT k ))
and
associated feature set D k =
( D R k , D T k , D T T k , D RT k ) .
The elements of graph g k are defined as (See
Figure 1):
• R = { 1 , ..., M } is the index set of all robots.
The index i and j will be used to specifically denote robots.
• T k
= { 1 , ..., N } is the index set of all remaining unserved tasks at decision epoch k .
The index p and q will be used to specifically denote tasks.
• E T T
k
= { ϵ T T pq | p ∈ T k , q ∈ T k } is the set of all directed edges from a task in T k to any task in T k .
We consider each edge as a random variable.
The task-to-task edge ϵ T T pq = 1 indicates the event that a robot that has just completed task p subsequently completes task q .
We denote the probability p ( ϵ T T pq = 1) ∈
[0 , 1] the presence probability of the edge ϵ T T pq .
• E RT k = { ϵ RT ip |
i ∈ R , p ∈ T k } is the set of all directed edges from a robot in R to a task in T k .
We say the robot-to-task edge ϵ RT ip = 1 indicates the event that robot i is assigned to the task p .
This edge is defined deterministically depending on the joint assignment action.
If robot i is assigned to task p , then p ( ϵ RT ip )
= 1 , otherwise 0.
The element of feature set D k associated with the graph g k is defined as:
• D R k = { d R i | i ∈ R} is the set of node features for the robot nodes in R at epoch k .
In MRRC, d R i is defined as the location of robot i at epoch k (epoch index k is omitted).
• D T k = { d T p | p ∈ T k } is the set of node features for the task nodes in T k at epoch k .
In MRRC, d T p is defined as the age of task p at epoch k (epoch index k is omitted).
• D T T k = { d T T pq | p ∈ T k , q ∈ T k } is the set of task-task edge features at epoch k .
d T T pq denotes the duration for a robot that has just completed task p to subsequently compete task q .
We call this duration task completion time.
In MRRC, a task completion time is given as a random variable (in practice, our method only requires a set of samples of random variable).
• D RT k = { d RT ip | i ∈ R , p ∈ T k } is the set of robot-task edge features at epoch k .
d RT ip denotes the traveling time for robot i to reach task p .
3.2 Action
An action a k , a joint assignment at epoch k , is defined as a maximal bipartite matching of the complete bipartite graph ( R , T k , E RT k ) composed of the robot nodes R , the remaining task nodes T k , and the fully connected edges between them E RT k .
That is, given the current state s k =
( g k , D k ) , a k is a subset of E RT k satisfying (1) no two robots can be assigned to the same tasks, and (ii) a robot may only remain without assignment when the number of robots exceeds the number of remaining tasks.
If ϵ RT ip ∈ a k , it means that robot i is assigned with task p at epoch k .
For example, Figure 1 shows the case where a k =
( ϵ RT 1 , 1 , ϵ RT 2 , 3 ) .
(Note, we equivalently may say this as ϵ RT 1 , 1 = ϵ RT 2 , 3 = 1 andotherwise.)
In MRRC, all robots are allowed to change their assignments at each decision epoch.
(In
IPMS and mTSP, only free machines/salesmen are newly assigned.)
3.3 State transition
As the joint assignment a k is executed given the current state s k
= ( g k , D k ) , the next state s k
+
=
( g k +1 , D k +1 ) is determined with the updated graph g k +1 and features D k +1 .
Graph update.
When the decision epoch corresponds to the point when task p is completed, the corresponding task node will be removed in the updated task nodes as T k
+
=
T k / { p } , and the task-task edges and robot-task edges,
E T T
k
+
and E RT k +1 , will be accordingly updated.
Feature update.
At decision epoch k + 1 , D k +1 =
( D R k +1 ,
D T k
+1 , D T T k +1 , D RT k +1 ) is determined.
In
MRRC, task locations
D R k +
=
{ d R i | i ∈ R} and task ages D T k
+1 =
{ d T p |
p ∈ T k +1 } are updated.
The robot-task edge features D RT k +1 will be updated according to D R
k +1 as well.
3.4 Reward and objective
At time 0 , each task is given an initial age which increases linearly in time.
A reward r k = r (cid:0) d T p (cid:1) is given when a task p ∈ T k , whose age is d T p , is served at epoch k .
We consider linear and nonlinear reward functions r for MRRC.
The objective is to learn a stationary policy π , a function that maps current state s into current action a , to maximize expected total collected rewards Q π ( s, a ) =:
E P,π [ (cid:80) ∞ k =
R ( s t k , a t k , s t k +1 ) | s t
= s, a t 0 = a ] .
Random graph embedding: RandStructure2Vec
We observe that when task completion time is not deterministic, a scheduling problem can be represented as an extension of probabilistic graphical models (PGMs), which we refer to as random
PGMs.
This section proposes a mean-field inference method, random struct2vec , for random PGMs to estimate the state-action value Q ( s k , a k ) in solving MRRC.
4.1 Random PGM for representing a state of MRRC
Given random variables X
= { X p } , suppose that we can factor the joint distribution p ( X ) as p ( X )
= 1 Z (cid:81) i ϕ i
( D i ) where ϕ
i ( D i ) denotes a marginal distribution or conditional distribution associated with a set of random variables
D i ; Z is a normalizing constant.
Then { X p } is called a probabilistic graphical model (PGM).
In a PGM, D i is called a clique and ϕ i ( D i ) is called the clique potential for D i , and D i is called the scope of ϕ i .
We often write simply ϕ i , suppressing D i .
Starting from a state s k and an action a k , one can conduct a random experiment of “sequential decision making using policy π ”.
In this random experiment, we can denote the events ‘How robots serve all remaining tasks in which sequence’ as scenarios .
For example, suppose that at time-step k we are given robots { r 1 , r 2 } , tasks { t 1 , t 2 , t 3 , t 4 , t 5 , t 6 } and we follow the policy π onward.
One possible scenario is that robot r 1 serves tasks { t 1 → t 4 } and robot r 2 serves tasks { t 3 → t 2 → t 5 → t 6 } (see Figure 1).
Note that the time when t 5 is served depends on the time when t 2 is served (state transition is Markovian); thus the reward from t 5 depends on the reward from t 2 .
As shown in Figure 1, {{ t 1 → t 4 } , { t 3 → t 2 → t 5 → t 6 }} can be represented as a single instance of a Bayesian Network.
Since scenario realization is random, we can construct the distribution over such scenarios using ’random’ Bayesian Network with random node X k
=
( s k , a k ) and clique potential ϕ .
For details, see section 5.1.
4.2 Mean-field inference with random PGM
Let X =
{ X p } be the set of all random variables in the inference problem.
Let G X be the set of all possible PGMs on X .
Let P : G X (cid:55)→ [0 , 1] be a probability measure on G X .
Define a random PGM on X as {G X , P} .
Note that the inference of {G X , P} will be difficult; |G X | is too large for inferring
P even using Monte-Carlo sampling approach.
To avoid this difficulty, we use the approximated inference using semi-cliques .
Suppose that we are given the set of all possible cliques on X as C X .
As a PGM will be realized according to P , only a few of the possible cliques in C X will be actually realized as an element of the PGM and become real cliques.
We call such potential clique elements of C X as semi-cliques .
Note that if we were given P , we could calculate the presence probability p m of the semi-clique D m as p m =
(cid:80) G ∈G X P ( G ) 1 D m ∈ G , where 1 denotes the indicator function.
Mean-field inference with random PGM.
We start from a specific inference problem and state the main theorem in more general way.
Consider a random PGM on X
=
( { H i } , { X j } ) where H k is the latent variable corresponding to the observed variable X k .
Our goal is to infer { H i } given { X j } by finding p ( { H i }|{ x j } ) .
In mean-field inference, we instead find a set of surrogate distributions { q { x j } ( H i ) } for which { H i } are independent.
Here q { x j } means that q is a function of { x j } ).
We next state Theorem 1.
in a very general manner.
The statement is the same as that of mean-field inference with PGM Koller & Friedman (2009) except that ours has the presence probability terms { p m } of semi-cliques {D m } .
The implication is that inference of presence probability of each semi-clique is enough to conduct mean-field inference, and the inference of {G X , P} is not needed.
Theorem 1 .
Random PGM based mean field inference.
Suppose a random PGM on X =
{ X p } is given, and the presence probability { p m } for all semi-cliques D m ∈ C X are known.
Then, the surrogate distribution { q p ( x p ) } in mean-field inference is optimal only if q p ( x p )
= 1 Z p exp (cid:110)(cid:80) m : X p ∈D m p m E ( D m −{ X p } ) ∼ q
[ln ϕ m ( D m , x p )] (cid:111) where Z p is a normalizer and ϕ m clique potential for D m .
For the general background and proof, see Appendix C.
RandStructure2Vec.
In Dai et al.
(2016), structure2vec was derived as a vector space embedding of mean-field inference with PGM.
For detailed background on vector space embedding, see Appendix
D. From Theorem 1 , we derive random structure2vec as a vector-space embedding of mean-field inference with random PGM.
Suppose that once a PGM is realized the PGM has joint distribution proportional to some factorization (cid:81)
p ϕ ( H p | I p ) (cid:81) p,q ϕ ( H p | H q ) (as in Dai et al.
Under this assumption, we can write { q { x j } ( H i ) } as { q x i ( H i ) } .
In Dai et al.
, they suggest that structure2vec is essentially a fixed point iteration ˜ µ p ← σ ( W 1 x p + W 2 (cid:80) q ̸ = p ˜ µ q ) where ˜ µ p is a latent vector for node p and x p is input for node p .
They show that, when we interpret ˜ µ i as a vector space injective embedding expressed as ˜ µ i = (cid:82) H ϕ ( h i )
q x i
( h i ) dh
i for some ϕ , structure2vec’s fixed point iteration is the embedding of fixed point iteration of mean-field inference with the PGM.
Lemma 1 states that we have a similar result for random PGM by including only the { p qp } information.
Lemma 1 .
Structure2vec for random PGM.
Assume that the presence probabilities { p qp } for all pairwise semi-cliques D qp ∈ C X are given.
Then embedding the fixed point equation in Theoremgenerates the fixed point equation ˜ µ p ← σ (cid:16) W 1 x p + W 2 (cid:80) q ̸ = p p qp ˜ µ q (cid:17) .
We refer to this fixed point iteration as random structure2vec.
(The proof of Lemma 1 can be found in Appendix E.)
Remarks.
Note that inference of {G X , P} is in general a difficult task.
One implication of Theorem 1 is that we transformed a difficult inference task into a simple inference task : inferring the presence probability of each semi-clique.
(See Appendix F for the algorithm that conducts this task.)
In addition, Lemma 1 provides a theoretical justification to ignore the inter-dependencies among edge presences when embedding a random graph using GNN.
When graph edges are not explicitly given or known to be random, the simplest heuristic one can use is to separately infer the presence probabilities of all edges and adjust the weights of GNN’s message propagation.
According to Lemma 1 , possible inter-dependencies among edges would not affect the quality of such heuristic inference.
Solving MRRC with RandStructure2Vec
This section describes how the proposed method solves the MRRC problem with the newly developed random structure2vec method.
The total solution (i.e., the sequential assignments of robots/machines to tasks) is found by iteratively repeating a sequential decision making.
This section specifically describes how to choose a joint assignment a k given the current state s k
= ( g k , D k ) .
The procedure for determining a joint assignment action is composed of (1) represent the state using a random
Bayesian Network, (2) estimate the Q-value using random graph embedding, and (3) select a joint assignment.
Figure 2 depicts the overall procedure.
This section focuses on MRRC; the appendices will provide the formulation, solution procedure, and results for IPMS and mTSP problems as well.
5.1 Representing a state using a random PGM
In this section, we describe how the MRRC problem state s k and one possible joint assignment a k can be represented as a Bayesian network.
Assume H p , the hidden random variable for task p , carries information about the benefit of serving task p .
Given a scenario (see section 3.1), H p depends on H q only if q is served after p by the same robot, and H p depends on X p , the observed features for task p .
For an MRRC problem, X p can be either a task’s assignment information d RT ip or a task’s age d T p .
Based on these definitions, a Bayesian Network for { H p } and { X p } is constructed as p ( { H p }|{ X p } ) =
(cid:81) p ϕ
( H p | X p ) (cid:81) p,q ϕ ( H p | H q ) .
This Bayesian Network corresponds to one scenario; there are many possible scenarios that can be realized randomly.
Since random PGM naturally models this characteristic, according to Lemma 1, we are justified to use random structure2vec with edge (semi-clique
presence probabilities { p ( ϵ T T pq ) } from in section 2.1.
5.2 Estimating state-action value using Order trainability-enabled Q-function
We illustrate how we can estimate a Q-function for MRRC by applying random structure2vec to the random PGM that represents ( s k , a k ).
Lemma 1 provides the fixed point equation ˜ µ p = σ (cid:16) W 1 x p + W 2 (cid:80) j ̸ = k p qp ˜ µ q (cid:17) to compute the embeddings ˜ µ p for node task p in a random PGM.
The embeddings ˜ µ p for p ∈
T
k are computed in an iterative manner using random structure2vec as: ˜ µ ( τ +1) p = σ (cid:16) W 1 x p + W 2 (cid:80) q ̸ = p p qp ˜ µ ( τ ) q (cid:17) .
We propose a network architecture composed of two-step sequential random structure2vec , which is composed of action embedding and value embedding .
Action Embedding.
The first random structure2vec layer embeds the joint assignment into the task nodes T k
= { 1 , ..., n } .
The action embedding ˜ µ Ap for task node p is defined as the fixed point for the equation ˜ µ
Ap = σ (cid:16) W A 1 x Ap + W A 2 (cid:80) q ̸ = p p qp ˜ µ
Aq (cid:17) where x Ap =
d RT ip (distance from robot i to task p ) when task p is assigned to robot i , x Ap = 0 when task p is not assigned.
The action node embeddings { ˜ µ
Ap |
p ∈ T k } , computed via iteration simultaneously by employing random structure2vec , provide sufficient information about the relative locations between robots and their assigned tasks.
Value Embedding.
The second random structure2vec layer embeds the task ages into the task nodes.
The value embedding ˜ µ Vp for task node p is defined as the fixed point for the equation ˜ µ
Vp = σ (cid:16) W V 1 x Vp + W V 2 (cid:80)
q ̸ = p p qp ˜ µ
Vq (cid:17) where x Vp =
(˜ µ Ap , d T p ) is the concatenation of the action embedding ˜ µ
Ap computed by the first layer and the task age d T p for node p .
The resulting value node embeddings { ˜ µ Vp | p ∈ T k } , computed in an iterative manner, provide sufficient information about how much value is likely in the local graph around each task by the specified joint assignment.
Computing Q
θ ( s k , a k ) .
To derive Q θ ( s k , a k ) , we aggregate the embedding vectors for all nodes by ˜ µ V = (cid:80) p ˜ µ
Vp to obtain one global vector ˜ µ V to embed the value affinity of the global graph.
We then use a neural network to map ˜ µ V into Q θ ( s k , a k ) .
The overall pseudo-code for estimating steps is provided in Appendix G.
It is essential for the proposed inference method can estimate the state action value Q θ ( s k , a k ) for varying graph size g k .
Let us provide the intuition related to scuh problem-size transferability.
For Action Embedding , transferability is trivial; the inference problem is a scale-free task locally around each node .
For Value Embedding , consider the ratio of robots to tasks.
The overall value affinity embedding will be underestimated if this ratio in the training environment is smaller than this ratio in the testing environment; overestimated overall otherwise.
The intuition is that this over/under-estimation does not matter in Q-function based policies as discussed in van Hasselt et al.
(2015) as long as the order of Q-function value among actions are the same.
That is, as long as the best assignments chosen are the same, i.e., arg max a k Q ( s k , a k ) =
arg max a k Q θ ( s k , a k ) , the magnitude of imprecision | Q ( s k , a k ) − Q
θ ( s k , a k ) | does not matter.
We call this property of an estimator order-transferability (with respect to the max operation).
5.3 Selecting a joint assignment using OTAP
With the previously introduced way to estimate Q θ ( s k , a k ) , we illustrate how to compute the joint assignment (action) a ∗ k , a maximal bipartite matching in the bipartite graph ( R , T k , E RT k ) , given the state s k
= ( g k , D k ) .
Specifically, we propose the order transferability-enabled auction policy (OTAP) that constructs a joint assignment a k through N = max ( |R| , | k | ) iterations of Bidding and
Consensus phases.
This auction follows the spirit of Sequential Single Item (SSI) auctioning (Koenig et al.
; each iteration adds one robot-task assignment to construct a full joint assignment.
Bidding-phase.
In the n th iteration of the bidding phase, given M ( n − 1) θ , the ordered set of n −robot-task edges in E RTk determined by the previous n − 1 iterations, all the unassigned robots bid their the most preferable task to conduct.
Respecting robot-task assignments determined in previous n − 1 iterations and ignoring other unassigned robots, each unassigned robot i select the best task assignment ϵ RT il that maximizes Q nθ ( s k , M ( n − 1) θ ∪ { ϵ RT ip } ) among all unassigned tasks p ∈ T k , where Q nθ is the θ -parameterized network with superscript n indicating that the state action value is estimated at n th iteration.
Then, robot i bids { ϵ RT iℓ , Q nθ ( s k , M ( n − 1) θ ∪ { ϵ RT iℓ } ) } to the auctioneer.
This bidding occurs simultaneously by all the unassigned robots at the n th iteration.
Since the number of ignored robots varies at each iteration, transferability of Q-function inference is crucial.
Consensus-phase.
In the consensus phase of n th iteration, the centralized auctioneer finds the bid with the best bid value, say { ϵ RT
i ∗ p ∗ , Q nθ ( s k , M ( n − 1) θ ∪{ ϵ RT i ∗ p ∗ } ) } (Here i ∗ and p ∗ denote the best robot task pair.)
Denote ϵ RT i ∗ p ∗ :
= m ( n ) θ .
The auctioneer sets everyone’s M ( n ) θ as M ( n ) θ =
M ( n − 1) θ ∪ m ( n ) θ and initiate bidding phase for the remaining unassigned robots.
These two phases iterate until reaching M ( N ) θ = { m (1) θ , . . .
, m ( N ) θ } .
This M ( N ) θ is chosen as the joint assignment a ∗ k of N -robots at time step k .
That is, π θ ( s k ) = a ∗ k .
The computational complexity for computing π θ is O
( | R | | T | ) and is only polynomial; see Appendix L.
5.4 Training Q-function using AFQI
The fitted Q-iteration (FQI) finds θ that minimizes E ( s k ,a k ,r k ,s k +1 ) ∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) + γ max a Q θ ( s k
+1 , a )]] where D denotes the distribution of training data.
We propose a new reinforcement learning method, which we call Auction-fitted Q-iteration (AFQI) , which replaces max a
Q θ ( s k , a ) used in the conventional FQI with OTAP.
That is, writing OTAP as π Q θ , AFQI finds θ that empirically minimizes E ( s k ,a k ,r k ,s k +1 ) ∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) + γQ θ ( s k +1 , π Q θ ( s k +1 ))]] .
In learning the parameters θ for Q θ ( s k , a k ) , we use the exploration strategy that perturbs the parameters θ randomly to actively explore the joint assignment space with OTAP.
While this method was originally developed for policy-gradient based methods Plappert et al.
(2017), exploration in parameter space is useful in our auction-fitted Q-iteration since it generates a reasonable combination of assignments.
Theoretical analysis
We show the proposed AFQI obtains at least 1 − 1 /e optimality and enables computation of the joint assignment in polynomial time.
This result is achieved by the order-transferability of the proposed
Q -function estimator and its use in selecting the joint assignment.
6.1 Performance bound of OTAP
Recall that Q n denotes the n -robot problem’s true Q -function.
In the same way as we defined M ( N ) θ above, denote the joint assignment chosen by OTAP as { Q n } Nn =1 as M ( N ) = { m (1) , . . .
, m ( N ) } .
Lemma 2.
If the Q -function approximator has order transferability, then M ( N ) =
M ( N ) θ .
For any decision epoch k , let M denote a set of robot-task pairs (a subset of E RT k ).
For any robot-task pair m ∈ E RT k , define ∆( m | M ) :
=
Q |M∪{ m }| ( s k , M∪{ m } ) − Q |M|
( s k , M ) as the the marginal value (under the true Q-functions) of adding robot-task pair m ∈ E RT k .
Lemma 2 enables us to use the result discussed in Nemhauser et al.
and achieve the result of Theorem 2.
Theorem 2.
Suppose that the Q -function approximation with the parameter value θ exhibits order transferability.
Denote M ( N ) θ as the result of OTAP using { Q nθ } Nn =1 and let M ∗ = argmax a k
Q | a k | ( s k , a k ) .
If ∆( m | M ) ≥ 0 , ∀M ⊂ E RTk , ∀ m ∈ E RT k , and the marginal value of adding one robot diminishes as the number of robots increases, i.e., ∆( m | M ) ≤ ∆( m | N ) , ∀N ⊂ M ⊂ E RTk , ∀ m ∈ E RT k , then the result of OTAP is at least better than 1 − 1 /e of an optimal assignment.
That is,
Q Nθ ( s k , M ( N ) θ ) ≥ Q |M ∗ | ( s k , M ∗ )(1 − 1 /e ) .
See Appendix H and K for the proofs.
6.2 Performance bound of AFQI
AFQI seeks to find θ that minimizes E ( s k ,a k ,r k ,s k +1 )
∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) + γQ θ ( s k +1 , π Q θ ( s k +1 ))]] .
Here, we use OTAP, denoted as π Q θ , instead of max a Q θ ( s k , a ) which is used in general fitted-Q iteration.
As we have seen in section 5.1, OTAP replaces the max operation with the auction algorithm with a provable performance bound compared with the max operation.
Lemma 3 allows us to use this performance bound to obtain a performance assurance on AFQI compared with FQI.
We only write an abbreviated version of the statement for brevity.
The formal description of conditions and the proof is provided in Appendix I.
Lemma 3.
Kang & Kumar (2021) Suppose that a 1 − 1 /r approximation algorithm is substituted for the max operation in FQI.
Then, the corresponding new Fitted Q-iteration’s performance is at least 1 − 1 /r optimal.
Corollary 1.
AFQI achieves at least 1 − 1 /e performance compared with the optimal policy.
[Table] [Table]
[Table]
6.3 Experiment settings
In the main text
[Table]
we focus on discrete-time & discrete-state (DTDS)
MRRC problems with deterministic and stochastic task completion times.
For CTCS deterministic problems with real-world datasets, see IPMS (Appendix A) and mTSP (Appendix B).
Environment.
Since there is no standard dataset for MRRC problems, we used the complex maze-like environment generator of Neller et al.
(2010) (code provided in Appendix 10).
This complex maze mimics the complex road layout of a city and random traffic, inducing nontrivial task completion times.
See the leftmost image of Figure 2 and the supplementary video.
We randomly generated a new maze for every training and testing experiment with randomly chosen initial task/robot locations.
To generate the task completion times, Dijkstra’s algorithm and dynamic programming were used for deterministic and stochastic environments, respectively.
In the stochastic environment, a robot makes its intended move with a certain probability.
(Cells with a dot: success with 55%, every other direction with 15% each.
Cells without a dot: 70% and 10%, respectively.)
A task is considered served when a robot reaches it.
We consider two reward rules: linearly decaying rewards f ( age ) =
max { 200 − age, 0 } and nonlinearly decaying rewards f ( age ) =
λ age with λ = 0.99, where age is the task age when served.
The initial age of tasks are uniformly distributed in the interval [0 , 100] .
Baselines.
For deterministic environments with linear rewards, where the corresponding MRRC can be formulated as a mixed-integer linear program (MILP), we consider the following two baselines:
• Optimal : Gurobi Gurobi Optimization (2019), an off-the-self the optimization solver for MILP, was used to solve the problems with 60-min time limit.
• Ekici et al.
: Ekici & Retharekar (2013), the most up-to-date heuristic for solving MRRC in the Operations Research community, was used the problems.
For stochastic environments or exponential rewards, to our knowledge, there is no literature addressing
MRRC with.
Thus, we construct an indirect baseline:
• Sequential Greedy Algorithm (SGA) : a general-purpose multi-robot task allocation algorithm
called SGA Han-Lim Choi et al.
The performance measure we used is ρ = Rewards
collected by the proposed method Reward collected
by the baseline .
Thus, the value of ρ greater than 100% indicates the proposed method collects more reward than the corresponding baseline algorithm.
Note that ρ against Optimal is always lower than 100%.
Note that we cannot provide other reinforcement learning-based heuristics as additional baselines since, to the best of our knowledge (for the class of NP-hard multi-robot/machine scheduling problems with decaying rewards), this paper is the first to propose a reinforcement learning-based heuristic.
6.4 Performance test.
Performance was tested under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.
See Table 1.
Our method achieves near-optimality for linear/deterministic rewards with 3% fewer rewards than optimal on average.
The standard deviation for ρ is provided in parentheses.
For other environments, we see that the %SGA ratio for linear/deterministic is well maintained.
Due to dataset generation’s dynamic programming computation complexity, we only consider 8 robots/50 tasks at maximum.
We considered larger size problems in IPMS experiments discussed in Appendix A.
6.5 Transferability test.
Table 2 provides comprehensive transferability test results.
The rows indicate training conditions, while the columns indicate testing conditions.
The results in the diagonal cells in red (cells with the same training size and testing size) serve as baselines (direct testing).
The results in the offdiagonal show the results for the transferability testing and demonstrate how the algorithms trained with different problem sizes perform well on test problems (zero-shot transfer).
We can see that lower-direction transfer tests (trained with larger size problems and tested with smaller size problems) show only a small loss in performance.
For upper-direction transfer tests (trained with smaller size problems and tested with larger size problems), the loss was up to 4 percent.
6.6 Scalability analysis.
For training complexity, we measured the training time required to achieve 93% optimality considering a deterministic environment with linear rewards.
Table 4 shows that training time may not necessarily increase as problem size gets larger, while the performance is fairly maintained.
MRRC can be formulated as a semi-MDP (SMDP) based multi-robot planning problem (e.g., Omidshafiei et al.
This problem’s complexity with R robots and T tasks and maximum H time horizon is O (( R ! /T !
( R − T )!)
H ) .
In our proposed method, this complexity is addressed by a combination of two complexities: computational complexity and training complexity.
For computational complexity of joint assignment decision at each timestep is O ( | R || T | 3 ) .
See Appendix L for details.
Concluding Remarks
In this paper, we addressed the challenge of developing a near-optimal learning-based method for solving NP-hard multi-robot/machine scheduling problems.
We developed a theory of meanfield inference for scheduling problems and a corresponding theoretically justified GNN method to precisely infer the Q-function.
We addressed the scalability issue of Fitted Q-Iteration methods for multi-robot/machine scheduling problems by providing a polynomial-time algorithm with a provable performance guarantee.
Simulation results demonstrate the effectiveness of the our methods.
Acknowledgement
Jinkyoo Park was supported by Institute of Information & communications Technology Planning
Evaluation (IITP) grant funded by the Korea government(MSIT)(2022-0-01032, Development of
Collective Collaboration Intelligence Framework for Internet of Autonomous Things).