Learning NP-Hard Joint-Assignment planning using GNN: Inference on a Random Graph and Provable Auction-Fitted Q-iteration
Abstract
The present paper explores the possibility of near-optimally solving multi-robot,multi-task NP-hard scheduling problems with time-dependent rewards using alearning-based algorithm.
In particular, we consider a class of robot/machinescheduling problems called the multi-robot reward collection problem (MRRC).
Such MRRC problems well model ride-sharing, pickup-and-delivery, and a varietyof related problems.
In rep resenting the MRRC problem as a sequential decisionmaking problem, we observe that each state can be represented as an extension ofprobabilistic graphical models (PGMs), which we refer to as random PGMs.
Wethen develop a mean-field inference method for random PGMs.
We then propose(1) an order-transferable Q-function estimator and (2) an order-transferabilityenabled auction to select a joint assignment in polynomial-time.
These result in areinforcement learning framework with at least 1 − 1 /e optimality.
Experimentalresults on solving MRRC problems highlight the near-optimality and transferabilityof the proposed methods.
We also consider identical parallel machine schedulingproblems (IPMS) and minimax multiple traveling salesman problems.
Introduction
Motivation Consider a set of identical robots seeking to serve a set of spatially distributed tasks.
Each task is given an initial age (which then increases linearly in time).
Greater rewards are given toyounger tasks when service is complete according to a predetermined reward rule.
Such problemsprevail in operations research, e g., dispatching drivers to transport customers or scheduling machinesin a factory.
Solving such highly structured NP-hard problems with the constraint of ‘no possibilityof two robots assigned to one task at once’ using mathematical optimization schemes is infeasibleor ineffective due to the expensive computational cost, especially when the problem size is large.
Applying decentralized approach using multi-agent modeling frameworkLong et al.
(2020); Rashidet al.
; Sunehag et al.
(2017) is a possible way to solve a large scale problems.
However, dueto the impossibility of inducing consensus among agents in achieving the global objective withouteffective communication Fischer et al.
(1985), such decentralized approaches are rarely used inindustries (e.g., factories).
Thus, this study focuses on centralized methods for solving MRRC.
Research Questions.
Many of such NP-hard scheduling problems have time-dependent rewards.
To the best of our knowledge, these problems have not yet been addressed by non-decentralizedlearning-based methods.
Even if one can, it must also be able to simultaneously address a fundamentalchallenge: the number of possible robot-task pairs to be considered increases exponentially.
Forinstance, scheduling 8 robots and 50 tasks involves 10 13 possible joint assignments at each time-step.
The main research question that the current study seeks to resolve is "how to design a computationally effective (i.e., scalability in terms of learning and decision making) a learning based centralizeddecision making scheme for solving a large-scale NP-hard scheduling problems?"
Proposed method and contributions.
The present paper explores the possibility of near-optimally solving multi-robot, multi-task NP-hard scheduling problems with time-dependent rewards using a learning-based algorithm.
The study formulates multi-robot, multi-task NP-hard scheduling problems in a sequential decision-making framework and derives a joint scheduling policy with a theoretical performance bound under reasonable assumptions.
The novelties of the current study are as follows:
• The study first observes that a state-joint assignment pair can be represented as a random PGM.After developing a theory of random PGM-based mean-field inference, we derive random structure2vec , a random PGM-based extension of structure2vec Dai et al.
(2016).• We estimate the Q -function Q ( s k , a k ) using layers of random structure2vec , where ( s k , a k ) isthe state-joint action pair.
Using an interpretation of a layer of structure2vec as a WeisfeilerLehman kernel (as in Dai et al.
, we design the estimator to possess a property we callorder-transferability.
This property enables transferability in problem size.•
We propose a joint assignment rule called order-transferability-enabled auction policy (OTAP)to address exponential growth in joint assignment space.
We propose auction-fitted Q -iteration(AFQI) by substitution of the argmax operation of fitted Q -iteration with OTAP to train theQ -function in a scalable manner.
We prove that AFQI results in a policy with polynomial-timecomputation that achieves at least 1 − 1 /e performance compared with the optimal policy.
Results and Impacts.
Using simulation experiments, we show that the proposed policy typicallyachieves 97% optimality for the multi-robot reward collection (MRRC) problem in a deterministicenvironment with linearly time-varying rewards.
This performance is well extended to experimentswith stochastic traveling times.
To the best of our knowledge, this result is the first to learn anear-optimal NP-hard multi-robot/machine scheduling policy with time-dependent rewards.
Related studies.
Recently, Dai et al.
(2017) showed that a graph neural network (GNN) called structure2vec Dai et al.(2016) can construct a solution for the Traveling Salesman Problem (TSP).
structure2vec is a popular
GNN derived from mean-field inference with a probabilistic graphical model (PGM).
Dai et al.
(2017)formulates the TSP as a Markov decision process (MDP) where a heuristically constructed PGMrepresents each state-next assignment pair.
They employ structure2vec derived from the heuristic
PGM to infer the Q -function, which they use to select the next assignment.
While their choiceof PGM was heuristic, their approach achieved near-optimality and transferability of their trainedsingle-robot scheduling algorithm to new single-robot scheduling problems with an unseen numberof tasks.
However, those successes were restricted to single-robot problems except for special caseswhere the problem can be modeled as a variant of single-robot TSP via multiple successive journeysof a single robot Nazari et al.
(2018); Kool et al.
Multi-Robot Reward Collection Problem
In the main text, we model a general multi robot/machine scheduling problem as a discrete-time,discrete-state (DTDS) sequential decision-making problem.
In the DTDS model, time advancesin fixed increments ∆ , i.e., t k = t 0 + ∆ × k where t k is the actual time after k decision epochshave passed.
For simplicity, we use k as a time index representing the k th decision epoch.
Inthis framework, s k denotes a state, and action a k denotes a joint assignment of robots/machines tounfinished tasks at the k th epoch.
The objective of the problem is to learn the optimum schedulingpolicy π θ : s k
→ a k that maximizes the reward collected or minimizes the total completion time(a.k.a.
makespan minimization).
Below is the formulation of MRRC.
We additionally propose a
The state s k at epoch k is represented as ( g k , D k ) where a graph g k
= (( R , T k ) , ( E T T k , E RT k )) andassociated feature set D k = ( D R k , D T k , D T T k , D RT k ) .
The elements of graph g k are defined as (See
Figure 1):
• R = { 1 , ..., M } is the index set of all robots.
The index i and j will be used to specifically denoterobots.• T k
= { 1 , ..., N } is the index set of all remaining unserved tasks at decision epoch k .
The index pand q will be used to specifically denote tasks.
• E T T k
= { ϵ T T pq | p ∈ T k , q ∈ T k } is the set of all directed edges from a task in T k to any task inT k .
We consider each edge as a random variable.
The task-to-task edge ϵ T T pq = 1 indicates theevent that a robot that has just completed task p subsequently completes task q .
We denote theprobability p ( ϵ T T pq = 1) ∈ [0 , 1] the presence probability of the edge ϵ T T pq .•
E RT k = { ϵ RT ip | i ∈ R , p ∈ T k } is the set of all directed edges from a robot in R to a task in T k .We say the robot-to-task edge ϵ RT ip = 1 indicates the event that robot i is assigned to the taskp .
This edge is defined deterministically depending on the joint assignment action.
If robot i isassigned to task p , then p ( ϵ RT ip ) = 1 , otherwise 0.
The element of feature set D k associated with the graph g k is defined as:
• D R k = { d R i | i ∈ R} is the set of node features for the robot nodes in R at epoch k .
In MRRC, d R iis defined as the location of robot i at epoch k (epoch index k is omitted).
• D T k = { d T p | p ∈ T k } is the set of node features for the task nodes in T k at epoch k .
In MRRC, d T pis defined as the age of task p at epoch k (epoch index k is omitted).
• D T T k = { d T T pq | p ∈ T k , q ∈ T k } is the set of task-task edge features at epoch k .
d T T pq denotes theduration for a robot that has just completed task p to subsequently compete task q .
We call thisduration task completion time.
In MRRC, a task completion time is given as a random variable(in practice, our method only requires a set of samples of random variable).• D RT k
= { d RT ip | i ∈ R , p ∈ T k } is the set of robot-task edge features at epoch k .
d RT ip denotes thetraveling time for robot i to reach task p .
3.2 Action
An action a k , a joint assignment at epoch k , is defined as a maximal bipartite matching of thecomplete bipartite graph ( R , T k , E RT k ) composed of the robot nodes R , the remaining task nodes T k ,and the fully connected edges between them E RT k .
That is, given the current state s k =
( g k , D k ) , a kis a subset of E RT k satisfying (1) no two robots can be assigned to the same tasks, and (ii) a robot mayonly remain without assignment when the number of robots exceeds the number of remaining tasks.
If ϵ RT ip ∈ a k , it means that robot i is assigned with task p at epoch k .
For example, Figure 1 showsthe case where a k =
( ϵ RT 1 , 1 , ϵ RT 2 , 3 ) .
(Note, we equivalently may say this as ϵ RT 1 , 1 = ϵ RT 2 , 3 = 1 and 0otherwise.)
In MRRC, all robots are allowed to change their assignments at each decision epoch.
(In
IPMS and mTSP, only free machines/salesmen are newly assigned.)
3.3 State transition
As the joint assignment a k is executed given the current state s k
= ( g k , D k ) , the next states k
+
=
( g k +1 , D k +1 ) is determined with the updated graph g k +1 and features D k +1 .
Graph update.
When the decision epoch corresponds to the point when task p is completed, thecorresponding task node will be removed in the updated task nodes as T k
+1 =
T k / { p } , and thetask-task edges and robot-task edges, E T T
k +1 and E RT k +1 , will be accordingly updated.
Feature update.
At decision epoch k + 1 , D k +1 =
( D R k +1 ,
D T k
+1 , D T T k +1 , D RT k +1 ) is determined.
In
MRRC, task locations
D R k +
=
{ d R i | i ∈ R} and task ages D T k
+1 =
{ d T p |
p ∈ T k +1 } are updated.
The robot-task edge features D RT k +1 will be updated according to D R
k +1 as well.
3.4 Reward and objective
At time 0 , each task is given an initial age which increases linearly in time.
A reward r k = r (cid:0) d T p (cid:1) isgiven when a task p ∈ T k , whose age is d T p , is served at epoch k .
We consider linear and nonlinearreward functions r for MRRC.
The objective is to maximize expected total collected rewards.
Random graph embedding: RandStructure2Vec
We observe that when task completion time is not deterministic, a scheduling problem can berepresented as an extension of probabilistic graphical models (PGMs), which we refer to as random
PGMs.
This section proposes a mean-field inference method, random struct2vec , for random PGMsto estimate the state-action value Q ( s k , a k ) in solving MRRC.
4.1 Random PGM for representing a state of MRRC
Given random variables X
= { X p } , suppose that we can factor the joint distribution p ( X ) asp ( X )
= 1 Z (cid:81) i ϕ i
( D i ) where ϕ
i ( D i ) denotes a marginal distribution or conditional distribution associated with a set of random variables
D i ; Z is a normalizing constant.
Then { X p } is called aprobabilistic graphical model (PGM).
In a PGM, D i is called a clique and ϕ i ( D i ) is called the cliquepotential for D i , and D i is called the scope of ϕ i .
We often write simply ϕ i , suppressing D i .
Starting from a state s k and an action a k , one can conduct a random experiment of “sequentialdecision making using policy π ”.
In this random experiment, we can denote the events ‘How robotsserve all remaining tasks in which sequence’ as scenarios .
For example, suppose that at time-step k weare given robots { r 1 , r 2 } , tasks { t 1 , t 2 , t 3 , t 4 , t 5 , t 6 } and we follow the policy π onward.
One possiblescenario is that robot r 1 serves tasks { t 1 → t 4 } and robot r 2 serves tasks { t 3 → t 2 → t 5 → t 6 }(see Figure 1).
Note that the time when t 5 is served depends on the time when t 2 is served (statetransition is Markovian); thus the reward from t 5 depends on the reward from t 2 .
As shown in Figure1, {{ t 1 → t 4 } , { t 3 → t 2 → t 5 → t 6 }} can be represented as a single instance of a Bayesian Network.
Since scenario realization is random, we can construct the distribution over such scenarios using’random’ Bayesian Network with random node X k =
( s k , a k ) and clique potential ϕ .
4.2 Mean-field inference with random PGM
Let X =
{ X p } be the set of all random variables in the inference problem.
Let G X be the set of allpossible PGMs on X .
Let P : G X (cid:55)→ [0 , 1] be a probability measure on G X .
Define a random PGMon X as {G X , P} .
Note that the inference of {G X , P} will be difficult; |G X | is too large for inferring
P even using Monte-Carlo sampling approach.
To avoid this difficulty, we use the approximatedinference using semi-cliques .
Suppose that we are given the set of all possible cliques on X as C X .
As a PGM will be realized according to P , only a few of the possible cliques in C X will be actuallyrealized as an element of the PGM and become real cliques.
We call such potential clique elementsof C X as semi-cliques .
Note that if we were given P , we could calculate the presence probability p mof the semi-clique D m as p m =
(cid:80) G ∈G X P ( G ) 1 D m ∈ G , where 1 denotes the indicator function.
Mean-field inference with random PGM.
We start from a specific inference problem and state themain theorem in more general way.
Consider a random PGM on X
=
( { H i } , { X j } ) where H k isthe latent variable corresponding to the observed variable X k .
Our goal is to infer { H i } given { X j }by finding p ( { H i }|{ x j } ) .
In mean-field inference, we instead find a set of surrogate distributions{ q { x j } ( H i ) } for which { H i } are independent.
Here q { x j } means that q is a function of { x j } ).
We next state Theorem 1.
in a very general manner.
The statement is the same as that of mean-fieldinference with PGM Koller & Friedman (2009) except that ours has the presence probability terms{ p m } of semi-cliques {D m } .
The implication is that inference of presence probability of eachsemi-clique is enough to conduct mean-field inference, and the inference of {G X , P} is not needed.
Theorem 1 .
Random PGM based mean field inference.
Suppose a random PGM on X =
{ X p }is given, and the presence probability { p m } for all semi-cliques D m ∈ C X are known.
Then, the surrogate distribution { q p ( x p ) } in mean-field inference is optimal only if q p ( x p ) =
[Equation]
p )]
[Equation]
where Z p is a normalizer and ϕ m cliquepotential for D m .
For the general background and proof, see Appendix C.
RandStructure2Vec.
In Dai et al.
(2016), structure2vec was derived as a vector space embedding ofmean-field inference with PGM.
For detailed background on vector space embedding, see Appendix
D. From Theorem 1 , we derive random structure2vec as a vector-space embedding of mean-fieldinference with random PGM.
Suppose that once a PGM is realized the PGM has joint distributionproportional to some factorization (cid:81)
p ϕ ( H p | I p ) (cid:81) p,q ϕ ( H p | H q ) (as in Dai et al.
Underthis assumption, we can write { q { x j } ( H i ) } as { q x i ( H i ) } .
In Dai et al.
, they suggest thatstructure2vec is essentially a fixed point iteration ˜
µ p ← σ ( W 1 x p + W 2 (cid:80) q ̸ = p ˜ µ q ) where ˜ µ p is alatent vector for node p and x p is input for node p .
They show that, when we interpret ˜ µ i as a vectorspace injective embedding expressed as ˜ µ i = (cid:82) H ϕ ( h i )
q x i
( h i ) dh
i for some ϕ , structure2vec’s fixedpoint iteration is the embedding of fixed point iteration of mean-field inference with the PGM.
Lemma1 states that we have a similar result for random PGM by including only the { p qp } information.
Lemma 1 .
Structure2vec for random PGM.
Assume that the presence probabilities { p qp } for allpairwise semi-cliques D qp ∈ C X are given.
Then embedding the fixed point equation in Theorem 1generates the fixed point equation ˜ µ p ← σ (cid:16) W 1 x p + W 2 (cid:80) q ̸ = p p qp ˜ µ q (cid:17) .
We refer to this fixed pointiteration as random structure2vec.
(The proof of Lemma 1 can be found in Appendix E.)
Remarks.
Note that inference of {G X , P} is in general a difficult task.
One implication of Theorem1 is that we transformed a difficult inference task into a simple inference task : inferring the presenceprobability of each semi-clique.
(See Appendix F for the algorithm that conducts this task.)
Inaddition, Lemma 1 provides a theoretical justification to ignore the inter-dependencies among edgepresences when embedding a random graph using GNN.
When graph edges are not explicitly given orknown to be random, the simplest heuristic one can use is to separately infer the presence probabilitiesof all edges and adjust the weights of GNN’s message propagation.
According to Lemma 1 , possibleinter-dependencies among edges would not affect the quality of such heuristic inference.
This section describes how the proposed method solves the MRRC problem with the newly developedrandom structure2vec method.
The total solution (i.e., the sequential assignments of robots/machinesto tasks) is found by iteratively repeating a sequential decision making.
This section specificallydescribes how to choose a joint assignment a k given the current state s k =
( g k , D k ) .
The procedurefor determining a joint assignment action is composed of (1) represent the state using a random
Bayesian Network, (2) estimate the Q-value using random graph embedding, and (3) select a jointassignment.
4.3 Representing a state using a random PGM
In this section, we describe how the MRRC problem state s k and one possible joint assignmenta k can be represented as a Bayesian network.
Assume H p , the hidden random variable for taskp , carries information about the benefit of serving task p .
Given a scenario (see section 3.1), H pdepends on H q only if q is served after p by the same robot, and H p depends on X p , the observedfeatures for task p .
For an MRRC problem, X p can be either a task’s assignment information d RT ip ora task’s age d T p .
Based on these definitions, a Bayesian Network for { H p } and { X p } is constructedas p ( { H p }|{ X p } )
=
(cid:81) p ϕ
( H p | X p ) (cid:81) p,q ϕ ( H p | H q ) .
This Bayesian Network corresponds to onescenario; there are many possible scenarios that can be realized randomly.
Since random PGM naturally models this characteristic, according to Lemma 1, we are justified to use random structure2vecwith edge (semi-clique
presence probabilities { p ( ϵ T T pq ) } from in section 2.1.
4.4 Estimating state-action value using Order trainability-enabled Q-function
We illustrate how we can estimate a Q-function for MRRC by applying random structure2vecto the random PGM that represents ( s k , a k ).
Lemma 1 provides the fixed point equation ˜ µ p =σ (cid:16) W 1 x p + W 2 (cid:80) j ̸ = k p qp ˜ µ q (cid:17) to compute the embeddings ˜ µ p for node task p in a random PGM.
The embeddings ˜ µ p for p ∈
T
k are computed in an iterative manner using random structure2vec as:˜ µ ( τ +1) p
=
σ (cid:16) W 1 x p + W 2 (cid:80) q ̸ = p p qp ˜ µ ( τ ) q (cid:17) .
We propose a network architecture composed of two-stepsequential random structure2vec , which is composed of action embedding and value embedding .
Action Embedding.
The first random structure2vec layer embeds the joint assignment into the tasknodes T k
= { 1 , ..., n } .
The action embedding ˜ µ Ap for task node p is defined as the fixed point for theequation ˜ µ
Ap = σ (cid:16) W A 1 x Ap + W A 2 (cid:80) q ̸ = p p qp ˜ µ Aq (cid:17) where x Ap =
d RT ip (distance from robot i to task p )when task p is assigned to robot i , x Ap = 0 when task p is not assigned.
The action node embeddings{ ˜ µ
Ap |
p ∈ T k } , computed via iteration simultaneously by employing random structure2vec , providesufficient information about the relative locations between robots and their assigned tasks.
Value Embedding.
The second random structure2vec layer embeds the task ages into the tasknodes.
The value embedding ˜ µ V p for task node p is defined as the fixed point for the equation˜ µ Vp = σ (cid:16) W V
1 x Vp + W V 2 (cid:80) q ̸ = p p qp ˜ µ
Vq (cid:17) where x Vp =
(˜ µ Ap , d T p ) is the concatenation of the actionembedding ˜ µ
Ap computed by the first layer and the task age d T p for node p .
The resulting value nodeembeddings { ˜ µ Vp | p ∈ T k } , computed in an iterative manner, provide sufficient information abouthow much value is likely in the local graph around each task by the specified joint assignment.
Computing Q
θ ( s k , a k ) .
To derive Q θ ( s k , a k ) , we aggregate the embedding vectors for all nodes by˜ µ V = (cid:80) p ˜ µ
Vp to obtain one global vector ˜ µ V to embed the value affinity of the global graph.
Wethen use a neural network to map ˜ µ V into Q θ ( s k , a k ) .
The overall pseudo-code for estimating stepsis provided in Appendix G.
It is essential for the proposed inference method can estimate the state action value Q θ ( s k , a k )for varying graph size g k .
Let us provide the intuition related to scuh problem-size transferability.
For Action Embedding , transferability is trivial; the inference problem is a scale-free task locallyaround each node .
For Value Embedding , consider the ratio of robots to tasks.
The overall valueaffinity embedding will be underestimated if this ratio in the training environment is smaller thanthis ratio in the testing environment; overestimated overall otherwise.
The intuition is that thisover/under-estimation does not matter in Q-function based policies as discussed in van Hasselt et al.(2015) as long as the order of Q-function value among actions are the same.
That is, as long asthe best assignments chosen are the same, i.e., arg max a k Q ( s k , a k )
=
arg max a k Q θ ( s k , a k ) , themagnitude of imprecision | Q ( s k , a k ) − Q
θ ( s k , a k ) | does not matter.
We call this property of anestimator order-transferability (with respect to the max operation).
4.5 Selecting
a joint assignment using OTAP
With the previously introduced way to estimate Q θ ( s k , a k ) , we illustrate how to compute the jointassignment (action) a ∗ k , a maximal bipartite matching in the bipartite graph ( R , T k , E RT k ) , giventhe state s k
=
( g k , D k ) .
Specifically, we propose the order transferability-enabled auction policy(OTAP) that constructs a joint assignment a k through N = max ( |R| , | k | ) iterations of Bidding and
Consensus phases.
Each iteration adds one robot-task assignment to construct a full joint assignment.
Bidding-phase.
In the n th iteration of the bidding phase, given M ( n − 1) θ , the ordered set of n − 1robot-task edges in E RTk determined by the previous n − 1 iterations, all the unassigned robots bidtheir the most preferable task to conduct.
Respecting robot-task assignments determined in previousn − 1 iterations and ignoring other unassigned robots, each unassigned robot i select the best taskassignment ϵ RT il that maximizes Q nθ ( s k , M ( n − 1) θ ∪ { ϵ RT ip } ) among all unassigned tasks p ∈ T k ,where Q nθ is the θ -parameterized network with superscript n indicating that the state action value isestimated at n th iteration.
Then, robot i bids { ϵ RT iℓ , Q nθ ( s k , M ( n − 1) θ ∪ { ϵ RT iℓ } ) } to the centralizedauctioneer.
This bidding occurs simultaneously by all the unassigned robots at the n th iteration.
Since the number of ignored robots varies at each iteration, transferability of Q-function inference iscrucial.
Consensus-phase.
In the consensus phase of n th iteration, the centralized auctioneer finds the bidwith the best bid value, say { ϵ RT
i ∗ p ∗ , Q nθ ( s k , M ( n − 1) θ ∪{ ϵ RT i ∗ p ∗ } ) } (Here i ∗ and p ∗ denote the best robottask pair.)
Denote ϵ RT i ∗ p ∗ :
= m ( n ) θ .
The auctioneer sets everyone’s M ( n ) θ as M ( n ) θ =
M ( n − 1) θ ∪ m ( n ) θand initiate bidding phase for the remaining unassigned robots.
These two phases iterate until reaching M ( N ) θ = { m (1) θ , . . .
, m ( N ) θ } .
This M ( N ) θ is chosen as thejoint assignment a ∗ k of N -robots at time step k .
That is, π θ ( s k ) = a ∗ k .
The computational complexityfor computing π θ is O
( | R | | T | ) and is only polynomial; see Appendix L.
4.6 Training Q-function using AFQI
The fitted Q-iteration (FQI) finds θ that minimizes E ( s k ,a k ,r k ,s k +1 ) ∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) +γ max a Q θ ( s k
+1 , a )]] where D denotes the distribution of training data.
We propose a new reinforcement learning method, which we call Auction-fitted Q-iteration (AFQI) , which replaces max a
Q θ ( s k , a ) used in the conventional FQI with OTAP.
That is, writing OTAP as π Q θ , AFQI finds θ thatempirically minimizes E ( s k ,a k ,r k ,s k +1 ) ∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) + γQ θ ( s k +1 , π Q θ ( s k +1 ))]] .
Theoretical analysis
We show the proposed AFQI obtains at least 1 − 1 /e optimality and enables computation of the jointassignment in polynomial time.
This result is achieved by the order-transferability of the proposed
Q -function estimator and its use in selecting the joint assignment.
5.1 Performance bound of OTAP
Recall that Q n denotes the n -robot problem’s true Q -function.
In the same way as we defined M ( N ) θabove, denote the joint assignment chosen by OTAP as { Q n } Nn =1 as M ( N ) = { m (1) , . . .
, m ( N ) } .
Lemma 2.
If the Q -function approximator has order transferability, then M ( N ) =
M ( N ) θ .
For any decision epoch k , let M denote a set of robot-task pairs (a subset of E RT k ).
For any robot-taskpair m ∈ E RT k , define ∆( m | M ) :
=
Q |M∪{ m }| ( s k , M∪{ m } ) − Q |M|
( s k , M ) as the the marginalvalue (under the true Q-functions) of adding robot-task pair m ∈ E RT k .
Lemma 2 enables us to usethe result discussed in Nemhauser et al.
and achieve the result of Theorem 2.
Theorem 2.
Suppose that the Q -function approximation with the parameter value θ exhibits ordertransferability.
Denote M ( N ) θ as the result of OTAP using { Q nθ } Nn =1 and let M ∗ = argmax a k
Q | a k | ( s k , a k ) .
If ∆( m | M ) ≥ 0 , ∀M ⊂ E RTk , ∀ m ∈ E RT k , and the marginal value of adding onerobot diminishes as the number of robots increases, i.e., ∆( m | M ) ≤ ∆( m | N ) , ∀N ⊂ M ⊂ E RT k ,∀ m ∈
E RTk , then the result of OTAP is at least better than 1 − 1 /e of an optimal assignment.
That is,
Q Nθ ( s k , M ( N ) θ ) ≥ Q |M ∗ | ( s k , M ∗ )(1 − 1 /e ) .
See Appendix H and K for the proofs.
5.2 Performance bound of AFQI
AFQI seeks to find θ that minimizes E ( s k ,a k ,r k ,s k +1 )
∼ D
[ Q θ ( s k , a k ) − [ r ( s k , a k ) +γQ θ ( s k +1 , π Q θ ( s k +1 ))]] .
Here, we use OTAP, denoted as π Q θ , instead of max a Q θ ( s k , a ) which isused in general fitted-Q iteration.
As we have seen in section 5.1, OTAP replaces the max operationwith the auction algorithm with a provable performance bound compared with the max operation.
Lemma 3 allows us to use this performance bound to obtain a performance assurance on AFQIcompared with FQI.
We only write an abbreviated version of the statement for brevity.
The formaldescription of conditions and the proof is provided in Appendix I.
Lemma 3.
Kang & Kumar (2021) Suppose that a 1 − 1 /r approximation algorithm is substituted forthe max operation in FQI.
Then, the corresponding new Fitted Q-iteration’s performance is at least1 − 1 /r optimal.
Corollary 1.
AFQI achieves at least 1 − 1 /e performance compared with the optimal policy.
[Table] [Table]
[Table]
5.3 Experiment settings
[Table]
In the main text, we focus on discrete-time & discrete-state (DTDS)
MRRC problems with deterministic and stochastic task completion times.
For CTCS deterministic problems with real-worlddatasets, see IPMS (Appendix A) and mTSP (Appendix B).
Environment.
Since there is no standard dataset for MRRC problems, we used the complex maze-likeenvironment generator of Neller et al.
(2010) (code provided in Appendix 10).
This complex mazemimics the complex road layout of a city and random traffic, inducing nontrivial task completiontimes.
We randomly generated a new maze for every training and testing experiment with randomlychosen initial task/robot locations.
To generate the task completion times, Dijkstra’s algorithm anddynamic programming were used for deterministic and stochastic environments, respectively.
In the deterministic environment, robots always succeed in their movement.
In the stochasticenvironment, a robot makes its intended move with a certain probability.
(Cells with a dot: successwith 55%, every other direction with 15% each.
Cells without a dot: 70% and 10%, respectively.)
Atask is considered served when a robot reaches it.
We consider two reward rules: linearly decayingrewards f ( age ) =
max { 200 − age, 0 } and nonlinearly decaying rewards f ( age ) =
λ age with λ =0.99, where age is the task age when served.
The initial age of tasks are uniformly distributed.
Baselines.
For deterministic environments with linear rewards, where the corresponding MRRC canbe formulated as a mixed-integer linear program (MILP), we consider the following two baselines:
• Optimal : Gurobi Gurobi Optimization (2019), an off-the-self the optimization solver for MILP,was used to solve the problems with 60-min time limit.
• Ekici et al. :
Ekici & Retharekar (2013), the most up-to-date heuristic for solving MRRC in theOperations Research community, was used the problems.
For stochastic environments or exponential rewards, to our knowledge, there is no literature addressing
MRRC with.
Thus, we construct an indirect baseline:
• Sequential Greedy Algorithm (SGA) : a general-purpose multi-robot task allocation algorithmcalled SGA Han-Lim Choi et al.
The performance measure we used is
ρ = rewards collected by the proposed method reward collected by the baseline
Thus, the value of ρ greater than 100% indicates the the proposed method collects more reward thanthe corresponding baseline algorithm.
Note that ρ against Optimal always lower than 100%.
Note that we cannot provide other reinforcement learning-based heuristics as additional baselinessince, to the best of our knowledge (for the class of NP-hard multi-robot/machine scheduling problemswith decaying rewards), this paper is the first to propose a reinforcement learning-based heuristic.
5.4 Performance test.
We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.
See Table .
Ourmethod achieves near-optimality for linear/deterministic rewards with 3% fewer rewards than optimalon average.
The standard deviation for ρ is provided in parentheses.
For others, we see that the%SGA ratio for linear/deterministic is well maintained in stochastic or nonlinear environments.
Dueto dataset generation’s dynamic programming computation complexity, we only consider 8 robots/50tasks at maximum.
We consider larger size problems in IPMS experiments discussed in Appendix A.
Transferability test
Table 2 provides comprehensive transferability test results.
The rows indicate training conditions,while the columns indicate testing conditions.
The results in the diagonal cells in red (cells withthe same training size and testing size) serve as baselines (direct testing).
The results in the offdiagonal show the results for the transferability testing and demonstrate how the algorithms trainedwith different problem sizes perform well on test problems (zero-shot transfer).
We can see thatlower-direction transfer tests (trained with larger size problems and tested with smaller size problems)show only a small loss in performance.
For upper-direction transfer tests (trained with smaller sizeproblems and tested with larger size problems), the loss was up to 4 percent.
5.6 Scalability analysis.
For training complexity, we measured the training time required to achieve 93% optimality consideringa deterministic environment with linear rewards.
Table 4 shows that training time may not necessarilyincrease as problem size gets larger, while the performance is fairly maintained.
MRRC can be formulated as a semi-MDP (SMDP) based multi-robot planning problem (e.g., Omidshafiei et al.
This problem’s complexity with R robots and T tasks and maximum H timehorizon is O
( R ! /T !
( R − T )!)
H ) .
In our proposed method, this complexity is addressed by a combination of two complexities: computational complexity and training complexity.
For computationalcomplexity of joint assignment decision at each timestep is O ( | R || T | 3 ) .
See Appendix L for details.
Concluding Remarks
In this paper, we addressed the challenge of developing a near-optimal learning-based methodfor solving NP-hard multi-robot/machine scheduling problems.
We developed a theory of meanfield inference for scheduling problems and a corresponding theoretically justified GNN method toprecisely infer the Q-function.
We addressed the scalability issue of Fitted Q-Iteration methods formulti-robot/machine scheduling problems by providing a polynomial-time algorithm with a provableperformance guarantee.
Simulation results demonstrate the effectiveness of the our methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
crowdsourcing
or
research
human
subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]