Original: L EARNING SCALABLE AND TRANSFERABLE MULTI - ROBOT / MACHINESEQUENTIAL ASSIGNMENT PLANNING VIA GRAPH EMBEDDING
Revised: L EARNING SCALABLE AND TRANSFERABLE MULTI - ROBOT / MACHINE PLANNING VIA GRAPH EMBEDDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Can the success  of reinforcement learning methods for simple combinatorial op timization problems be extended to multi -robot sequential assignment planning
Revised: There has recently been some success in the use of reinforcement learning methods for single robot combinatorial optimization problems.

Original: In addition to the challenge of achieving near-optimal performance in large problems, transferability to an unseen number of robots and tasks is another key challenge for real-world applications. In this paper, we suggest a method that achieves the ﬁrst success in both challenges for robot/machine scheduling problems.
Revised: In this paper, we develop the ﬁrst learning-based method for multi robot/machine planning problems with combinatorial nature. One real-world concern is the capability to achieve transferability to an unseen number of robots and tasks. The method developed here, for the ﬁrst time, enables such transferability.

Original: Our method comprises of three components.
Revised: Our method is comprised of three components.

Original: First , we show a robot schedul ing problemcan be expressed as a random probabilistic graphical model ( PGM)
Revised: First, we illustrate how to represent a robot planning problem as an extension of probabilistic graphical models (PGMs) which we refer to as random PGMs.

Original: We develop a mean-ﬁeld inference method for  random PGM and use it for Qfunction inference.
Revised: We develop a mean-ﬁeld inference method for such random PGMs and use it for Q -function inference.

Original: Second, we show that transferability can be achieved by carefullydesigning two-step sequential encoding of problem state.
Revised: Second, we show that transferability can be achieved by carefully encoding the problem state via a two-step sequential process.

Original: Third, we resolve the computational scalability issue of ﬁtted Q -iteration by suggesting a heuristic auction-based Q-iteration ﬁtting method enabled by  transferabilitywe achieved.
Revised: Third, we resolve the computational scalability issue of ﬁtted Q -iteration. This is achieved by employing an auction-based heuristic as a substitute for the max operation in the Bellman equation. The auction is enabled by the transferability achieved.

Original: We apply our method  to discrete-time, discrete space problems (Multi-Robot Reward Collection (MRRC
Revised: Our method is applicable to discrete-time, discrete space problems such as multirobot reward collection (MRRC).

Original: ) and scalably achieve 97% optimality with transferability.
Revised: For such problems, with determinsitic assumptoins, we scalably achieve 97% optimality with transferability.

Original: This optimality is maintained under stochastic contexts.
Revised: This optimality is maintained under stochastic contexts.

Original: By extending our method to  continuous time, continuous space formulation, we claim to be the ﬁrst learning-based method with scalable performance among multi-machine scheduling problems; our method scalability achieves comparable performance to popular metaheuristics in Identical parallel machine scheduling (IPMS) problems.
Revised: By extending our method to a continuous time, continuous space formulation, the approach is the ﬁrst learning-based method for machine scheduling problems. Our method scalably achieves comparable performance to popular metaheuristics when applied to identical parallel machine scheduling (IPMS) problems in a deterministic context.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imagine we are given a set of robots and want to service a set of spatially distributed tasks.
Revised: Suppose that we are given a set of robots and seek to serve a set of spatially distributed tasks.

Original: A reward is given when serving each task fast (decaying reward collection), or when ﬁnishing the whole task fast (makespan minimization).
Revised: A reward is given for serving each task promptly - resulting in a time-decaying reward collection problem - or when completing the entire set of tasks - resulting in a makespan minimization problem.

Original: As physical control and routing of individual robots are becoming highly capable [Li (2017)], efﬁcient orchestration of robots arises as the important remaining issue.
Revised: As the capability to control and route individual robots has increased [Li (2017)], efﬁcient orchestration of robots arises as an important remaining concern for such problems.

Original: Recent advances in probabilistic inference [Agha-mohammadi et al.
Revised: Multi-robot planning problems.

Original: (2014)] allows us to infer the probabilistic distribution of time required for controlling and routing a robot to complete a task.
Revised: In this paper, we focus on orchestration problems that can be formulated as robot planning problems. A key assumption in such orchestration problems is that we are given information on the “duration of time required for an assigned robot to complete a task”. This duration may be deterministic (e.g. as in a Traveling Salesman Problem (TSP) or Vehicle Routing Problem (VRP)) or random with given probability distribution (c.f., [Omidshaﬁei et al.

Original: Wewill callthe random variable with this probability distribution task completion time .
Revised: We call this duration the task completion time .

Original: Using task completion time information, this orchestration problem can be formulated as a stochastic scheduling planning problem with sequential decision making [Omidshaﬁei et al.
Revised: Due to their combinatorial nature, robot planning problems suffer from exponential computational complexity. Even in the context of single-robot scheduling problems (e.g., TSP) scalability is a concern.

Original: As pointed in [Omidshaﬁei et al. (2017)], one of the most important bottleneck in this problem is inscalability; as the number of robots and tasks increases, existing methods suffer from either exponentially increasing computation time or signiﬁcant performance loss [Rossi et al. 
Revised: Planning for multiple robots exacerbates the scalability issue. While scalable heuristic methods have been developed for various deterministic multi-robot planning problems (c.f., [Rossi et al. (2018)]), no heuristic methods have been developed for stochastic multi-robot planning problems that are simultaneously both near optimal and scalable.

Original:  Recently, there has been some seminar works in learning-based  algorithms that scalably solve traveling salesman problem (TSP) [Bello et al.
Revised: Learning-based planning methods. Recently, seminal learning-based planning algorithms have been developed for the scalable solution of TSPs [Bello et al.

Original: ; Dai et al.
Revised: ; Dai et al.

Original: (2017); Kool et al.
Revised: (2017); Kool et al.

Original: They formulate TSP as single-robot sequential assignment decision-making problems minimizing traveling cost.
Revised: They showed that learning methods can exploit the recurring structure of TSP and thus can generate near-optimal solution in a very fast computation time.

Original: However, there has not yet been reported success in multi-robot scheduling problem except for special cases when the problem can be extended as a variant of single-robot traveling cost minimization method with multiple routing of a single robot [Nazari et al.
Revised: However, those successes were restricted to single-robot problems except for special cases when the problem can be modeled as a variant of single-robot TSP via multiple successive journeys of a single robot [Nazari et al.

Original: Near-optimal performance with scalability.
Revised: Near-optimal planning with scalability.

Original: In this paper, we introduce a general type of problem called Multi-Robot Reward Collection (MRRC)problems and show that this type of problems can be scalably solved by our method with near-optimal performance.
Revised: In this paper, we address a general type of problem called Multi-Robot Reward Collection (MRRC) and show that this problem can be scalably solved by our method with near-optimal performance.

Original: While learning-based methods are generally believed to suffer exponentially increasing training requirements as problem size (number of robots and tasks) increases, our method’s training requirement is empirically shown not to scale  while maintaining near-optimal performance.
Revised: Generally, the training requirements for learningbased methods increase exponentially with the problem size (number of robots and tasks); c.f., [Li (2017)]. We empirically demonstrate that the training requirements of proposed method scale well while maintaining near-optimal performance.

Original: While MRRC was simulated under discrete state/time space in this paper, this success extends to problems with continuous state/time space; our method claims to be the ﬁrst learning-based method with scalable performance among machine scheduling problems; our method achieves scalability (e.g. 10 machines, 100 tasks) with comparable performance to popular heuristics in Identical Parallel Machine Scheduling (IPMS).
Revised: While we assumed discrete-state discrete-time conditions for MRRC, the method extends to continuous-state continuous-time problems. Our method is also the ﬁrst learning-based method with scalable performance that can address machine scheduling problems. The approach achieves scalability (e.g. 10 machines, 100 tasks) with comparable performance to popular heuristics when applied to Identical Parallel Machine Scheduling (IPMS).

Original: Transferability.
Revised: Transferability.

Original: In addition to scalability, we claim that our method possesses transferability. Transferability means that a trained policy can be applied to new environments with a small loss of performance [Wang et al. 
Revised: The proposed method possesses transferability in that a trained policy can be applied to new environments with a small loss of performance [Wang et al. Such capability is important for real-world applications.

Original: In real-world applications, engineers aren’t given a whole real system for training, but rather given a small testbedenvironment for training; even if the whole system is given for training,  system design orthe number of robots would be frequently changed during operation.
Revised: Engineers are rarely able to train a heuristic on a largescale system during operation, but rather use a small-scale testbed for training. Even if the whole system is available for training, the system design or number of robots may frequently change during operation.

Original: We show that our method achieves transferability with almost no performance loss.
Revised: We show that our method achieves transferability with small performance loss.

Original: Our method stems from the work of [Dai et al.
Revised: Proposed methods. In the seminal paper [Dai et al.

Original: In this seminar paper, they observe that combinatorial optimization problems such as TSP can be formulated asa sequential decision making problem: at each decision step, they grow a partial solution s by choosing increment a with minimum future cost estimate  Q ( s, a )  . 
Revised: , the authors observed that combinatorial optimization problems such as TSP can be formulated as sequential decision making problems. Decision making in such a sequential framework relies on an estimate of future costs Q ( s, a ) for an existing task sequence s and candidate next task a . With this estimate, given the prior decisions s at each decision step, they select the next task a to minimize the future cost estimate.

Original: Under this observation, [Dai et al.
Revised: [Dai et al.

Original: (2017)]’s solution framework can be described by the following three aspects.
Revised: (2017)]’s solution framework relies on the following three assumptions.

Original: For each combinatorial optimization problem, one can carefully ﬁnd a heuristic way to induce a graph representation of ( s, a ) (in case of
Revised: For each combinatorial optimization problem, one can heuristically choose how to induce a graph representation of ( s, a ) .

Original:  TSP, the paper induces a fully connected graph for every possible next increment).
Revised: In the case of TSP, the paper induces a fully connected graph for every possible next task.

Original: This induced graph representation can be considered as a probabilistic graphical model (PGM)
Revised: This induced graph representation can be considered as a probabilistic graphical model (PGM)

Original: [Koller &
Revised: [Koller &

Original: Friedman (2009)]).
Revised: Friedman (2009)].

Original: This model can be used for a graph -based mean-ﬁeld inference method called structure2vec
Revised: This PGM can be used with a graph-based mean-ﬁeld inference method called structure2vec

Original: [Dai et al.
Revised: [Dai et al.

Original: (2016)] to infer Q ( s, a ) and solve a combinatorial optimization problem.
Revised: (2016)] to infer Q ( s, a ) for use in combinatorial optimization problems.

Original: 3) Inference of Q ( s, a ) can be learned by  reinforcement framework called ﬁtted Q-iteration.
Revised: Inference of Q ( s, a ) can be learned by the reinforcement framework called ﬁtted Q-iteration.

Original: Our method shares the overall solution frameworkwith [Dai et al. but designed speciﬁcally to achieve scalability and transferability in multi-robot sequential assignment decision making problems. To be speciﬁc our method differs from [Dai et al.
Revised: We create a solution framework to achieve scalability and transferability for multi-robot planning that builds in numerous directions upon the foundation of [Dai et al.

Original: (2017)] in the following aspects:
Revised: (2017)] as follows:

Original: State representation andtheory of mean-ﬁeld inference with random PGM.
Revised: State representation and mean-ﬁeld inference theory for random PGM.

Original: Instead of heuristically inducing a PGM, weanalytically show that a robot scheduling problem exactly induces atype of random PGM.
Revised: Instead of heuristically inducing a PGM, we show that a robot scheduling problem exactly induces a random PGM.

Original: Since there exists no mean-ﬁeld inference theory for random PGM, we develop the theory and corresponding new structure2vec equation with simple extra inference task.
Revised: Since there exists no mean-ﬁeld inference theory for random PGM we develop the theory and corresponding new structure2vec iteration.

Original: Sequential encoding of information for transferability.
Revised: Sequential encoding of information for transferability.

Original: To achieve transferability in terms of the number of robots and tasks, we carefully design a two-step hierarchical mean-ﬁeld inference [Ranganath et al.
Revised: To achieve transferability in terms of the number of robots and tasks, we carefully design a two-step hierarchical mean-ﬁeld inference [Ranganath et al.

Original: Eachinference step is designed to infer certain information: the ﬁrst step is designed to infer ‘each task’s relative graphical distance fromrobots’; the  second step is designed to infer Q ( s, a ) ( a here means a joint assignment of robots).
Revised: Each step is designed to infer certain information. The ﬁrst step is designed to infer each task’s relative graphical distance from the robots . The second step is designed to infer Q ( s, a ) ( a here refers to a joint assignment of robots).

Original: While the ﬁrst step is by its nature transferable to any number of tasks and robots, the transferability in inference of the second step is achieved by  scale-free characteristic of ﬁtted Q-iteration [van Hasselt et al.
Revised: While the ﬁrst step is by its nature transferable to any number of tasks and robots, the transferability in inference of the second step is achieved by the scale-free characteristic of ﬁtted Q-iteration [van Hasselt et al.

Original: ; relative magnitudes of Q ( s, a ) values instead of their actual magnitudes is important in selecting action a .3.
Revised: That is, the relative magnitudes of Q ( s, a ) values are sufﬁcient to select an action a .

Original: Auction-based assignment.
Revised: Auction-based assignment.

Original: Even if we can infer Q ( s, a ) precisely, the computation time for ﬁnding action a with maximum Q ( s, a )  exponentially increases as robots and task increases.
Revised: Even if we can infer Q ( s, a ) precisely, the computation time required to select an action a using the maximum Q ( s, a ) operation exponentially increases as robots and tasks increase.

Original: To solve this issue, we suggest a heuristic auction that is enabled by  transferability of  Q ( s, a ) inference.
Revised: To resolve this issue, we suggest a heuristic auction that is enabled by the transferability of our Q ( s, a ) inference.

Original: While this heuristic auction selects a with  polynomial computation complexity, it also gives a surprisingly good choice of a  (In fact this heuristic auction increases the performance empirically).
Revised: Even though this heuristic auction selects a with only polynomial computational complexity, it provides surprisingly good choices for a . (In fact, this heuristic auction increases the performance empirically relative to using the max operation.)

Original: Auction-ﬁtted Q-iteration.
Revised: Auction-ﬁtted Q-iteration.

Original: The heuristic auction-based action selection can be incorporated into learning (ﬁtting) Q ( s, a ) with approximation.
Revised: The heuristic auction-based action selection can be incorporated into learning (ﬁtting) Q ( s, a ) .

Original: To be speciﬁc, we use the auction-based action selection scheme, instead of  typical max-operator based action selection, in  Bellman equation during ﬁtted Q-iteration.
Revised: To be speciﬁc, we use the auction-based action selection scheme, instead of the typical max-operator based action selection, in the Bellman equation during ﬁtted Q-iteration.

Original: We call this new learning framework as auction-ﬁtted Q-iteration .
Revised: We call this new learning framework as auction-ﬁtted Q-iteration .

Original:  M ULTI - ROBOT / MACHINE SCHEDULING PROBLEM FORMULATION
Revised:  M ULTI - ROBOT / MACHINE SCHEDULING PROBLEM FORMULATION

Original: Throughout this study, we formulate problems as centralized sequential assignment planning problem: we assume an assignment decision maker who instantaneously achieves necessary information to infer the distribution ofall possible task completion times (this indicates perfect communication).
Revised: We consider centralized sequential assignment planning problems. We assume perfect communication, that is, the decision maker can always determine the distribution of task completion times.

Original: 2. M ULTI -R OBOT R EWARD C OLLECTION (MRRC)
Revised: 2. M ULTI -R OBOT R EWARD C OLLECTION (MRRC)

Original: Here we formulate MRRC in a discrete-state, discrete-time sequential assignment decision making problem (In appendix A, we discuss how MRRC can be formulated with continuous-state and continuous-time space and can be solved by using our method).
Revised: While we consider extended versions of the problem in Appendix A 2 , here we formulate an MRRC as a discrete-time, discrete-state planning problem.

Original: The initial location and ending location of robots and tasks are arbitrary on a grid (e.g., grid world).
Revised: The initial location and ending location of robots and tasks are arbitrary on a grid (e.g., grid world).

Original: We assume that  task is serviced immediately after a robot arrives (In appendix A, we extend how our method can be easily extended to address setup time and processing time).
Revised: We assume that a task is served immediately after a robot arrives.

Original: Under these assumptions, at each time-step, we can assign every robot to every remaining task.
Revised: Under these assumptions, at each time-step, we can assign every robot to every remaining task.

Original: This  problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:
Revised: This MRRC problem can be cast as a Markov Decision Process (MDP) whose state, action, and reward are deﬁned as follows.

Original: State.
Revised: State.

Original: State s t at time step t is a directed graph G t =
Revised: The state s t at time step t is a directed graph G t =

Original: ( R t ∪ T t , E t ) : R t is the set of all robots and T t is the set of all tasks;
Revised: ( R t ∪ T t , E t ) . R t is the set of available robots at time step t . T t is the set of all remaining unserved tasks at time step t .

Original: The set of directed edges E t =
Revised: The set of directed edges E t =

Original: E RTt ∪ E TTt where a directed edge (cid:15) r i t j ∈ E RTt is a random variable  which denotes  task completion time of robot i in R t  to service task j in T t and a directed edge (cid:15) t i t j ∈ E TTt denotes a task completion time of a robot which just ﬁnished serving task i in T t  to service task j in T t  .
Revised: E RTt ∪ E TTt . A directed edge (cid:15) r i t j ∈ E RTt has a random variable as its weight which denotes the task completion time for robot i in R t if it is assigned at time step t to serve task j in T t (note this time subsumes the information about the robot’s present location). A directed edge (cid:15) t i t j ∈ E TTt has as its weight the task completion time for a robot which just ﬁnished serving task i in T t (and is therefore located where task i resided prior to its completion) to serve task j in T t if it is assigned at time step t .

Original: Action.
Revised: Action.

Original: The action a t at time step t is the joint assignment of robots given the current state s t =
Revised: The action a t at time step t is the joint assignment of robots given the current state s t =

Original: G t .
Revised: G t .

Original: The feasible action should satisfythe two constraints: No two robots can be assigned to  a task; some robots may not be assigned when  number of robots are more than remaining tasks.
Revised: Feasible actions should satisfy two constraints: (i) no two robots can be assigned to the same task and (ii) a robot may not be assigned when the number of robots exceeds the number of remaining tasks 3 .

Original: To best address those restrictions, we deﬁne an action a t at time t as a maximal bipartite matching in  bipartite subgraph (( R t , T t) , E RTt )of graph G t .
Revised: To articulate an action, note ﬁrst that the two set of nodes R t and T t are disjoint. As such, the sub-graph ( R t ∪ T t , E RTt ) of graph G t is bipartite. We thus deﬁne an action a t at time t as a maximal bipartite matching in the bipartite sub-graph ( R t ∪ T t , E RTt ) .

Original: For example, robot i in R t is matched with task j in T t in an action a t if we assign robot i to task j at time-step t .
Revised: For example, robot i in R t is matched with task j in T t in an action a t if we assign robot i to task j at time step t .

Original: We denote the set of all possible actions at time-step t as A t .
Revised: We denote the set of all possible actions at time step t as A t .

Original: Reward.
Revised: Reward.

Original: In MRRC, Each task has an arbitrarily determined initial age.
Revised: In MRRC, each task has an arbitrarily determined initial age.

Original: At each time-step, the age of each task increases by one.
Revised: At each time-step, the age of each task increases by one.

Original: When a task is serviced, a reward is determined  only by its age when serviced.
Revised: When a task is served, a reward is determined based only on its age at the time of service.

Original: Denote this reward rule as R ( t ) .

Original: One can easily see that whether a task is served at timestep t is completely determined by s t , a t and s t +1  .
Revised: Note that the state and assignment information s t , a t and s t +1 is sufﬁcient to determine the reward at time step t + 1 .

Original: Therefore, wecan denote the reward[-we get with s t , a t and s t +
-] as R ( s t , a t , s t +1 ) .
Revised: As such we denote the reward[-we get with s t , a t and s t +
-] as R ( s t , a t , s t +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s t to an action a t .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s t to an action a t .

Original: Given  s 0 initial state, an MRRC problem can be expressed as a problem ofﬁnding anoptimal assignment policy φ ∗ such that
Revised: Given an initial state s 0 , the MRRC seeks to maximize the sum of expected rewards through time by the selection of an assignment policy φ ∗ satisfying

Original: [Equation]
Revised: [Equation]

Original: 2. I DENTICAL PARALLEL MACHINE SCHEDULING (IPMS) MAKE - SPAN MINIMIZATION
Revised: 2. I DENTICAL PARALLEL MACHINE SCHEDULING (IPMS) MAKE - SPAN MINIMIZATION

Original: IPMS is a  problem deﬁned in continuous state/continuous time space.
Revised: IPMS is a continuous-time continuous-state problem consisting of diverse tasks which must be served by identical machines.

Revised: Once service of a task i begins, it requires a deterministic duration of time τ i to complete - we call this the processsing time.

Original: Machines are all identical, but processing times of tasks are all different.
Revised: This time is the same independent of which machine serves the task.

Original: In this paper, we discuss IPMS with ‘sequence-dependent setup time’.
Revised: We incorporate one popular extension and allow ‘sequence-dependent setup times’.

Original: In this case, a machine’s setup time required for servicing a  task i is determined by its previously served task j .
Revised: In this case, a machine must conduct a setup prior to serving each task. The duration of this setup depends on the current task i and the task j that was previously served on that machine - we call this the setup time.

Original: In this case, the task completion time  is  the sum of  setup time and processing time.
Revised: The completion time for each task is thus the sum of the setup time and processing time.

Original: Under this setting, we solve  IPMS problem for make-span minimization objective discussed in [Kurz et al.
Revised: Under this setting, we solve the IPMS problem for make-span minimization as discussed in [Kurz et al.

Original: (2001)]; That is, minimizing the total time spent from  start  to end to ﬁnish all tasks.
Revised: That is, we seek to minimize the total time spent from the start time to the completion of the last task.

Original:  IPMS formulation resembles  MRRC formulation in continuous time space with continuous state space (Appendix B) and we relegate the detailed formulation to the Appendix.
Revised: The IPMS formulation resembles our MRRC formulation in continuous-time and continuous-space and we relegate the detailed formulation to Appendix B.

Original:  R OBOT ASSIGNMENT AT EACH TIME - STEP
Revised:  C HOICE OF ASSIGNMENT AT EACH TIME - STEP

Revised: In Section 2, we formulated multi-robot/machine planning problems as sequential joint assignment decision problems.

Original: As in [Dai et al.
Revised: As in [Dai et al.

Original: , we employ Q -function based action choice; at state s t , we  choose action a t using Q ( s t , a t ) inference. However, even if we can infer Q ( s t , a t ) well, Q -function based action choice suffers from exponentially increasing computational complexity with the number of agents.
Revised: , we will select a joint assignment using a Q-function based policy. Since we thus choose action a t with the largest inferred Q ( s t , a t ) value in state s t , the development of a Q ( s t , a t ) inference method is a key issue. Toward this end and motivated by these robot planning problems, we provide new results in random PGM-based mean-ﬁeld inference methods and a subsequent extension of the graph-neural network based inference method called structure2vec

Original: This section introduces 1) how we can express ( s t , a t ) as a graph representation 2) how we can make inference about Q ( s t , a t ) in a transferable way using the sequential graph embedding 3) how we can select a joint assignment of robots with polynomial computation complexity by using the heuristic auction and  the transferable inference on Q ( s t , a t ) .
Revised: [Dai et al. in Section 3.1. In Section 3.2, we discuss how a careful encoding of information using the extended structure2vec of Section 3.1 enables precise and transferable Q ( s t , a t ) inference. Since the computational complexity required to identify the best joint assignment is exponential with respect to the number of robots and tasks, Section 3.3 discusses how the transferability of our Q ( s t , a t ) inference method enables a good action choice heuristic with polynomial computational complexity.

Original: 3. R  ANDOM PGM- REPRESENTATION AND A NEW THEORY FOR MEAN - FIELD INFERENCE
Revised: 3. R OBOT SCHEDULING AS R ANDOM PGM- BASED MEAN - FIELD INFERENCE

Revised: PGM.

Revised: Given random variables { X k } , suppose that joint distribution of { X k } can be factored as

Revised: P ( X 1 , . . . , X n ) = 1 Z (cid:81) i φ i (cid:0) D i (cid:1) where φ i ( D i ) denotes a marginal distribution or conditional distribution on a set of random variables

Original: This section discusses how our problems can be represented as arandom probabilistic graphical model (PGM) for structure2vec inference method.
Revised: D i . Z is a normalizing constant. Then { X k } is called a probabilistic graphical model (PGM). In a PGM, D i is called a clique and φ i ( D i ) is called a clique potential for D i .

Revised: When we suppress φ i ( D i ) as φ i , D i is referred to as the scope of φ i .

Revised: PGM-based mean-ﬁeld inference.

Revised: One popular use of this PGM information is PGM-based mean-ﬁeld inference.

Revised: In mean-ﬁeld inference, we ﬁnd a surrogate distribution Q ( X 1 , . . .

Revised: , X n ) = (cid:81)

Revised: i Q i ( x i ) that has smallest Kullback-Leibler distance to original joint distribution P ( X 1 , . . .

Revised: , X n ) .

Original: While one can heuristically induce a PGM as [Dai et al.
Revised: We then use this surrogate distribution to solve the original inference problem. [Koller & Friedman (2009)] shows that when we are given PGM information, { Q i ( x i ) } can be analytically computed by a ﬁxed point equation. Despite that this usefulness, in most inference problems it is unrealistic to assume we know or can infer probability distributions of a PGM. This limitation was addressed in [Dai et al.

Original: (2017)] for structure2vec, we illustrate that a robot scheduling problem induces an exact graphical model we call ’random’ PGM.
Revised: (2016)] using a method called structure2vec. Structure2vec. [Dai et al. (2016)] suggests that an inference problem with graph-structured data (e.g. a molecule classiﬁcation problem) can be seen as a particular PGM structure that consists of two types of random variables.

Original: Since there is no mean-ﬁeld inference theory for random 
Revised: One type of random variables { X k } is one that serves as input of inference problem (e.g. X k denotes atomic number of atom k ). Another type of random variables { H k } is latent random variable where H k is a latent random variable related to X k . Existence of probabilistic relationships among { H k } are assumed heuristically from graph structure of data. Then the particular PGM structure they assume is P ( { H k } , { X k } )

Original: PGM, we develop a new theory and corresponding new structure2vec in this section.
Revised: ∝ (cid:81) k ∈V φ ( H k | X k ) (cid:81) k,i ∈V φ ( H k | H i ) , where V denotes the set of vertex indexes. The goal of mean-ﬁeld inference problem is to ﬁnd a surrogate distribution Q k ( h k ) for posterior marginal P ( { h k }|{ x k } ) . However, we can’t compute { Q k ( h k ) } since we are not given φ ( H k | H i ) nor

Revised: φ ( H k | X k ) .

Revised: To overcome this limitation, [Dai et al.

Original: While PGM-based mean-ﬁeld inference methods require prior knowledge of whole probabilistic relationships (called cliques) of PGM, structure2vec requires only  the structure( existence of each possible cliques) of  PGM with learning of neural networks (in ﬁxed point iterationsthat surrogates the ﬁxed-point iterations of mean-ﬁeld inference with PGM).
Revised: (2016)] develops a method called structure2vec that only requires the structure of the PGM for mean-ﬁeld inference. structure2vec embeds the mean-ﬁeld inference procedure, i.e. ﬁxed point iteration on { Q k ( h k ) } , into ﬁxed point iterations of neural networks on vectors { ˜ µ k } . Derivation of such ﬁxed point iterations of neural networks can be found in Dai et al. and can be written as ˜ µ k = σ (cid:16) W 1 x k + W 2 (cid:80) j (cid:54) =

Original: The observation we make is that in robot scheduling problems the structure of PGM itself is random .
Revised: k ˜ µ j (cid:17) where σ denotes Relu function and W denotes parameters of neural networks. Robot scheduling as random PGM-based mean-ﬁeld inference.

Revised: All applications of structure2vec in [Dai et al.

Original: Suppose that we know the future path of all robots from  a given state.
Revised: (2016; 2017)] heuristically decide the structure of PGM of each data point from its graph structure. The key observation we make is that inference problems in robot scheduling exactly induce a ‘random’ PGM structure (to be precise, a ‘random’ Bayesian Network). Given that we start from state s t and action a t , consider a random experiment “sequential decision making using policy φ ”. In this experiment, we can deﬁne an event as ‘How robots serve all the remaining tasks in which sequence’.

Revised: We call one such event a ‘scenario’.

Original: For each task t i ∈ T t , deﬁne a random variable X i as ‘some characteristic of task t
Revised: For each task t i ∈ T t , deﬁne a random variable X i as ‘a characteristic of task t

Original: i ’ (e.g. when task i is served).
Revised: i ’ (e.g. when task i is served).

Original: Given  the future path, the relationship among { X i } is awell-deﬁned PGM (a Bayesian Network - see Appendix C).
Revised: Given a scenario, the relationships among { X i } satisfy as a Bayesian Network. For details, see Appendix C)

Original: However, this future path will be random because of the randomness of task completion times.
Revised: Note that we do not know which scenario will occur from time t and thus do not know which PGM will be realized. Besides, the inference of probability of each scenario is challenging.

Original: We thus deﬁne ‘random’
Revised: Putting aside this problem for a while, we ﬁrst deﬁne a ‘random PGM’ and ‘semi-cliques’. Denote the set of all random variables in the inference problem as X =

Original: PGM: A  probabilistic model of how a PGM is randomly chosen from a set of  possible PGMs with { X i } and set of all possible probabilistic relationships  C (we call them ‘semi-cliques’).
Revised: { X i } . A random PGM is a probabilistic model of how a PGM is randomly chosen from a set of all possible PGMs on X 4 . Next, denote the set of all possible probabilistic relationships on X as C X . We call them ‘semi-cliques’.

Original: In robot scheduling problem, a semi-clique D ij ∈ C  is a conditional dependence
Revised: In robot scheduling problem, a semi-clique D ij ∈ C X is a conditional dependence

Original: X i | X j .
Revised: X i | X j .

Original: The semi-clique D ij presents as an actual clique if and only if the robot which ﬁnishes task t i chooses task t
Revised: The semi-clique D ij presents as an actual clique if and only if the robot which ﬁnishes task t i chooses task t

Original: j as  next task.
Revised: j as the next task.

Original: Note that inference of probabilitymodel for all possible future path of robots with random task completion times is challenging.
Revised: We will now prove that we don’t have to infer the probability of each scenario, i.e. random PGM model itself.

Original: The following theorem for mean-ﬁeld inference with random PGM is an extension of mean-ﬁeld inference with PGM [Koller & Friedman (2009)] and suggests that only a simple inference task is required: inferring the presence probability of each semi-cliques.
Revised: The following theorem for mean-ﬁeld inference with random PGM is an extension of mean-ﬁeld inference with PGM [Koller & Friedman (2009)] and suggests that only a simple inference task is required: inference of the presence probability of each semi-cliques.

Original: Theorem 1.
Revised: Theorem 1.

Original: Mean ﬁeld inferencewith random PGM. Suppose that a  PGM is to be randomly chosen among PGMs with { X k }and semi-cliques C .
Revised: Random PGM based mean ﬁeld inference Suppose we are given a random PGM on X = { X k } .

Original: Denote the presence probability of semiclique D m ∈ C as p m  .
Revised: Also, assume that we know presence probability { p m } for all semi-cliques C X = {D m } .

Original: The latent variable distribution { q k ( x k ) } in mean-ﬁeld inference is locally optimal only if
Revised: The latent variable distribution { Q k ( x k ) } in mean-ﬁeld inference is locally optimal only if

Original: [Equation]
Revised: [Equation]

Original: where Z k is a normalizing constant and φ m is the probabilistic relationship for clique m .
Revised: where Z k is a normalizing constant and φ m is the clique potential for clique m .

Original: From this new theory, we can derive the structure2vec inference method with random PGM.
Revised: From this new result, we can develop the structure2vec inference method for random PGM.

Original: As in [Dai et al.
Revised: As in [Dai et al.

Original: , we restrict our discussion to when every semi-clique is between two random variables.
Revised: , we restrict our discussion to when every semi-clique is between two random variables.

Original: In this case, a semi-clique can be written as D ij with its presence probability p ij .
Revised: In this case, a semi-clique can be written as D ij with its presence probability p ij .

Original: Lemma 1.
Revised: Lemma 1.

Original: Structure2vec for random PGM Suppose we are given a random PGM  with  { X k } and semi-cliquesC
Revised: Structure2vec for random PGM. Suppose we are given a random PGM model with X = { X k } . Also, assume that we know presence probability { p ij } for all semi-cliques

Original:  = {D ij }with { p ij } .
Revised: C X = {D ij } .

Original: The ﬁxed point iteration in Theorem 1 for posterior marginal p ( { H k }|{ x k } ) can be embedded in a nonliner function mapping with embedding vector ˜ µ k (which summarizes the latent variable distribution q k ( x k ) ) and X k as:
Revised: The ﬁxed point iteration in Theorem 1 for posterior marginal P ( { H k }|{ x k } ) can be embedded in a nonlinear function mapping with embedding vector ˜ µ k as

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1 and lemma 1 .
Revised: Proof of Thorem 1 and lemma 1 .

Original: For brevity, proofs are relegated to the Appendix D and E (We use a neural network as a nonlinear function approximator with W 1 and W 2 trainable parameters)
Revised: For brevity, proofs are relegated to the Appendix D and E.

Original: Corollary 1.
Revised: Corollary 1.

Original: For a robot scheduling problem with set of tasks t i ∈ T t , the random PGM representation for structure2vec in lemma 1 is (( T t , E TTt ) , { p ij } ) where { p ij } denotes the probability of a robot choosing task t i after serving t j .
Revised: For a robot scheduling problem with set of tasks t i ∈ T t , the random PGM representation for structure2vec in lemma 1 is (( T t , E TTt ) , { p ij } ) where { p ij } denotes the probability of a robot choosing task t i after serving t j .

Original: { p ij } inference procedure employed in this paper is as follows.
Revised: { p ij } inference procedure employed in this paper is as follows.

Original: Denote ages of task i, j as age i , age j .
Revised: Denote ages of task i, j as age i , age j .

Original: Note that if we generate M samples of (cid:15) ij as { e kij } Mk =1 , then 1 M (cid:80) Mk =1 f ( e kij , age i , age j ) is an unbiased and consistent estimator of E [ f ( (cid:15) ij , age i , age j )] .
Revised: Note that if we generate M samples of (cid:15) ij as { e kij } Mk =1 , then 1 M (cid:80) Mk =1 f ( e kij , age i , age j ) is an unbiased and consistent estimator of E [ f ( (cid:15) ij , age i , age j )] .

Original: For each sample k , for each task i and task j , we form a vector of u kij =
Revised: For each sample k , for each task i and task j , we form a vector of u kij =

Original: ( e kij , age i , age j ) and compute g ij = (cid:80) Mk =1 1 M W 1 ( relu ( W 2 u kij ) .
Revised: ( e kij , age i , age j ) and compute g ij = (cid:80) Mk =1 1 M W 1 ( relu ( W 2 u kij ) .

Original: We obtain { p ij } from { g ij } using softmax.
Revised: We obtain { p ij } from { g ij } using softmax.

Original: Algorithm details are in Appendix F.
Revised: Algorithm details are in Appendix F.

Original: 3. T RANSFERABLE INFERENCE OF Q- FUNCTION USING  STRUCTURE 2 VEC
Revised: 3. I NFERENCE OF Q- FUNCTION USING NEW STRUCTURE 2 VEC

Original: The key to transferable inference for Q ( s t , a t ) is the two-step sequential state embedding with hierarchical variational inference [Ranganath et al.
Revised: In this section, we show how Q ( s t , a t ) can be precisely and transeferably inferred using a two-step structure2vec inference method (For theoretical justiﬁcations on hierarchical variational inference, see Ranganath et al.

Original: We here suppose that we are given a graph representation (( T t , E TTt ) , { p ij } ) .
Revised: We here assume that we are given ( T t , E TTt ) and inferred { p ij } so that

Original: Then we can apply Corollary 1 of section 3.1 to infer Q ( s t , a t ) .
Revised: Corollary 1 can be applied.

Original: For brevity, we illustrate the inference procedure for the special case when task completion time is not a random variable, but a value in R (Appendix G illustrates how we can combine random sampling to inference procedure to deal with task completion times as a random variable).
Revised: For brevity, we illustrate the inference procedure for the special case when task completion time is deterministic (Appendix G illustrates how we can combine random sampling to inference procedure to deal with task completion times as a random variable).

Original: Step 1.
Revised: Step 1.

Original: Distance Embedding.
Revised: Distance Embedding.

Original: Theﬁrst step is designed to infer each task’s ‘relative graphical distance’ from all robots. As illustrated in Dai et al. (2016), the output vectors { ˜ µ 1 k } of structure2vec embeds a local graph structure around that vector node.
Revised: The output vectors { ˜ µ 1 k } of structure2vec embeds a local graph information around that vector node [Dai et al. We here focus on embedding information of robot locations around a task node and thus infer each task’s ‘relative graphical distance’ from robots around it.

Original: As the input of ﬁrst structure2vec ( { x k } in lemma 1), we only use robot assignment information (if t k is an assigned task, we set x k as ‘task completion time of assignment’; if t k is not an assigned task:, we set x k = 0 ).
Revised: As the input of ﬁrst structure2vec ( { x k } in lemma 1), we only use robot assignment information (if t k is an assigned task, we set x k as ‘task completion time of assignment’; if t k is not an assigned task:, we set x k = 0 ).

Original: This procedure is illustrated in Figure ?? .
Revised: This procedure is illustrated in Figure 1.

Original: According to [Dai et al.
Revised: According to [Dai et al.

Original: (2016)], the output vectors { ˜ µ 1 k } of structure2vec will include sufﬁcient information about the relative graphical distance from all robots to each task.
Revised: (2016)], the output vectors { ˜ µ 1 k } of structure2vec will include sufﬁcient information about the relative graphical distance from all robots to each task.

Original: Step 2.
Revised: Step 2.

Original: Value Embedding.
Revised: Value Embedding.

Original: The second step is designed to infer ’How much value is likely in the local graph around each task’.
Revised: The second step is designed to infer ’How much value is likely in the local graph around each task’.

Original: Remind that vectors { ˜ µ 1 k } , output vectors of the ﬁrst step, carries information about the relative graphical distance from all robots to each task.
Revised: Remind that vectors { ˜ µ 1 k } , output vectors of the ﬁrst step, carries information about the relative graphical distance from all robots to each task.

Original: We concatenate ‘age’ of each tasks { age k } to each corresponding vector in { ˜ µ 1 k } and use the resulting graph as an input ( { x k } in lemma 1) of second structure2vec, as illustrated in Figure ?? .
Revised: We concatenate ‘age’ of each tasks { age k } to each corresponding vector in { ˜ µ 1 k } and use the resulting graph as an input ( { x k } in lemma 1) of second structure2vec, as illustrated in Figure 1.

Original: Again, vectors { ˜ µ 2 k } of the output graph of second structure2vec operation embeds a local graph structure around each node.
Revised: Again, vectors { ˜ µ 2 k } of the output graph of second structure2vec operation embeds a local graph structure around each node.

Original: Our intuition is that { ˜ µ 2 k } includes sufﬁcient information about ’How much value is likely in the local graph around each task’.
Revised: Our intuition is that { ˜ µ 2 k } includes sufﬁcient information about ’How much value is likely in the local graph around each task’.

Original: Step 3.
Revised: Step 3.

Original: Computing Q ( s t , a t ) .
Revised: Computing Q ( s t , a t ) .

Original: To infer Q ( s t , a t ) , we aggregate the embedding vectors for all nodes, i.e., ˜ µ 2 = (cid:80) k
Revised: To infer Q ( s t , a t ) , we aggregate the embedding vectors for all nodes, i.e., ˜ µ 2 = (cid:80) k

Original: ˜ µ 2 k to get one vector ˜ µ 2 which embeds the ‘value likeliness’ of the global graph.
Revised: ˜ µ 2 k to get one vector ˜ µ 2 which embeds the ‘value likeliness’ of the global graph.

Original: We then use a layer of neural network to map ˜ µ 2 into Q ( s t , a t ) .
Revised: We then use a layer of neural network to map ˜ µ 2 into Q ( s t , a t ) .

Original: The detailed algorithm of above whole procedure (combined with random task completion times) is illustrated in Appendix G.
Revised: The detailed algorithm of above whole procedure (combined with random task completion times) is illustrated in Appendix G.

Original: Why are each inference steps transferable?
Revised: Why are each inference steps transferable?

Original: For the ﬁrst step, it is trivial; the inference problem is a scale-free task.
Revised: For the ﬁrst step, it is trivial; the inference problem is a scale-free task.

Original: In the second step, the ‘value likeliness’ will be underestimated or overestimated according to the ratio of (number of robots / number of tasks) in a local graph: underestimated if  ratio in training environment is smaller than the ratio in the testing environment; overestimated otherwise.
Revised: In the second step, the ‘value likeliness’ will be underestimated or overestimated according to the ratio of (number of robots / number of tasks) in a local graph: underestimated if the ratio in training environment is smaller than the ratio in the testing environment; overestimated otherwise.

Original: The key idea solving this problem is that this over/under-estimation does not matter in Q-function based action decision [van Hasselt et al.
Revised: The key idea solving this problem is that this over/under-estimation does not matter in Q-function based action decision [van Hasselt et al.

Original: (2015)] as long as the order of Q-function value among actions are the same.
Revised: (2015)] as long as the order of Q-function value among actions are the same.

Original: While analytic justiﬁcation of this order invariance is beyond this paper’s scope, the fact that there is no over/underestimation issue in the ﬁrst step inference problem helps this justiﬁcation.
Revised: While analytic justiﬁcation of this order invariance is beyond this paper’s scope, the fact that there is no over/underestimation issue in the ﬁrst step inference problem helps this justiﬁcation.

Original: 3. A CTION S ELECTION U SING H EURISTIC A UCTION
Revised: 3. A CTION S ELECTION U SING H EURISTIC A UCTION

Original: In Q-function based action choice, at each time-step t , we ﬁnd an action with largest Q ( s t , a t ) .
Revised: In Q-function based action choice, at each time-step t , we ﬁnd an action with largest Q ( s t , a t ) .

Original: We call this action choice operation ’max-operation’.
Revised: We call this action choice operation ’max-operation’.

Original: The problem in max-operation in  multi-robot setting is that the number of computation exponentially increases as the number of robots and tasks increases.
Revised: The problem in max-operation in the multi-robot setting is that the number of computation exponentially increases as the number of robots and tasks increases.

Original: In this section, we show that transferability of Q -function inference enables designing an efﬁcient heuristic auction that replaces max operation.
Revised: In this section, we show that transferability of Q -function inference enables designing an efﬁcient heuristic auction that replaces max operation.

Original: We call it auction-based policy(ADP) and denote it as φ Q θ , where Q θ indicates that we compute φ Q θ using current Q θ estimator.
Revised: We call it auction-based policy(ADP) and denote it as φ Q θ , where Q θ indicates that we compute φ Q θ using current Q θ estimator.

Original: At time-step t , a state s t is a graph G t =
Revised: At time-step t , a state s t is a graph G t =

Original: ( R t ∪T t , E t ) as deﬁned in section 2.1.
Revised: ( R t ∪T t , E t ) as deﬁned in section 2.1.

Original: Our ADP , φ Q θ , ﬁnds an action a t (which is a matching in bipartite graph (( R t , T t ) , E RTt ) of graph G t ) through iterations between two phases: the bidding phase and the consensus phase.
Revised: Our ADP , φ Q θ , ﬁnds an action a t (which is a matching in bipartite graph (( R t ∪T t ) , E RTt ) of graph G t ) through iterations between two phases: the bidding phase and the consensus phase.

Original: We start with a bidding phase.
Revised: We start with a bidding phase.

Original: All robots initially know the matching determined in previous iterations.
Revised: All robots initially know the matching determined in previous iterations.

Original: We denote this matching as Y , a bipartite subgraph of (( R t , T t ) , E RTt ) .
Revised: We denote this matching as Y , a bipartite subgraph of (( R t ∪T t ) , E RTt ) .

Original: When making a bid, a robot r i ignores all other unassigned robots.
Revised: When making a bid, a robot r i ignores all other unassigned robots.

Original: For example, suppose robot r i considers t j for bidding.
Revised: For example, suppose robot r i considers t j for bidding.

Original: For r i , Y ∪ (cid:15) ij is a proper action (according to deﬁnition in section 2.1) in a ‘unassigned robot-ignored’ problem.
Revised: For r i , Y ∪ (cid:15) ij is a proper action (according to deﬁnition in section 2.1) in a ‘unassigned robot-ignored’ problem.

Original: Robot r i thus can compute Q ( s t , Y ∪ (cid:15) r i t j ) of ‘unassigned robot-ignored’ problem for all unassigned task t j .
Revised: Robot r i thus can compute Q ( s t , Y ∪ (cid:15) r i t j ) of ‘unassigned robot-ignored’ problem for all unassigned task t j .

Original: If task t ∗ is with the highest value, robot r i bids { (cid:15) r i t ∗ , Q ( s t , Y ∪ (cid:15) r
Revised: If task t ∗ is with the highest value, robot r i bids { (cid:15) r i t ∗ , Q ( s t , Y ∪ (cid:15) r

Original: i t ∗ ) } to auctioneer.
Revised: i t ∗ ) } to auctioneer.

Original: Since number of robots ignored by r i is different at each iteration, transferability of Q-function inference plays key role.
Revised: Since number of robots ignored by r i is different at each iteration, transferability of Q-function inference plays key role.

Original: The consensus phase is simple.
Revised: The consensus phase is simple.

Original: The auctioneer ﬁnds the bid with the best value, say { (cid:15) ∗ , bid value with (cid:15) ∗ } .
Revised: The auctioneer ﬁnds the bid with the best value, say { (cid:15) ∗ , bid value with (cid:15) ∗ } .

Original: Then auctioneer updates everyone’s
Revised: Then auctioneer updates everyone’s

Original: Y as Y ∪ { (cid:15) ∗ } .
Revised: Y as Y ∪ { (cid:15) ∗ } .

Original: These bidding and consensus phases are iterated until we can’t add an edge to Y anymore.
Revised: These bidding and consensus phases are iterated until we can’t add an edge to Y anymore.

Original: Then the central decision maker chooses Y as φ Q θ ( s k ) .
Revised: Then the central decision maker chooses Y as φ Q θ ( s k ) .

Original: One can easily verify that the computational complexity of computing φ Q θ is O ( | L R | | L T | ) , which is only polynomial.
Revised: One can easily verify that the computational complexity of computing φ Q θ is O ( | L R | | L T | ) , which is only polynomial.

Original: While theoretical performance guarantee of this heuristic auction is out of this paper’s scope, in section 5 we show that empirically this heuristic achieves near-optimal performance.
Revised: While theoretical performance guarantee of this heuristic auction is out of this paper’s scope, in section 5 we show that empirically this heuristic achieves near-optimal performance.

Original:  L EARNING ALGORITHM 4. A UCTION - FITTED Q- ITERATION FRAMEWORK
Revised:  L EARNING ALGORITHM 4. A UCTION - FITTED Q- ITERATION FRAMEWORK

Original: In ﬁtted Q-iteration, we ﬁt θ of Q θ ( s t , a t ) with stored data using Bellman optimality equation.
Revised: In ﬁtted Q-iteration, we ﬁt θ of Q θ ( s t , a t ) with stored data using Bellman optimality equation.

Original: That is, chooses θ that makes E (cid:2) Q
Revised: That is, chooses θ that makes E (cid:2) Q

Original: θ ( s k , a k ) −
Revised: θ ( s k , a k ) −

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: small.
Revised: small.

Original: Note that every update of θ needs at least one max-operation.
Revised: Note that every update of θ needs at least one max-operation.

Original: To solve this issue, we suggest a learning framework we call auction-ﬁtted Q -iteration.
Revised: To solve this issue, we suggest a learning framework we call auction-ﬁtted Q -iteration.

Original: What we do is simple: when we update θ , we use auction-based policy(ADP) deﬁned in section 3.3 instead of max-operation.
Revised: What we do is simple: when we update θ , we use auction-based policy(ADP) deﬁned in section 3.3 instead of max-operation.

Original: That is, we seek the parameter θ that minimizes
Revised: That is, we seek the parameter θ that minimizes

Original: E (cid:2)
Revised: E (cid:2)

Original: [Equation]
Revised: [Equation]

Original: 4. E XPLORATION FOR A UCTION
Revised: 4. E XPLORATION FOR A UCTION

Original: How can we conduct exploration in Auction-ﬁtted Q-iteration framework?
Revised: How can we conduct exploration in Auction-ﬁtted Q-iteration framework?

Original: Unfortunately, we can’t use (cid:15) -greedy method since such randomly altered assignment is very likely to cause a catastrophic result in problems with combinatorial nature.
Revised: Unfortunately, we can’t use (cid:15) -greedy method since such randomly altered assignment is very likely to cause a catastrophic result in problems with combinatorial nature.

Original: In this paper, we suggest that parameter space exploration
Revised: In this paper, we suggest that parameter space exploration

Original: [Plappert et al.
Revised: [Plappert et al.

Original: (2017)] can be applied.
Revised: (2017)] can be applied.

Original: Recall that we use Q θ ( s k , a k ) to get policy
Revised: Recall that we use Q θ ( s k , a k ) to get policy

Original: φ Q θ ( s k ) .
Revised: φ Q θ ( s k ) .

Original: Note that θ denotes all neural network parameters used in the structure2vec iterations introduced in Section 5.
Revised: Note that θ denotes all neural network parameters used in the structure2vec iterations introduced in Section 5.

Original: Since Q θ ( s k , a k ) is parametrized by θ , exploration with φ Q θ ( s k ) can be performed by exploration with parameter θ .
Revised: Since Q θ ( s k , a k ) is parametrized by θ , exploration with φ Q θ ( s k ) can be performed by exploration with parameter θ .

Original: Such exploration in parameter space has been introduced in the policy gradient RL literature.
Revised: Such exploration in parameter space has been introduced in the policy gradient RL literature.

Original: While this method was originally developed for policy gradient based methods, exploration in parameter space can be particularly useful in auction-ﬁtted Q-iteration.
Revised: While this method was originally developed for policy gradient based methods, exploration in parameter space can be particularly useful in auction-ﬁtted Q-iteration.

Original: The detailed application is as follows.
Revised: The detailed application is as follows.

Original: When conducting exploration, apply a random perturbation on the neural network parameters θ in structure2vec.
Revised: When conducting exploration, apply a random perturbation on the neural network parameters θ in structure2vec.

Original: The resulting a perturbation in the Q-function used for decision making via the auction-based policy φ Q θ ( s k ) throughout that problem.
Revised: The resulting a perturbation in the Q-function used for decision making via the auction-based policy φ Q θ ( s k ) throughout that problem.

Original: Similarly, when conducting exploitation, the current surrogate Q-function is used throughout the problem.
Revised: Similarly, when conducting exploitation, the current surrogate Q-function is used throughout the problem.

Original: [Table] Updates for the surrogate Q-function may only occur after each problem is complete (and typically after a group of problems) [Table]
Revised: Updates for the surrogate Q-function may only occur after each problem is complete (and typically after a group of problems).

Original:  E XPERIMENT 5. MRRC
Revised:  E XPERIMENT 5. MRRC

Original: For MRRC, we conduct a simulation experiment for a discrete time, discrete state environment.
Revised: For MRRC, we conduct a simulation experiment for a discrete time, discrete state environment.

Original: We use maze (see Figure ?? ) generator of UC Berkeley CS188 Pacman project [Neller et al.
Revised: We use maze (see Figure 1) generator of UC Berkeley CS188 Pacman project [Neller et al.

Original: (2010)] to generate large size mazes.
Revised: (2010)] to generate large size mazes.

Original: We generated a new maze for every training and testing experiments.
Revised: We generated a new maze for every training and testing experiments.

Original: Under the deterministic environment, the robot succeeds its movement 100%.
Revised: Under the deterministic environment, the robot succeeds its movement 100%.

Original: Under stochastic environment, a robot succeeds its intended movement in 55% on the grid with dots and for every other direction 15% each; on the grid without dots, the rates are 70% and 10%.
Revised: Under stochastic environment, a robot succeeds its intended movement in 55% on the grid with dots and for every other direction 15% each; on the grid without dots, the rates are 70% and 10%.

Original: As described in section 2, routing problems are already solved.
Revised: As described in section 2, routing problems are already solved.

Original: That is, each robot knows how to optimally (in expectation) reach a task.
Revised: That is, each robot knows how to optimally (in expectation) reach a task.

Original: To ﬁnd an optimal routing policy, we use Dijkstra’s algorithm for deterministic environments and dynamic programming for stochastic environments.
Revised: To ﬁnd an optimal routing policy, we use Dijkstra’s algorithm for deterministic environments and dynamic programming for stochastic environments.

Original: The central assignment decision maker has enough samples of task completion time for every possible route.
Revised: The central assignment decision maker has enough samples of task completion time for every possible route.

Original: We consider two reward rules: Linearly decaying rewards obey f ( age ) = 200 − age until reaching 0, where age is the task age when served; For nonlinearly decaying rewards, f ( t ) =
Revised: We consider two reward rules: Linearly decaying rewards obey f ( age ) = 200 − age until reaching 0, where age is the task age when served; For nonlinearly decaying rewards, f ( t ) =

Original: λ t for λ = 0.99.
Revised: λ t for λ = 0.99.

Original: Initial age of tasks were uniformly distributed in the interval [0 , 100] .
Revised: Initial age of tasks were uniformly distributed in the interval [0 , 100] .

Original: Performance test.
Revised: Performance test.

Original: We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/ linear rewards, stochastic/nonlinear rewards.
Revised: We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.

Original: There are two baselines used for performance test: exact baseline and indirect baseline.
Revised: There are three baselines used for performance test: exact baseline, heuristic baseline, and indirect baseline.

Original: For the experiment with deterministic with linearly decaying rewards, an exact optimal solution for mixedinteger linear program exists and can be used as a baseline.
Revised: For the experiment with deterministic with linearly decaying rewards, an exact optimal solution for mixed-integer exists and can be used as a baseline.

Original: We solve this program using Gurobi with 60-min cut to get the baseline.
Revised: We solve this program using Gurobi with 60-min cut to get the baseline.

Revised: We also implemented the most up-to-date heuristic for MRRC in [Ekici & Retharekar (2013)].

Original: For any other experiments with nonlinearly decaying rewards or stochastic environment, such an exact optimal solution or other heuristics methods does not exist.
Revised: For any other experiments with nonlinearly decaying rewards or stochastic environment, such an exact optimal solution or other heuristics methods does not exist.

Original: In these cases, we should be conservative when talking about performance.
Revised: In these cases, we should be conservative when talking about performance.

Original: Our strategy is to construct a indirect baseline using a universally applicable algorithm called Sequential greedy algorithm (SGA) [Han-Lim Choi et al.
Revised: Our strategy is to construct a indirect baseline using a universally applicable algorithm called Sequential greedy algorithm (SGA) [Han-Lim Choi et al.

Original: SGA is a polynomial-time task allocation algorithm that shows decent scalable performance to both linear and non-linear rewards.
Revised: SGA is a polynomial-time task allocation algorithm that shows decent scalable performance to both linear and non-linear rewards.

Original: For stochastic environments, we use mean task completion time for task allocation and re-allocate the whole tasks at every time-steps.
Revised: For stochastic environments, we use mean task completion time for task allocation and re-allocate the whole tasks at every timesteps.

Original: We construct our indirect baseline as ‘ratio between our method and SGA for experiments with deterministic-linearly decaying rewards’.
Revised: We construct our indirect baseline as ‘ratio between our method and SGA for experiments with deterministic-linearly decaying rewards’.

Original: Showing that this ratio is maintained for stochastic environments in both linear/nonlinear rewards sufﬁces our purpose.
Revised: Showing that this ratio is maintained for stochastic environments in both linear/nonlinear rewards sufﬁces our purpose.

Original: Table 1 shows experiment results for (# of robots, # of tasks)
Revised: Table 1 shows experiment results for (# of robots, # of tasks)

Original: =
Revised: =

Original: For linear/deterministic rewards, our proposed method achieves nearoptimality (all above 95% optimality).
Revised: For linear/deterministic rewards, our proposed method achieves nearoptimality (all above 95% optimality).

Original: While there is no exact or comparable performance baseline for experiments under other environments, indirect baseline (%SGA) at least shows that our method does not lose %SGA for stochastic environments compared with %SGA for deterministic environments in both linear and nonlinear rewards.[Table] [Table]
Revised: While there is no exact or comparable performance baseline for experiments under other environments, indirect baseline (%SGA) at least shows that our method does not lose %SGA for stochastic environments compared with %SGA for deterministic environments in both linear and nonlinear rewards.

Original: Scalability test.
Revised: Scalability test.

Original: We count the training requirements for 93% optimality for seven problem sizes (# of robots N R , # of tasks N T ) =
Revised: We count the training requirements for 93% optimality for seven problem sizes (# of robots N R , # of tasks N T ) =

Original: (2 , 20) , (3 , 20) , (5 , 30) , (5 , 40) , (8 , 40) , (8 , 50) with deterministic/linearly decaying rewards (we can compare optimality only in this case).
Revised: (2 , 20) , (3 , 20) , (5 , 30) , (5 , 40) , (8 , 40) , (8 , 50) with deterministic/linearly decaying rewards (we can compare optimality only in this case).

Original: As we can see in Table 2, the training requirement shown not to scale as problem size increases [Table]
Revised: As we can see in Table 2, the training requirement shown not to scale as problem size increases.

Original: Transferability test .
Revised: Transferability test .

Original: Suppose that we trained our learning algorithm with problems of three robots and 30 Tasks.
Revised: Suppose that we trained our learning algorithm with problems of three robots and 30 Tasks.

Original: We can claim transferability of our algorithm if our algorithm achieves similar performance for testing with problems of 8 robots and 50 tasks when compared with the algorithm speciﬁcally trained with problems of 8 robots and 50 tasks, the same size as testing.
Revised: We can claim transferability of our algorithm if our algorithm achieves similar performance for testing with problems of 8 robots and 50 tasks when compared with the algorithm speciﬁcally trained with problems of 8 robots and 50 tasks, the same size as testing.

Original: Table 3 shows the experiment result trained with three robots and 30 tasks and tested with six combinations (# of robots N R , # of tasks N T ) =
Revised: Table 3 shows our comprehensive experiment to test transeferability.

Original: (2 , 20) , (3 , 20) , (5 , 30) , (5 , 40) , (8 , 40) , (8 , 50) with deterministic/linearly decaying rewards (we can compare optimality only in this case). Values in ‘Same size as testing size’ are from table 1 and shown here for comparison purposes.
Revised: The results in the diagonals (where training size and testing size is the same) becomes a baseline, and we can compare how the networks trained with different problem size did well compare to those results. We could see that lower-direction transfer tests (trained with larger size problem and tested with smaller size problems) shows only a [Table]

Original: As we can see in table 3, we achieve transferability within 3% loss (in large problems nearly 0% loss or better).
Revised: [Table] small loss in performance. For upper-direction transfer tests (trained with smaller size problem and tested with larger size problem), performance loss was up 4 percent.

Original: Ablation study.
Revised: Ablation study.

Original: There are three components in our proposed method: 1) a careful encoding of information using two-layers of structure2vec, 2) new structure2vec equation with random PGM and 3) an auction-based assignment.
Revised: There are three components in our proposed method: 1) a careful encoding of information using two-layers of structure2vec, 2) new structure2vec equation with random PGM and 3) an auction-based assignment.

Original: Each component was removed from the full method and tested to check the necessity of the component.
Revised: Each component was removed from the full method and tested to check the necessity of the component.

Original: We test the performance in a deterministic/linearly decaying rewards (so that there is an optimal solution available for comparison).
Revised: We test the performance in a deterministic/linearly decaying rewards (so that there is an optimal solution available for comparison).

Original: The experimental results are shown in Figure ?? .
Revised: The experimental results are shown in Figure 2.

Original: While the full method requires more training steps, only the full method achieves near-optimal performance.
Revised: While the full method requires more training steps, only the full method achieves near-optimal performance.

Original: For IPMS, we test it with continuous time, continuous state environment.
Revised: For IPMS, we test it with continuous time, continuous state environment.

Original: While there have been many learning-based methods proposed for (single) robot scheduling problems, to the best our knowledge our method is the ﬁrst learning method to claim scalable performance among machine- [Table]
Revised: While there have been many learning-based methods proposed for (single) robot scheduling problems, to the best our knowledge our method is the ﬁrst learning method to claim scalable performance among machinescheduling problems.

Original: scheduling problems. Hence, in this case, we focus on showing comparable performance for large problems, instead of attempting to show the superiority of our method compared with heuristics speciﬁcally designed for IPMS (actually no heuristic was speciﬁcally designed to solve our exact problem (makespan minimization, sequence-dependent setup with no restriction on setup times))
Revised: Hence, in this case, we focus on showing comparable performance for large problems, instead of attempting to show the superiority of our method compared with heuristics speciﬁcally designed for IPMS (actually no heuristic was speciﬁcally designed to solve our exact problem (makespan minimization, sequence-dependent setup with no restriction on setup times))

Original: For each task, processing times is determined using uniform [16, 64].
Revised: For each task, processing times is determined using uniform [16, 64].

Original: For every (task i , task j ) ordered pair, a unique setup time is determined using uniform [0, 32].
Revised: For every (task i , task j ) ordered pair, a unique setup time is determined using uniform [0, 32].

Original: As illustrated in section 2, we want to minimize make-span.
Revised: As illustrated in section 2, we want to minimize make-span.

Original: As a benchmark for IPMS, we use Google OR-Tools library 
Revised: As a benchmark for IPMS, we use Google OR-Tools library [Table] 5R [Table]

Revised: 8R

Original: Google (2012).
Revised: [Table] 8R [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Google (2012).

Original: This library provides metaheuristics such as Greedy Descent, Guided Local Search,
Revised: This library provides metaheuristics such as Greedy Descent, Guided Local Search,

Original: Simulated Annealing, Tabu Search.
Revised: Simulated Annealing, Tabu Search.

Original: We compare our algorithm’s result with the heuristic with the best result for each experiment.
Revised: We compare our algorithm’s result with the heuristic with the best result for each experiment.

Original: We consider cases with 3 , 5 , 7 , 10 machines and 50 , 75 , 100 jobs.
Revised: We consider cases with 3 , 5 , 7 , 10 machines and 50 , 75 , 100 jobs.

Original: The results are provided in Table 4.
Revised: The results are provided in Table 4.

Original: Makespan obtained by our method divided by the makespan obtained in the baseline is provided.
Revised: Makespan obtained by our method divided by the makespan obtained in the baseline is provided.

Original: Although our method has limitations in problems with a small number of tasks, it shows comparable performance to a large number of tasks and shows its value as the ﬁrst learning-based machine scheduling method that achieves scalable performance.
Revised: Although our method has limitations in problems with a small number of tasks, it shows comparable performance to a large number of tasks and shows its value as the ﬁrst learning-based machine scheduling method that achieves scalable performance.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We presented a learning-based method that achieves the ﬁrst success for multi-robot/machine scheduling problems in both challenges: scalable performance and tranferability.
Revised: We presented a learning-based method that achieves the ﬁrst success for multi-robot/machine scheduling problems in both challenges: scalable performance and tranferability.

Original: We identiﬁed that robot scheduling problems have an exact representation as random PGM.
Revised: We identiﬁed that robot scheduling problems have an exact representation as random PGM.

Original: We developed a meanﬁeld inference theory for random PGM and extended structure2vec method of Dai et al.
Revised: We developed a meanﬁeld inference theory for random PGM and extended structure2vec method of Dai et al.

Original: To overcome the limitations of ﬁtted Q-iteration, a heuristic auction that was enabled by transferability is suggested.
Revised: To overcome the limitations of ﬁtted Q-iteration, a heuristic auction that was enabled by transferability is suggested.

Original: Through experimental evaluation, we demonstrate our method’s success for
Revised: Through experimental evaluation, we demonstrate our method’s success for

Original: MRRC problems under  deterministic/stochastic environment.
Revised: MRRC problems under a deterministic/stochastic environment.

Original: Our method also claims to be the ﬁrst learning-based algorithm that achieves scalable performance among machine scheduling algorithms; our method achieves a comparable performance in a scalable manner.
Revised: Our method also claims to be the ﬁrst learning-based algorithm that achieves scalable performance among machine scheduling algorithms; our method achieves a comparable performance in a scalable manner.

Original: A MRRC WITH CONTINUOUS STATE / CONTINUOUS TIME SPACE FORMULATION , OR WITH SETUP TIME AND PROCESSING TIME
Revised: Our method for MRRC problems can be easily extended to ride-sharing problems or package delivery problems. Given a set of all user requests to serve, those problems can be formulated as a MRRC problem. For both ride-sharing and package delivery, it is reasonable to assume that the utility of a user depends on when she is completely serviced.

Revised: We can model how the utility of a user decreases over time since when it appears and set the objective function of problems as maximizing total collected user utility.

Revised: Now consider a task ‘deliver user (or package) from A to B’.

Revised: This is actually a task ”Move to location A and then move to location B”.

Revised: If we know the completion time distribution of each move (as we did for MRRC), the task completion time is simply the sum of two random variables corresponding to task completion time distribution of the moves in the task.

Revised: Indeed, ride-sharing or package delivery problems are of such tasks (We can ignore charging moves for simplicity, and also we don’t have to consider simple relocation of vehicles or robots since we don’t consider random customer arrivals).

Revised: Therefore, both ride-sharing problems and package delivery problems can be formulated as MRRC problems.

Revised: A MRRC WITH CONTINUOUS STATE / CONTINUOUS TIME SPACE FORMULATION , OR WITH SETUP TIME AND PROCESSING TIME

Original: In continuous state/continuous time space formulation, the initial location and ending location of robots and tasks are arbitrary on R 2 .
Revised: In continuous state/continuous time space formulation, the initial location and ending location of robots and tasks are arbitrary on R 2 .

Original: At every moment at least a robot ﬁnishes a task, we make assignment decision for a free robot(s).
Revised: At every moment at least a robot ﬁnishes a task, we make assignment decision for a free robot(s).

Original: We call this moments as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .
Revised: We call this moments as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .

Original: , t k , . . . ) .
Revised: , t k , . . . ) .

Original: Abusing this notation slightly, we use ( · ) t k = ( · ) k .
Revised: Abusing this notation slightly, we use ( · ) t k = ( · ) k .

Original: Task completion time can consist of three components: travel time, setup time and processing time.
Revised: Task completion time can consist of three components: travel time, setup time and processing time.

Original: While a robot in the travel phase or setup phase may be reassigned to other tasks, we can’t reassign a robot in the processing phase.
Revised: While a robot in the travel phase or setup phase may be reassigned to other tasks, we can’t reassign a robot in the processing phase.

Original: Under these assumptions, at each decision epoch robot r i is given a set of tasks it can assign itself: if it is in the traveling phase or setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.
Revised: Under these assumptions, at each decision epoch robot r i is given a set of tasks it can assign itself: if it is in the traveling phase or setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.

Original: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:
Revised: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:

Original: State.
Revised: State.

Original: State s k at decision epoch
Revised: State s k at decision epoch

Original: k is a directed graph G k
Revised: k is a directed graph G k

Original: =
Revised: =

Original: ( R k ∪ T k , E k ) :
Revised: ( R k ∪ T k , E k ) :

Original: R k is the set of all robots and T k is the set of all tasks;
Revised: R k is the set of all robots and T k is the set of all tasks;

Original: The set of directed edges E k
Revised: The set of directed edges E k

Original: =
Revised: =

Original: E RTk ∪E TTk where a directed edge (cid:15) r i t
Revised: E RTk ∪E TTk where a directed edge (cid:15) r i t

Original: j ∈ E RTk is a random variable which denotes task completion time of robot i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TT k denotes a task completion time of a robot which just ﬁnished serving task i in T k to service task j in T k .
Revised: j ∈ E RTk is a random variable which denotes task completion time of robot i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a robot which just ﬁnished serving task i in T k to service task j in T k .

Original: E RTk contains information about each robot’s possible assignments: E RTk =
Revised: E RTk contains information about each robot’s possible assignments: E RTk =

Original: ∪ i E r i k , where E r i t is a singleton set if robot
Revised: ∪ i E r i k , where E r i t is a singleton set if robot

Original: i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from robot r
Revised: i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from robot r

Original: i to remaining tasks that are not in the processing phase.
Revised: i to remaining tasks that are not in the processing phase.

Original: Action.
Revised: Action.

Original: The action a k at decision epoch k is the joint assignment of robots given the current state s k = G k .
Revised: The action a k at decision epoch k is the joint assignment of robots given the current state s k = G k .

Original: The feasible action should satisfy the two constraints:
Revised: The feasible action should satisfy the two constraints:

Original: No two robots can be assigned to a task; some robots may not be assigned when number of robots are more than remaining tasks.
Revised: No two robots can be assigned to a task; some robots may not be assigned when number of robots are more than remaining tasks.

Original: To best address those restrictions, we deﬁne an action a k at time t as a maximal bipartite matching in bipartite sub-graph (( R k , T k ) , E RTk ) of graph G k .
Revised: To best address those restrictions, we deﬁne an action a k at time t as a maximal bipartite matching in bipartite sub-graph (( R k ∪ T k ) , E RTk ) of graph G k .

Original: For example, robot i in R k is matched with task j in T k in an action a k if we assign robot i to task j at decision epoch t .
Revised: For example, robot i in R k is matched with task j in T k in an action a k if we assign robot i to task j at decision epoch t .

Original: We denote the set of all possible actions at epoch k as A k .
Revised: We denote the set of all possible actions at epoch k as A k .

Original: Reward.
Revised: Reward.

Original: In MRRC, Each task has an arbitrarily determined initial age.
Revised: In MRRC, Each task has an arbitrarily determined initial age.

Original: At each decision epoch, the age of each task increases by one.
Revised: At each decision epoch, the age of each task increases by one.

Original: When a task is serviced, a reward is determined only by its age when serviced.
Revised: When a task is serviced, a reward is determined only by its age when serviced.

Original: Denote this reward rule as R ( k ) .
Revised: Denote this reward rule as R ( k ) .

Original: One can easily see that whether a task is served at epoch k is completely determined by s k , a k and s k
Revised: One can easily see that whether a task is served at epoch k is completely determined by s k , a k and s k

Original: +1 .
Revised: +1 .

Original: Therefore, we can denote the reward we get with s k , a k and s k
Revised: Therefore, we can denote the reward we get with s k , a k and s k

Original: +

Revised: +

Original: as R ( s k , a k , s k +1 ) .
Revised: as R ( s k , a k , s k +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .

Original: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that
Revised: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that

Original: [Equation]
Revised: [Equation]

Original:  I DENTICAL PARALLEL MACHINE SCHEDULING PROBLEM FORMULATION
Revised: B I DENTICAL PARALLEL MACHINE SCHEDULING PROBLEM FORMULATION

Original: As written in 2.2, IPMS is a problem deﬁned in continuous state/continuous time space.
Revised: As written in 2.2, IPMS is a problem deﬁned in continuous state/continuous time space.

Original: Machines are all identical, but processing times of tasks are all different.
Revised: Machines are all identical, but processing times of tasks are all different.

Original: In this paper, we discuss IPMS with ‘sequence-dependent setup time’.
Revised: In this paper, we discuss IPMS with ‘sequence-dependent setup time’.

Original: A machine’s setup time required for servicing a task i is determined by its previously served task j .
Revised: A machine’s setup time required for servicing a task i is determined by its previously served task j .

Original: In this case, the task completion time is the sum of setup time and processing time.
Revised: In this case, the task completion time is the sum of setup time and processing time.

Original: Under this setting, we solve IPMS problem for make-span minimization objective discussed in [Kurz et al.
Revised: Under this setting, we solve IPMS problem for make-span minimization objective discussed in [Kurz et al.

Original: (The constraints are different in this problem though);
Revised: (The constraints are different in this problem though);

Original: That is, minimizing total time spent from start to end to ﬁnish all tasks.
Revised: That is, minimizing total time spent from start to end to ﬁnish all tasks.

Original: Every time there is a ﬁnished task, we make assignment decision for a free machine.
Revised: Every time there is a ﬁnished task, we make assignment decision for a free machine.

Original: We call this times as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .
Revised: We call this times as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .

Original: , t k , . . . ) .
Revised: , t k , . . . ) .

Original: Abusing this notation slightly, we use ( · ) t k = ( · ) k .
Revised: Abusing this notation slightly, we use ( · ) t k = ( · ) k .

Original: Task completion time for a machine to a task consists of two components: processing time and setup time.
Revised: Task completion time for a machine to a task consists of two components: processing time and setup time.

Original: While a machine in setup phase may be reassigned to another task, we can’t reassign a machine in the processing phase.
Revised: While a machine in setup phase may be reassigned to another task, we can’t reassign a machine in the processing phase.

Original: Under these assumptions, at each epoch, a machine r i is given a set of tasks it can assign: if it is in the setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.
Revised: Under these assumptions, at each epoch, a machine r i is given a set of tasks it can assign: if it is in the setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.

Original: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:
Revised: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:

Original: State.
Revised: State.

Original: State s k at decision epoch
Revised: State s k at decision epoch

Original: k is a directed graph G k
Revised: k is a directed graph G k

Original: =
Revised: =

Original: ( R k ∪ T k , E k ) :
Revised: ( R k ∪ T k , E k ) :

Original: R k is the set of all machines and T k is the set of all tasks;
Revised: R k is the set of all machines and T k is the set of all tasks;

Original: The set of directed edges E k
Revised: The set of directed edges E k

Original: =
Revised: =

Original: E RTk ∪ E TTk where a directed edge (cid:15) r i t j ∈ E RTk is a random variable which denotes task completion time of machine i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a machine which just ﬁnished serving task i in T k to service task j in T k .
Revised: E RTk ∪ E TTk where a directed edge (cid:15) r i t j ∈ E RTk is a random variable which denotes task completion time of machine i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a machine which just ﬁnished serving task i in T k to service task j in T k .

Original: E RTk contains information about each robot’s possible assignments: E RTk =
Revised: E RTk contains information about each robot’s possible assignments: E RTk =

Original: ∪ i E r i k , where E r i k is a singleton set if machine i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from machine r i to remaining tasks that are not in the processing phase.
Revised: ∪ i E r i k , where E r i k is a singleton set if machine i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from machine r i to remaining tasks that are not in the processing phase.

Original: Action.
Revised: Action.

Original: Deﬁned the same as MRRC with continuous state/time space.
Revised: Deﬁned the same as MRRC with continuous state/time space.

Original: Reward.
Revised: Reward.

Original: In IPMS, time passes between decision epoch t and decision epoch t +1 .
Revised: In IPMS, time passes between decision epoch t and decision epoch t +1 .

Original: Denote this time as T t .
Revised: Denote this time as T t .

Original: One can easily see that T t is completely determined by s k , a k and s k
Revised: One can easily see that T t is completely determined by s k , a k and s k

Original: +1 .
Revised: +1 .

Original: Therefore, we can denote the reward we get with s k , a k and s k +1 as T ( s k , a k , s k +1 ) .
Revised: Therefore, we can denote the reward we get with s k , a k and s k +1 as T ( s k , a k , s k +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .

Original: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that
Revised: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that

Original: [Equation]
Revised: [Equation]

Original: C B AYESIAN N ETWORK REPRESENTATION
Revised: C B AYESIAN N ETWORK REPRESENTATION

Original: Here we analytically show that robot scheduling problem randomly induces a random Bayesian
Revised: Here we analytically show that robot scheduling problem randomly induces a random Bayesian

Original: Network from state s t .
Revised: Network from state s t .

Original: To do this, we ﬁrst deﬁne a concept ‘scenario at s t , a t with policy φ ’.

Original: Given starting state s t and action a t , a person canmay repeat a random experiment of sequential decision makingfrom s t , a t using  φ .
Revised: Given starting state s t and action a t , a person can repeat a random experiment of “sequential decision making using policy φ ”.

Original: In this random experiment, we can deﬁne events ‘How robots serve all remaining tasks in which sequence’.
Revised: In this random experiment, we can deﬁne events ‘How robots serve all remaining tasks in which sequence’.

Original: We call such an event a ‘scenario’.
Revised: We call such an event a ‘scenario’.

Original: For example, suppose that at time-step t we are given robots { A, B } , tasks { 1 , 2 , 3 , 4 , 5 } , and policy φ .
Revised: For example, suppose that at time-step t we are given robots { A, B } , tasks { 1 , 2 , 3 , 4 , 5 } , and policy φ .

Original: One possible scenario S ∗ can be { robot A serves task 3 → 1 → 2 and robot B serves task 5 → 4 } .
Revised: One possible scenario S ∗ can be { robot A serves task 3 → 1 → 2 and robot B serves task 5 → 4 } .

Original: Deﬁne random variable
Revised: Deﬁne random variable

Original: X k a task characteristic, e.g. ‘The time when task k is serviced’.
Revised: X k a task characteristic, e.g. ‘The time when task k is serviced’.

Original: The question is, ‘Given a scenario S ∗ , what is the relationship among random variables { X k } ’?
Revised: The question is, ‘Given a scenario S ∗ , what is the relationship among random variables { X k } ’?

Original: Recall that in our sequential decision making formulation we are given all the ‘task completion time’ information in the s t description.
Revised: Recall that in our sequential decision making formulation we are given all the ‘task completion time’ information in the s t description.

Original: Note that, task completion time is only dependent on the previous task and assigned task.
Revised: Note that, task completion time is only dependent on the previous task and assigned task.

Original: In our example above, under scenario S ∗ ‘when task 2 is served’ is only dependent on ‘when task 1 is served’.
Revised: In our example above, under scenario S ∗ ‘when task 2 is served’ is only dependent on ‘when task 1 is served’.

Original: That is, P ( X 2 | X 1 , X 3 , S ∗ ) =
Revised: That is, P ( X 2 | X 1 , X 3 , S ∗ ) =

Original: P ( X 2 | X 1 , S ∗ ) .
Revised: P ( X 2 | X 1 , S ∗ ) .

Original: This relationship is called ‘conditional independence’.
Revised: This relationship is called ‘conditional independence’.

Original: Given a scenario S ∗ , every relationship among { X i | S ∗ } can be expressed using this kind of relationship among random variables.
Revised: Given a scenario S ∗ , every relationship among { X i | S ∗ } can be expressed using this kind of relationship among random variables.

Original: A graph with this special relationship is called ‘Bayesian Network’
Revised: A graph with this special relationship is called ‘Bayesian Network’

Original: [Koller & Friedman (2009)], a probabilistic graphical model.
Revised: [Koller & Friedman (2009)], a probabilistic graphical model.

Original: D P ROOF OF T HEOREM 1.
Revised: D P ROOF OF T HEOREM 1.

Original: We start from formally stating what a probabilistic graphical model is.

Original: Given a set of random variables { X k } and set of semi-cliques C , suppose that we can express the joint distribution of { X k } as

Original: [Equation] where φ i clique potential for clique D i ∈ C and Z is a normalizing constant. Sometimes, we will suppress the explicit dependence on D i and write simply φ i . These functions φ i are often referred to as clique potentials.
Revised: We ﬁrst deﬁne necessary deﬁnitions for our proof.

Original: The collection D i is referred to as the scope of clique potential φ i .

Original: The set of cliques in a probabilistic graphical model is called a factorization and expressed as notation S ⊆ C .

Original: In a random graphical model, a PGM is chosen among  PGMs with { X k } and semi-cliques C .
Revised: In a random PGM, a PGM is chosen among all possible PGMs on { X k } and semi-cliques C .

Original: Denote the set of all possible factorization as F = {S 1 , S 2 , ..., S N } where a factorization with index k is denoted as S k ⊆ C .
Revised: Denote the set of all possible factorization as F = {S 1 , S 2 , ..., S N } where a factorization with index k is denoted as S k ⊆ C .

Original: Suppose we are given P (  S m  ) .
Revised: Suppose we are given P ( {S = S m } ) .

Original: For each semi-clique D i in C , deﬁne a binary random variable V
Revised: For each semi-clique D i in C , deﬁne a binary random variable V

Original: i : F (cid:55)→ { 0 , 1 } with value 0 for the factorization that does not include semi-clique D i and value 1 for the factorization that include semi-clique D i .
Revised: i : F (cid:55)→ { 0 , 1 } with valuefor the factorization that does not include semi-clique D i and value 1 for the factorization that include semi-clique D i .

Original: Let V be a random vector V
Revised: Let V be a random vector V

Original: = (cid:0) V 1 , V 2 , . .
Revised: = (cid:0) V 1 , V 2 , . .

Original: . , V | C | (cid:1) .
Revised: . , V | C | (cid:1) .

Original: Then we can express P ( X 1 , . . .
Revised: Then we can express P ( X 1 , . . .

Original: , X n | V ) ∝ (cid:81) | C | i =1 (cid:2)
Revised: , X n | V ) ∝ (cid:81) | C | i =1 (cid:2)

Original: φ i (cid:0) D i (cid:1)(cid:3) V i .In other words, { (cid:2) φ i (cid:0) D i (cid:1)(cid:3) V i } | C | i =1 factors
Revised: φ i (cid:0) D i (cid:1)(cid:3) V i .

Original: P ( X 1 , . . .

Original: , X n | V ) .

Original: We denote (cid:2)
Revised: We denote (cid:2)

Original: φ i (cid:0) D i (cid:1)(cid:3) V i as ψ ( D i ) .
Revised: φ i (cid:0) D i (cid:1)(cid:3) V i as ψ ( D i ) .

Original: Given { P (  S k ) } , each semi-clique D
Revised: Given { P ( {S = S m } ) } , each semi-clique D

Original: i ’s presence probability p
Revised: i ’s presence probability p

Original: i can be simply calculated; clique i ’s presence probability p
Revised: i can be simply calculated; clique i ’s presence probability p

Original: i is simply the sum of probabilities of all factorizations which include clique i , that is, p i = (cid:80) m : D i ∈S m P (  S m  ) .
Revised: i is simply the sum of probabilities of all factorizations which include clique i , that is, p i = (cid:80) m : D i ∈S m P ( {S = S m } ) .

Original: Now we prove Theorem 1.
Revised: Now we prove Theorem 1.

Original: In mean-ﬁeld inference, we want to ﬁnd a distribution Q ( X 1 , . . .
Revised: In mean-ﬁeld inference, we want to ﬁnd a distribution Q ( X 1 , . . .

Original: , X n ) = (cid:81)
Revised: , X n ) = (cid:81)

Original: ni =1 Q i ( X i ) such that the cross-entropy between it and a target distribution is minimized.
Revised: ni =1 Q i ( X i ) such that the cross-entropy between it and a target distribution is minimized.

Original: Following the notation in Koller  Friedman (2009) , the mean ﬁeld inference problem can written as the following optimization problem. 
Revised: Following the notation in Koller & Friedman (2009), the mean ﬁeld inference problem can written as the following optimization problem. min

Original: [Equation]
Revised: [Equation]

Original: Here D ( (cid:81)
Revised: Here D ( (cid:81)

Original: i Q i | P
Revised: i Q i | P

Original: [Equation]

Original:  X 1 , . ..
Revised: ( X 1 , . .

Original: , X n | V )) can be expressed as D ( (cid:81)
Revised: , X n | V )) can be expressed as D ( (cid:81)

Original: i Q i 
Revised: i Q i | P

Original: [Equation]
Revised: ( X 1 , . . , X n | V )) =

Original: P
Revised: E

Original: [Equation] X 1 , . . .[Equation]
Revised: Q [ln ( (cid:81) i Q i )] − E Q [ln ( P ( X 1 , . . .

Original:  X n | [Equation]
Revised: , X n | V ))] .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Hence, the above optimization problem can be written as
Revised: Hence [Equation] the above optimization problem can be written as

Original: [Equation]
Revised: [Equation]

Original: In Koller & Friedman (2009), the ﬁxed point equation is derived by solving an analogous equation to (2) without the presence of the p i .
Revised: In Koller & Friedman (2009), the ﬁxed point equation is derived by solving an analogous equation to (1) without the presence of the p i .

Original: Theorem 1 follows by proceeding as in Koller & Friedman (2009) with straightforward accounting for p i .
Revised: Theorem 1 follows by proceeding as in Koller & Friedman (2009) with straightforward accounting for p i .

Original: E P ROOF OF L EMMA 1.
Revised: E P ROOF OF L EMMA 1.

Original: Since we assume semi-cliques are only between two random variables, we can denote C
Revised: Since we assume semi-cliques are only between two random variables, we can denote C

Original: = D ij and presence probabilities as { p ij } where i, j are node indexes.
Revised: = D ij and presence probabilities as { p ij } where i, j are node indexes.

Original: Denote the set of nodes as V .
Revised: Denote the set of nodes as V .

Original: From here, we follow the approach of Dai et al.
Revised: From here, we follow the approach of Dai et al.

Original: and assume that the joint distribution of random variables can be written as
Revised: and assume that the joint distribution of random variables can be written as

Original: [Equation]
Revised: [Equation]

Original: Expanding the ﬁxed-point equation for the mean ﬁeld inference from Theorem 1, we obtain:
Revised: Expanding the ﬁxed-point equation for the mean ﬁeld inference from Theorem 1, we obtain:

Original: [Equation]
Revised: [Equation]

Original: This ﬁxed-point equation for Q k ( h k ) is a function of { Q j ( h j ) } j (cid:54) =
Revised: This ﬁxed-point equation for Q k ( h k ) is a function of { Q j ( h j ) } j (cid:54) =

Original: k such that
Revised: k such that

Original: [Equation]
Revised: [Equation]

Original: As in Dai et al.
Revised: As in Dai et al.

Original: [Equation] this equation can be expressed as a Hilbert space embedding of the form
Revised: (2016), this equation can be expressed as a Hilbert space embedding of the form

Original: [Equation]
Revised: [Equation]

Original: where ˜ µ k indicates a vector that encodes Q k ( h k ) .
Revised: where ˜ µ k indicates a vector that encodes Q k ( h k ) .

Original: In this paper, we use the nonlinear mapping ˜ T (based on a neural network form ) suggested in Dai et al. 
Revised: In this paper, we use the nonlinear mapping ˜ T (based on a neural network form ) suggested in Dai et al. [Equation] =

Original: [Equation] 
Revised: [Equation] 

Original:  F P RESENCE PROBABILITY INFERENCE
Revised:  F P RESENCE PROBABILITY INFERENCE

Original: Let V denote the set of nodes.
Revised: Let V denote the set of nodes.

Original: In lines 1 and 2 , the likelihood of the existence of a directed edge from each node m to node n is computed by calculating W 1 (cid:0) relu (cid:0)
Revised: In lines 1 and 2 , the likelihood of the existence of a directed edge from each node m to node n is computed by calculating W 1 (cid:0) relu (cid:0)

Original: W 2 u kmn (cid:1)(cid:1) and averaging over the M samples.
Revised: W 2 u kmn (cid:1)(cid:1) and averaging over the M samples.

Original: In lines 3 and 4 , we use the soft-max function to obtain p m,n .
Revised: In lines 3 and 4 , we use the soft-max function to obtain p m,n .

Original: 1 For m, n ∈ V do
Revised: 1 For m, n ∈ V do

Original: [Equation]
Revised: [Equation]

Original: 3 For m, n ∈ V do
Revised: 3 For m, n ∈ V do

Original: [Equation]
Revised: [Equation]

Original: G T ASK COMPLETION TIME AS A RANDOM VARIABLE
Revised: G T ASK COMPLETION TIME AS A RANDOM VARIABLE

Original: We combine random sampling and inference procedure suggested in section 3.2 and Figure ?? .
Revised: We combine random sampling and inference procedure suggested in section 3.2 and Figure 1.

Original: Denote the set of task with a robot assigned to it as T A .
Revised: Denote the set of task with a robot assigned to it as T A .

Original: Denote a task in T A as t i and the robot assigned to t i as r t i .
Revised: Denote a task in T A as t i and the robot assigned to t i as r t i .

Original: The corresponding edge in E RT for this assignment is (cid:15) r
Revised: The corresponding edge in E RT for this assignment is (cid:15) r

Original: ti t i .
Revised: ti t i .

Original: The key idea is to use samples of (cid:15) r ti t i to generate N number of sampled Q ( s, a ) value and average them to get the estimate of E ( Q ( s, a )) .
Revised: The key idea is to use samples of (cid:15) r ti t i to generate N number of sampled Q ( s, a ) value and average them to get the estimate of E ( Q ( s, a )) .

Original: First, for l = 1 . . .
Revised: First, for l = 1 . . .

Original: N we conduct the following procedure.
Revised: N we conduct the following procedure.

Original: For each task t i in T A , we sample one data e
Revised: For each task t i in T A , we sample one data e

Original: lr ti t i .
Revised: lr ti t i .

Original: Using those samples and { p ij } , we follow the whole procedure illustrated in section 3.2 to get Q ( s, a ) l .
Revised: Using those samples and { p ij } , we follow the whole procedure illustrated in section 3.2 to get Q ( s, a ) l .

Original: Second, we get the average of { Q ( s, a ) l } l = N l =1 to get the estimate of E ( Q ( s, a )) , 1 N (cid:80) l = N l =1 Q ( s, a ) l .
Revised: Second, we get the average of { Q ( s, a ) l } l = N l =1 to get the estimate of E ( Q ( s, a )) , 1 N (cid:80) l = N l =1 Q ( s, a ) l .

Original: The complete algorithm of section 3.2 with task completion time as a random variable is given as below.
Revised: The complete algorithm of section 3.2 with task completion time as a random variable is given as below.

Original: 1 age i = age of node

Original: [-i 
-]
Revised: [-i 
-]γ

Original: The set of nodes for assigned tasks ≡ T A 3 Initialize { µ (
Revised: Q l = W i ∈V γ 20 Q avg = 1 N (cid:80) Nl =

Original: ) i } , { γ (0) i }
Revised: Q