Original: L EARNING SCALABLE AND TRANSFERABLE MULTI - ROBOT / MACHINE  PLANNING VIA GRAPH EMBEDDING
Revised: L EARNING SCALABLE AND TRANSFERABLE MULTI - ROBOT / MACHINE SEQUENTIAL ASSIGNMENT PLANNING VIA GRAPH EMBEDDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There has recently been some success in the use of reinforcement learning methods for single robot combinatorial optimization problems.
Revised: There has recently been some success in the use of reinforcement learning methods for single robot combinatorial optimization problems.

Original: In this paper, we develop the ﬁrst learning-based method for multi robot/machine planning problems with combinatorial nature.
Revised: In this paper, we develop the ﬁrst learning-based method for multi robot/machine planning problems with combinatorial nature.

Original: One real-world concern is the capability to achieve transferability to an unseen number of robots and tasks.
Revised: One real-world concern is the capability to achieve transferability to an unseen number of robots and tasks.

Original: The method developed here, for the ﬁrst time, enables such transferability.
Revised: The method developed here, for the ﬁrst time, enables such transferability.

Original: Our method is comprised of three components.
Revised: Our method is comprised of three components.

Original: First, we illustrate how to represent a robot planning problem as an extension of probabilistic graphical models (PGMs) which we refer to as random PGMs.
Revised: First, we illustrate how to represent a robot planning problem as an extension of probabilistic graphical models (PGMs) which we refer to as random PGMs.

Original: We develop a mean-ﬁeld inference method for such random PGMs and use it for Q -function inference.
Revised: We develop a mean-ﬁeld inference method for such random PGMs and use it for Q -function inference.

Original: Second, we show that transferability can be achieved by carefully encoding the problem state via a two-step sequential process.
Revised: Second, we show that transferability can be achieved by carefully encoding the problem state via a two-step sequential process.

Original: Third, we resolve the computational scalability issue of ﬁtted Q -iteration.
Revised: Third, we resolve the computational scalability issue of ﬁtted Q -iteration.

Original: This is achieved by employing an auction-based heuristic as a substitute for the max operation in the Bellman equation.
Revised: This is achieved by employing an auction-based heuristic as a substitute for the max operation in the Bellman equation.

Original: The auction is enabled by the transferability achieved.
Revised: The auction is enabled by the transferability achieved.

Original: Our method is applicable to discrete-time, discrete space problems such as multirobot reward collection (MRRC).
Revised: Our method is applicable to discrete-time, discrete space problems such as multirobot reward collection (MRRC).

Original: For such problems, with determinsitic assumptoins, we scalably achieve 97% optimality with transferability.
Revised: For such problems, with determinsitic assumptoins, we scalably achieve 97% optimality with transferability.

Original: This optimality is maintained under stochastic contexts.
Revised: This optimality is maintained under stochastic contexts.

Original: By extending our method to a continuous time, continuous space formulation, the approach is the ﬁrst learning-based method for machine scheduling problems.
Revised: By extending our method to a continuous time, continuous space formulation, the approach is the ﬁrst learning-based method for machine scheduling problems.

Original: Our method scalably achieves comparable performance to popular metaheuristics when applied to identical parallel machine scheduling (IPMS) problems in a deterministic context.
Revised: Our method scalably achieves comparable performance to popular metaheuristics when applied to identical parallel machine scheduling (IPMS) problems in a deterministic context.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Suppose that we are given a set of robots and seek to serve a set of spatially distributed tasks.
Revised: Suppose that we are given a set of robots and seek to serve a set of spatially distributed tasks.

Original: A reward is given for serving each task promptly - resulting in a time-decaying reward collection problem - or when completing the entire set of tasks - resulting in a makespan minimization problem.
Revised: A reward is given for serving each task promptly - resulting in a time-decaying reward collection problem - or when completing the entire set of tasks - resulting in a makespan minimization problem.

Original: As the capability to control and route individual robots has increased [Li (2017)], efﬁcient orchestration of robots arises as an important remaining concern for such problems.
Revised: As the capability to control and route individual robots has increased [Li (2017)], efﬁcient orchestration of robots arises as an important remaining concern for such problems.

Original: Multi-robot planning problems.
Revised: Multi-robot planning problems.

Original: In this paper, we focus on orchestration problems that can be formulated as robot planning problems.
Revised: In this paper, we focus on orchestration problems that can be formulated as robot planning problems.

Original: A key assumption in such orchestration problems is that we are given information on the “duration of time required for an assigned robot to complete a task”.
Revised: A key assumption in such orchestration problems is that we are given information on the “duration of time required for an assigned robot to complete a task”.

Original: This duration may be deterministic (e.g. as in a Traveling Salesman Problem (TSP) or Vehicle
Revised: This duration may be deterministic (e.g. as in a Traveling Salesman Problem (TSP) or Vehicle

Original: Routing Problem (VRP)) or random with given probability distribution (c.f., [Omidshaﬁei et al.
Revised: Routing Problem (VRP)) or random with given probability distribution (c.f., [Omidshaﬁei et al.

Original: We call this duration the task completion time .
Revised: We call this duration the task completion time .

Original: Due to their combinatorial nature, robot planning problems suffer from exponential computational complexity.
Revised: Due to their combinatorial nature, robot planning problems suffer from exponential computational complexity.

Original: Even in the context of single-robot scheduling problems (e.g., TSP) scalability is a concern.
Revised: Even in the context of single-robot scheduling problems (e.g., TSP) scalability is a concern.

Original: Planning for multiple robots exacerbates the scalability issue.
Revised: Planning for multiple robots exacerbates the scalability issue.

Original: While scalable heuristic methods have been developed for various deterministic multi-robot planning problems (c.f., [Rossi et al.
Revised: While scalable heuristic methods have been developed for various deterministic multi-robot planning problems (c.f., [Rossi et al.

Original: (2018)]), no heuristic methods have been developed for stochastic multi-robot planning problems that are simultaneously both near optimal and scalable.
Revised: (2018)]), no heuristic methods have been developed for stochastic multi-robot planning problems that are simultaneously both near optimal and scalable.

Original: Learning-based planning methods.
Revised: Learning-based planning methods.

Original: Recently, seminal learning-based planning algorithms have been developed for the scalable solution of TSPs [Bello et al.
Revised: Recently, seminal learning-based planning algorithms have been developed for the scalable solution of TSPs [Bello et al.

Original: ; Dai et al.
Revised: ; Dai et al.

Original: (2017); Kool et al.
Revised: (2017); Kool et al.

Original: They showed that learning methods can exploit the recurring structure of TSP and thus can generate near-optimal solution in a very fast computation time.
Revised: They showed that learning methods can exploit the recurring structure of TSP and thus can generate near-optimal solution in a very fast computation time.

Original: However, those successes were restricted to single-robot problems except for special cases when the problem can be modeled as a variant of single-robot TSP via multiple successive journeys of a single robot [Nazari et al.
Revised: However, those successes were restricted to single-robot problems except for special cases when the problem can be modeled as a variant of single-robot TSP via multiple successive journeys of a single robot [Nazari et al.

Original: Near-optimal planning with scalability.
Revised: Near-optimal planning with scalability.

Original: In this paper, we address a general type of problem called Multi-Robot Reward Collection (MRRC) and show that this problem can be scalably solved by our method with near-optimal performance.
Revised: In this paper, we address a general type of problem called Multi-Robot Reward Collection (MRRC) and show that this problem can be scalably solved by our method with near-optimal performance.

Original: Generally, the training requirements for learningbased methods increase exponentially with the problem size (number of robots and tasks); c.f., [Li (2017)].
Revised: Generally, the training requirements for learningbased methods increase exponentially with the problem size (number of robots and tasks); c.f., [Li (2017)].

Original: We empirically demonstrate that the training requirements of proposed method scale well while maintaining near-optimal performance.
Revised: We empirically demonstrate that the training requirements of proposed method scale well while maintaining near-optimal performance.

Original: While we assumed discrete-state discrete-time conditions for MRRC, the method extends to continuous-state continuous-time problems.
Revised: While we assumed discrete-state discrete-time conditions for MRRC, the method extends to continuous-state continuous-time problems.

Original: Our method is also the ﬁrst learning-based method with scalable performance that can address machine scheduling problems.
Revised: Our method is also the ﬁrst learning-based method with scalable performance that can address machine scheduling problems.

Original: The approach achieves scalability (e.g. 10 machines, 100 tasks) with comparable performance to popular heuristics when applied to Identical Parallel Machine Scheduling (IPMS).
Revised: The approach achieves scalability (e.g. 10 machines, 100 tasks) with comparable performance to popular heuristics when applied to Identical Parallel Machine Scheduling (IPMS).

Original: Transferability.
Revised: Transferability.

Original: The proposed method possesses transferability in that a trained policy can be applied to new environments with a small loss of performance [Wang et al.
Revised: The proposed method possesses transferability in that a trained policy can be applied to new environments with a small loss of performance [Wang et al.

Original: Such capability is important for real-world applications.
Revised: Such capability is important for real-world applications.

Original: Engineers are rarely able to train a heuristic on a largescale system during operation, but rather use a small-scale testbed for training.
Revised: Engineers are rarely able to train a heuristic on a largescale system during operation, but rather use a small-scale testbed for training.

Original: Even if the whole system is available for training, the system design or number of robots may frequently change during operation.
Revised: Even if the whole system is available for training, the system design or number of robots may frequently change during operation.

Original: We show that our method achieves transferability with small performance loss.
Revised: We show that our method achieves transferability with small performance loss.

Original: Proposed methods.
Revised: Proposed methods.

Original: In the seminal paper [Dai et al.
Revised: In the seminal paper [Dai et al.

Original: , the authors observed that combinatorial optimization problems such as TSP can be formulated as sequential decision making problems.
Revised: , the authors observed that combinatorial optimization problems such as TSP can be formulated as sequential decision making problems.

Original: Decision making in such a sequential framework relies on an estimate of future costs Q ( s, a ) for an existing task sequence s and candidate next task a .
Revised: Decision making in such a sequential framework relies on an estimate of future costs Q ( s, a ) for an existing task sequence s and candidate next task a .

Original: With this estimate, given the prior decisions s at each decision step, they select the next task a to minimize the future cost estimate.
Revised: With this estimate, given the prior decisions s at each decision step, they select the next task a to minimize the future cost estimate.

Original: [Dai et al.
Revised: [Dai et al.

Original: (2017)]’s solution framework relies on the following three assumptions.
Revised: (2017)]’s solution framework relies on the following three assumptions.

Original: For each combinatorial optimization problem, one can heuristically choose how to induce a graph representation of ( s, a ) .
Revised: For each combinatorial optimization problem, one can heuristically choose how to induce a graph representation of ( s, a ) .

Original: In the case of TSP, the paper induces a fully connected graph for every possible next task.
Revised: In the case of TSP, the paper induces a fully connected graph for every possible next task.

Original: This induced graph representation can be considered as a probabilistic graphical model (PGM)
Revised: This induced graph representation can be considered as a probabilistic graphical model (PGM)

Original: [Koller &
Revised: [Koller &

Original: Friedman (2009)].
Revised: Friedman (2009)].

Original: This PGM can be used with a graph-based mean-ﬁeld inference method called structure2vec
Revised: This PGM can be used with a graph-based mean-ﬁeld inference method called structure2vec

Original: [Dai et al.
Revised: [Dai et al.

Original: (2016)] to infer Q ( s, a ) for use in combinatorial optimization problems.
Revised: (2016)] to infer Q ( s, a ) for use in combinatorial optimization problems.

Original: Inference of Q ( s, a ) can be learned by the reinforcement framework called ﬁtted Q-iteration.
Revised: Inference of Q ( s, a ) can be learned by the reinforcement framework called ﬁtted Q-iteration.

Original: We create a solution framework to achieve scalability and transferability for multi-robot planning that builds in numerous directions upon the foundation of [Dai et al.
Revised: We create a solution framework to achieve scalability and transferability for multi-robot planning that builds in numerous directions upon the foundation of [Dai et al.

Original: (2017)] as follows:
Revised: (2017)] as follows:

Original: State representation and mean-ﬁeld inference theory for random PGM.
Revised: State representation and mean-ﬁeld inference theory for random PGM.

Original: Instead of heuristically inducing a PGM, we show that a robot scheduling problem exactly induces a random PGM.
Revised: Instead of heuristically inducing a PGM, we show that a robot scheduling problem exactly induces a random PGM.

Original: Since there exists no mean-ﬁeld inference theory for random PGM
Revised: Since there exists no mean-ﬁeld inference theory for random PGM

Original: we develop the theory and corresponding new structure2vec iteration.
Revised: we develop the theory and corresponding new structure2vec iteration.

Original: Sequential encoding of information for transferability.
Revised: Sequential encoding of information for transferability.

Original: To achieve transferability in terms of the number of robots and tasks, we carefully design a two-step hierarchical mean-ﬁeld inference [Ranganath et al.
Revised: To achieve transferability in terms of the number of robots and tasks, we carefully design a two-step hierarchical mean-ﬁeld inference [Ranganath et al.

Original: Each step is designed to infer certain information.
Revised: Each step is designed to infer certain information.

Original: The ﬁrst step is designed to infer each task’s relative graphical distance from the robots .
Revised: The ﬁrst step is designed to infer each task’s relative graphical distance from the robots .

Original: The second step is designed to infer Q ( s,
Revised: The second step is designed to infer Q ( s,

Original: a ) ( a here refers to a joint assignment of robots).
Revised: a ) ( a here refers to a joint assignment of robots).

Original: While the ﬁrst step is by its nature transferable to any number of tasks and robots, the transferability in inference of the second step is achieved by the scale-free characteristic of ﬁtted Q-iteration [van Hasselt et al.
Revised: While the ﬁrst step is by its nature transferable to any number of tasks and robots, the transferability in inference of the second step is achieved by the scale-free characteristic of ﬁtted Q-iteration [van Hasselt et al.

Original: That is, the relative magnitudes of Q ( s, a ) values are sufﬁcient to select an action a .
Revised: That is, the relative magnitudes of Q ( s, a ) values are sufﬁcient to select an action a .

Original: Auction-based assignment.
Revised: Auction-based assignment.

Original: Even if we can infer Q ( s, a ) precisely, the computation time required to select an action a using the maximum Q ( s, a ) operation exponentially increases as robots and tasks increase.
Revised: Even if we can infer Q ( s, a ) precisely, the computation time required to select an action a using the maximum Q ( s, a ) operation exponentially increases as robots and tasks increase.

Original: To resolve this issue, we suggest a heuristic auction that is enabled by the transferability of our Q ( s, a ) inference.
Revised: To resolve this issue, we suggest a heuristic auction that is enabled by the transferability of our Q ( s, a ) inference.

Original: Even though this heuristic auction selects a with only polynomial computational complexity, it provides surprisingly good choices for a .
Revised: Even though this heuristic auction selects a with only polynomial computational complexity, it provides surprisingly good choices for a .

Original: (In fact, this heuristic auction increases the performance empirically relative to using the max operation.)
Revised: (In fact, this heuristic auction increases the performance empirically relative to using the max operation.)

Original: Auction-ﬁtted Q-iteration.
Revised: Auction-ﬁtted Q-iteration.

Original: The heuristic auction-based action selection can be incorporated into learning (ﬁtting) Q ( s, a ) .
Revised: The heuristic auction-based action selection can be incorporated into learning (ﬁtting) Q ( s, a ) .

Original: To be speciﬁc, we use the auction-based action selection scheme, instead of the typical max-operator based action selection, in the Bellman equation during ﬁtted Q-iteration.
Revised: To be speciﬁc, we use the auction-based action selection scheme, instead of the typical max-operator based action selection, in the Bellman equation during ﬁtted Q-iteration.

Original: We call this new learning framework as auction-ﬁtted Q-iteration .
Revised: We call this new learning framework as auction-ﬁtted Q-iteration .

Original:  M ULTI - ROBOT / MACHINE SCHEDULING PROBLEM FORMULATION
Revised:  M ULTI - ROBOT / MACHINE SCHEDULING PROBLEM FORMULATION

Original: We consider centralized sequential assignment planning problems.
Revised: We consider centralized sequential assignment planning problems.

Original: We assume perfect communication, that is, the decision maker can always determine the distribution of task completion times.
Revised: We assume perfect communication, that is, the decision maker can always determine the distribution of task completion times.

Original: 2. M ULTI -R OBOT R EWARD C OLLECTION (MRRC)
Revised: 2. M ULTI -R OBOT R EWARD C OLLECTION (MRRC)

Original: While we consider extended versions of the problem in Appendix A 2 , here we formulate an MRRC as a discrete-time, discrete-state planning problem.
Revised: While we consider extended versions of the problem in Appendix A 2 , here we formulate an MRRC as a discrete-time, discrete-state planning problem.

Original: The initial location and ending location of robots and tasks are arbitrary on a grid (e.g., grid world).
Revised: The initial location and ending location of robots and tasks are arbitrary on a grid (e.g., grid world).

Original: We assume that a task is served immediately after a robot arrives.
Revised: We assume that a task is served immediately after a robot arrives.

Original: Under these assumptions, at each time-step, we can assign every robot to every remaining task.
Revised: Under these assumptions, at each time-step, we can assign every robot to every remaining task.

Original: This MRRC problem can be cast as a Markov Decision Process (MDP) whose state, action, and reward are deﬁned as follows.
Revised: This MRRC problem can be cast as a Markov Decision Process (MDP) whose state, action, and reward are deﬁned as follows.

Original: State.
Revised: State.

Original: The state s t at time step t is a directed graph G t =
Revised: The state s t at time step t is a directed graph G t =

Original: ( R t ∪ T t , E t ) .
Revised: ( R t ∪ T t , E t ) .

Original: R t is the set of available robots at time step t .
Revised: R t is the set of available robots at time step t .

Original: T t is the set of all remaining unserved tasks at time step t .
Revised: T t is the set of all remaining unserved tasks at time step t .

Original: The set of directed edges E t =
Revised: The set of directed edges E t =

Original: E RTt ∪ E TTt .
Revised: E RTt ∪ E TTt .

Original: A directed edge (cid:15) r i t j ∈ E RTt has a random variable as its weight which denotes the task completion time for robot i in R t if it is assigned at time step t to serve task j in
Revised: A directed edge (cid:15) r i t j ∈ E RTt has a random variable as its weight which denotes the task completion time for robot i in R t if it is assigned at time step t to serve task j in

Original: T t (note this time subsumes the information about the robot’s present location).
Revised: T t (note this time subsumes the information about the robot’s present location).

Original: A directed edge (cid:15) t i t j ∈ E TTt has as its weight the task completion time for a robot which just ﬁnished serving task i in T t (and is therefore located where task i resided prior to its completion) to serve task j in T t if it is assigned at time step t .
Revised: A directed edge (cid:15) t i t j ∈ E TTt has as its weight the task completion time for a robot which just ﬁnished serving task i in T t (and is therefore located where task i resided prior to its completion) to serve task j in T t if it is assigned at time step t .

Original: Action.
Revised: Action.

Original: The action a t at time step t is the joint assignment of robots given the current state s t =
Revised: The action a t at time step t is the joint assignment of robots given the current state s t =

Original: G t .
Revised: G t .

Original: Feasible actions should satisfy two constraints: (i) no two robots can be assigned to the same task and (ii) a robot may not be assigned when the number of robots exceeds the number of remaining tasks 3 .
Revised: Feasible actions should satisfy two constraints: (i) no two robots can be assigned to the same task and (ii) a robot may not be assigned when the number of robots exceeds the number of remaining tasks 3 .

Original: To articulate an action, note ﬁrst that the two set of nodes R t and T t are disjoint.
Revised: To articulate an action, note ﬁrst that the two set of nodes R t and T t are disjoint.

Original: As such, the sub-graph ( R t ∪ T t , E RTt ) of graph G t is bipartite.
Revised: As such, the sub-graph ( R t ∪ T t , E RTt ) of graph G t is bipartite.

Original: We thus deﬁne an action a t at time t as a maximal bipartite matching in the bipartite sub-graph ( R t ∪ T t , E RTt ) .
Revised: We thus deﬁne an action a t at time t as a maximal bipartite matching in the bipartite sub-graph ( R t ∪ T t , E RTt ) .

Original: For example, robot i in R t is matched with task j in T t in an action a t if we assign robot i to task j at time step t .
Revised: For example, robot i in R t is matched with task j in T t in an action a t if we assign robot i to task j at time step t .

Original: We denote the set of all possible actions at time step t as A t .
Revised: We denote the set of all possible actions at time step t as A t .

Original: Reward.
Revised: Reward.

Original: In MRRC, each task has an arbitrarily determined initial age.
Revised: In MRRC, each task has an arbitrarily determined initial age.

Original: At each time-step, the age of each task increases by one.
Revised: At each time-step, the age of each task increases by one.

Original: When a task is served, a reward is determined based only on its age at the time of service.
Revised: When a task is served, a reward is determined based only on its age at the time of service.

Original: Note that the state and assignment information s t , a t and s t +1 is sufﬁcient to determine the reward at time step t + 1 .
Revised: Note that the state and assignment information s t , a t and s t +1 is sufﬁcient to determine the reward at time step t + 1 .

Original: As such we denote the reward as R ( s t , a t , s t +1 ) .
Revised: As such we denote the reward as R ( s t , a t , s t +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s t to an action a t .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s t to an action a t .

Original: Given an initial state s 0 , the MRRC seeks to maximize the sum of expected rewards through time by the selection of an assignment policy φ ∗ satisfying
Revised: Given an initial state s 0 , the MRRC seeks to maximize the sum of expected rewards through time by the selection of an assignment policy φ ∗ satisfying

Original: [Equation]
Revised: [Equation]

Original: 2. I DENTICAL PARALLEL MACHINE SCHEDULING (IPMS) MAKE - SPAN MINIMIZATION
Revised: 2. I DENTICAL PARALLEL MACHINE SCHEDULING (IPMS) MAKE - SPAN MINIMIZATION

Original: IPMS is a continuous-time continuous-state problem consisting of diverse tasks which must be served by identical machines.
Revised: IPMS is a continuous-time continuous-state problem consisting of diverse tasks which must be served by identical machines.

Original: Once service of a task i begins, it requires a deterministic duration of time τ i to complete - we call this the processsing time.
Revised: Once service of a task i begins, it requires a deterministic duration of time τ i to complete - we call this the processsing time.

Original: This time is the same independent of which machine serves the task.
Revised: This time is the same independent of which machine serves the task.

Original: We incorporate one popular extension and allow ‘sequence-dependent setup times’.
Revised: We incorporate one popular extension and allow ‘sequence-dependent setup times’.

Original: In this case, a machine must conduct a setup prior to serving each task.
Revised: In this case, a machine must conduct a setup prior to serving each task.

Original: The duration of this setup depends on the current task
Revised: The duration of this setup depends on the current task

Original: i and the task j that was previously served on that machine - we call this the setup time.
Revised: i and the task j that was previously served on that machine - we call this the setup time.

Original: The completion time for each task is thus the sum of the setup time and processing time.
Revised: The completion time for each task is thus the sum of the setup time and processing time.

Original: Under this setting, we solve the IPMS problem for make-span minimization as discussed in [Kurz et al.
Revised: Under this setting, we solve the IPMS problem for make-span minimization as discussed in [Kurz et al.

Original: That is, we seek to minimize the total time spent from the start time to the completion of the last task.
Revised: That is, we seek to minimize the total time spent from the start time to the completion of the last task.

Original: The IPMS formulation resembles our MRRC formulation in continuous-time and continuous-space and we relegate the detailed formulation to Appendix B.
Revised: The IPMS formulation resembles our MRRC formulation in continuous-time and continuous-space and we relegate the detailed formulation to Appendix B.

Original:  C HOICE OF ASSIGNMENT AT EACH TIME - STEP
Revised:  C HOICE OF ASSIGNMENT AT EACH TIME - STEP

Original: In Section 2, we formulated multi-robot/machine planning problems as sequential joint assignment decision problems.
Revised: In Section 2, we formulated multi-robot/machine planning problems as sequential joint assignment decision problems.

Original: As in [Dai et al.
Revised: As in [Dai et al.

Original: , we will select a joint assignment using a Q-function based policy.
Revised: , we will select a joint assignment using a Q-function based policy.

Original: Since we thus choose action a t with the largest inferred Q ( s t , a t ) value in state s t , the development of a Q ( s t , a t ) inference method is a key issue.
Revised: Since we thus choose action a t with the largest inferred Q ( s t , a t ) value in state s t , the development of a Q ( s t , a t ) inference method is a key issue.

Original: Toward this end and motivated by these robot planning problems, we provide new results in random PGM-based mean-ﬁeld inference methods and a subsequent extension of the graph-neural network based inference method called structure2vec
Revised: Toward this end and motivated by these robot planning problems, we provide new results in random PGM-based mean-ﬁeld inference methods and a subsequent extension of the graph-neural network based inference method called structure2vec

Original: [Dai et al.
Revised: [Dai et al.

Original: in Section 3.1.
Revised: in Section 3.1.

Original: In Section 3.2, we discuss how a careful encoding of information using the extended structure2vec of Section 3.1 enables precise and transferable Q ( s t , a t ) inference.
Revised: In Section 3.2, we discuss how a careful encoding of information using the extended structure2vec of Section 3.1 enables precise and transferable Q ( s t , a t ) inference.

Original: Since the computational complexity required to identify the best joint assignment is exponential with respect to the number of robots and tasks, Section 3.3 discusses how the transferability of our Q ( s t , a t ) inference method enables a good action choice heuristic with polynomial computational complexity.
Revised: Since the computational complexity required to identify the best joint assignment is exponential with respect to the number of robots and tasks, Section 3.3 discusses how the transferability of our Q ( s t , a t ) inference method enables a good action choice heuristic with polynomial computational complexity.

Original: 3. R OBOT SCHEDULING AS R ANDOM PGM- BASED MEAN - FIELD INFERENCE
Revised: 3. R OBOT SCHEDULING AS R ANDOM PGM- BASED MEAN - FIELD INFERENCE

Original: PGM.
Revised: PGM.

Original: Given random variables { X k } , suppose that joint distribution of { X k } can be factored as
Revised: Given random variables { X k } , suppose that joint distribution of { X k } can be factored as

Original: P ( X 1 , . . . , X n ) = 1 Z (cid:81) i φ i (cid:0) D i (cid:1) where φ i ( D i ) denotes a marginal distribution or conditional distribution on a set of random variables
Revised: P ( X 1 , . . . , X n ) = 1 Z (cid:81) i φ i (cid:0) D i (cid:1) where φ i ( D i ) denotes a marginal distribution or conditional distribution on a set of random variables

Original: D i .
Revised: D i .

Original: Z is a normalizing constant.
Revised: Z is a normalizing constant.

Original: Then { X k } is called a probabilistic graphical model (PGM).
Revised: Then { X k } is called a probabilistic graphical model (PGM).

Original: In a PGM, D i is called a clique and φ i ( D i ) is called a clique potential for D i .
Revised: In a PGM, D i is called a clique and φ i ( D i ) is called a clique potential for D i .

Original: When we suppress φ i ( D i ) as φ i , D i is referred to as the scope of φ i .
Revised: When we suppress φ i ( D i ) as φ i , D i is referred to as the scope of φ i .

Original: PGM-based mean-ﬁeld inference.
Revised: PGM-based mean-ﬁeld inference.

Original: One popular use of this PGM information is PGM-based mean-ﬁeld inference.
Revised: One popular use of this PGM information is PGM-based mean-ﬁeld inference.

Original: In mean-ﬁeld inference, we ﬁnd a surrogate distribution Q ( X 1 , . . .
Revised: In mean-ﬁeld inference, we ﬁnd a surrogate distribution Q ( X 1 , . . .

Original: , X n ) = (cid:81)
Revised: , X n ) = (cid:81)

Original: i Q i ( x i ) that has smallest Kullback-Leibler distance to original joint distribution P ( X 1 , . . .
Revised: i Q i ( x i ) that has smallest Kullback-Leibler distance to original joint distribution P ( X 1 , . . .

Original: , X n ) .
Revised: , X n ) .

Original: We then use this surrogate distribution to solve the original inference problem.
Revised: We then use this surrogate distribution to solve the original inference problem.

Original: [Koller & Friedman (2009)] shows that when we are given PGM information, { Q i ( x i ) } can be analytically computed by a ﬁxed point equation.
Revised: [Koller & Friedman (2009)] shows that when we are given PGM information, { Q i ( x i ) } can be analytically computed by a ﬁxed point equation.

Original: Despite that this usefulness, in most inference problems it is unrealistic to assume we know or can infer probability distributions of a PGM.
Revised: Despite that this usefulness, in most inference problems it is unrealistic to assume we know or can infer probability distributions of a PGM.

Original: This limitation was addressed in [Dai et al.
Revised: This limitation was addressed in [Dai et al.

Original: (2016)] using a method called structure2vec.
Revised: (2016)] using a method called structure2vec.

Original: Structure2vec.
Revised: Structure2vec.

Original: [Dai et al.
Revised: [Dai et al.

Original: (2016)] suggests that an inference problem with graph-structured data (e.g. a molecule classiﬁcation problem) can be seen as a particular PGM structure that consists of two types of random variables.
Revised: (2016)] suggests that an inference problem with graph-structured data (e.g. a molecule classiﬁcation problem) can be seen as a particular PGM structure that consists of two types of random variables.

Original: One type of random variables { X k } is one that serves as input of inference problem (e.g. X k denotes atomic number of atom k ).
Revised: One type of random variables { X k } is one that serves as input of inference problem (e.g. X k denotes atomic number of atom k ).

Original: Another type of random variables { H k } is latent random variable where H k is a latent random variable related to X k .
Revised: Another type of random variables { H k } is latent random variable where H k is a latent random variable related to X k .

Original: Existence of probabilistic relationships among { H k } are assumed heuristically from graph structure of data.
Revised: Existence of probabilistic relationships among { H k } are assumed heuristically from graph structure of data.

Original: Then the particular PGM structure they assume is P ( { H k } , { X k } )
Revised: Then the particular PGM structure they assume is P ( { H k } , { X k } )

Original: ∝ (cid:81) k ∈V φ ( H k | X k ) (cid:81) k,i ∈V φ ( H k | H i ) , where V denotes the set of vertex indexes.
Revised: ∝ (cid:81) k ∈V φ ( H k | X k ) (cid:81) k,i ∈V φ ( H k | H i ) , where V denotes the set of vertex indexes.

Original: The goal of mean-ﬁeld inference problem is to ﬁnd a surrogate distribution Q k ( h k ) for posterior marginal
Revised: The goal of mean-ﬁeld inference problem is to ﬁnd a surrogate distribution Q k ( h k ) for posterior marginal

Original: P ( { h k }|{ x k } ) .
Revised: P ( { h k }|{ x k } ) .

Original: However, we can’t compute { Q k ( h k ) } since we are not given φ ( H k | H i ) nor
Revised: However, we can’t compute { Q k ( h k ) } since we are not given φ ( H k | H i ) nor

Original: φ ( H k | X k ) .
Revised: φ ( H k | X k ) .

Original: To overcome this limitation, [Dai et al.
Revised: To overcome this limitation, [Dai et al.

Original: (2016)] develops a method called structure2vec that only requires the structure of the PGM for mean-ﬁeld inference.
Revised: (2016)] develops a method called structure2vec that only requires the structure of the PGM for mean-ﬁeld inference.

Original: structure2vec embeds the mean-ﬁeld inference procedure, i.e. ﬁxed point iteration on { Q k ( h k ) } , into ﬁxed point iterations of neural networks on vectors { ˜ µ k } .
Revised: structure2vec embeds the mean-ﬁeld inference procedure, i.e. ﬁxed point iteration on { Q k ( h k ) } , into ﬁxed point iterations of neural networks on vectors { ˜ µ k } .

Original: Derivation of such ﬁxed point iterations of neural networks can be found in Dai et al.
Revised: Derivation of such ﬁxed point iterations of neural networks can be found in Dai et al.

Original: and can be written as ˜ µ k = σ (cid:16) W 1 x k + W 2 (cid:80) j (cid:54) =
Revised: and can be written as ˜ µ k = σ (cid:16) W 1 x k + W 2 (cid:80) j (cid:54) =

Original: k ˜ µ j (cid:17) where σ denotes
Revised: k ˜ µ j (cid:17) where σ denotes

Original: Relu function and W denotes parameters of neural networks.
Revised: Relu function and W denotes parameters of neural networks.

Original: Robot scheduling as random PGM-based mean-ﬁeld inference.
Revised: Robot scheduling as random PGM-based mean-ﬁeld inference.

Original: All applications of structure2vec in [Dai et al.
Revised: All applications of structure2vec in [Dai et al.

Original: (2016; 2017)] heuristically decide the structure of PGM of each data point from its graph structure.
Revised: (2016; 2017)] heuristically decide the structure of PGM of each data point from its graph structure.

Original: The key observation we make is that inference problems in robot scheduling exactly induce a ‘random’ PGM structure (to be precise, a ‘random’ Bayesian Network).
Revised: The key observation we make is that inference problems in robot scheduling exactly induce a ‘random’ PGM structure (to be precise, a ‘random’ Bayesian Network).

Original: Given that we start from state s t and action a t , consider a random experiment “sequential decision making using policy φ ”.
Revised: Given that we start from state s t and action a t , consider a random experiment “sequential decision making using policy φ ”.

Original: In this experiment, we can deﬁne an event as ‘How robots serve all the remaining tasks in which sequence’.
Revised: In this experiment, we can deﬁne an event as ‘How robots serve all the remaining tasks in which sequence’.

Original: We call one such event a ‘scenario’.
Revised: We call one such event a ‘scenario’.

Original: For each task t i ∈ T t , deﬁne a random variable X i as ‘a characteristic of
Revised: For each task t i ∈ T t , deﬁne a random variable X i as ‘a characteristic of

Original: task t
Revised: task t

Original: i ’ (e.g. when task i is served).
Revised: i ’ (e.g. when task i is served).

Original: Given a scenario, the relationships among { X i } satisfy as a Bayesian Network.
Revised: Given a scenario, the relationships among { X i } satisfy as a Bayesian Network.

Original: For details, see Appendix C)
Revised: For details, see Appendix C)

Original: Note that we do not know which scenario will occur from time t and thus do not know which PGM will be realized.
Revised: Note that we do not know which scenario will occur from time t and thus do not know which PGM will be realized.

Original: Besides, the inference of probability of each scenario is challenging.
Revised: Besides, the inference of probability of each scenario is challenging.

Original: Putting aside this problem for a while, we ﬁrst deﬁne a ‘random PGM’ and ‘semi-cliques’.
Revised: Putting aside this problem for a while, we ﬁrst deﬁne a ‘random PGM’ and ‘semi-cliques’.

Original: Denote the set of all random variables in the inference problem as X =
Revised: Denote the set of all random variables in the inference problem as X =

Original: { X i } .
Revised: { X i } .

Original: A random PGM is a probabilistic model of how a PGM is randomly chosen from a set of all possible PGMs on X 4 .
Revised: A random PGM is a probabilistic model of how a PGM is randomly chosen from a set of all possible PGMs on X 4 .

Original: Next, denote the set of all possible probabilistic relationships on X as C X .
Revised: Next, denote the set of all possible probabilistic relationships on X as C X .

Original: We call them ‘semi-cliques’.
Revised: We call them ‘semi-cliques’.

Original: In robot scheduling problem, a semi-clique D ij ∈ C X is a conditional dependence
Revised: In robot scheduling problem, a semi-clique D ij ∈ C X is a conditional dependence

Original: X i | X j .
Revised: X i | X j .

Original: The semi-clique D ij presents as an actual clique if and only if the robot which ﬁnishes task t i chooses task t
Revised: The semi-clique D ij presents as an actual clique if and only if the robot which ﬁnishes task t i chooses task t

Original: j as the next task.
Revised: j as the next task.

Original: We will now prove that we don’t have to infer the probability of each scenario, i.e. random PGM model itself.
Revised: We will now prove that we don’t have to infer the probability of each scenario, i.e. random PGM model itself.

Original: The following theorem for mean-ﬁeld inference with random PGM is an extension of mean-ﬁeld inference with PGM [Koller & Friedman (2009)] and suggests that only a simple inference task is required: inference of the presence probability of each semi-cliques.
Revised: The following theorem for mean-ﬁeld inference with random PGM is an extension of mean-ﬁeld inference with PGM [Koller & Friedman (2009)] and suggests that only a simple inference task is required: inference of the presence probability of each semi-cliques.

Original: Theorem 1.
Revised: Theorem 1.

Original: Random PGM based mean ﬁeld inference Suppose we are given a random PGM on X
Revised: Random PGM based mean ﬁeld inference Suppose we are given a random PGM on X

Original: = {
Revised: = {

Original: X k } .
Revised: X k } .

Original: Also, assume that we know presence probability { p m } for all semi-cliques C X
Revised: Also, assume that we know presence probability { p m } for all semi-cliques C X

Original: = {D m } .
Revised: = {D m } .

Original: The latent variable distribution { Q k ( x k ) } in mean-ﬁeld inference is locally optimal only if
Revised: The latent variable distribution { Q k ( x k ) } in mean-ﬁeld inference is locally optimal only if

Original: [Equation]
Revised: [Equation]

Original: where Z k is a normalizing constant and φ m is the clique potential for clique m .
Revised: where Z k is a normalizing constant and φ m is the clique potential for clique m .

Original: From this new result, we can develop the structure2vec inference method for random PGM.
Revised: From this new result, we can develop the structure2vec inference method for random PGM.

Original: As in [Dai et al.
Revised: As in [Dai et al.

Original: , we restrict our discussion to when every semi-clique is between two random variables.
Revised: , we restrict our discussion to when every semi-clique is between two random variables.

Original: In this case, a semi-clique can be written as D ij with its presence probability p ij .
Revised: In this case, a semi-clique can be written as D ij with its presence probability p ij .

Original: Lemma 1.
Revised: Lemma 1.

Original: Structure2vec for random PGM.
Revised: Structure2vec for random PGM.

Original: Suppose we are given a random PGM model with X
Revised: Suppose we are given a random PGM model with X

Original: = { X k } .
Revised: = { X k } .

Original: Also, assume that we know presence probability { p ij } for all semi-cliques
Revised: Also, assume that we know presence probability { p ij } for all semi-cliques

Original: C X =
Revised: C X =

Original: {D ij } .
Revised: {D ij } .

Original: The ﬁxed point iteration in Theorem 1 for posterior marginal P ( { H k }|{ x k } ) can be embedded in a nonlinear function mapping with embedding vector ˜ µ k as
Revised: The ﬁxed point iteration in Theorem 1 for posterior marginal P ( { H k }|{ x k } ) can be embedded in a nonlinear function mapping with embedding vector ˜ µ k as

Original: [Equation]
Revised: [Equation]

Original: Proof of Thorem 1 and lemma 1 .
Revised: Proof of Thorem 1 and lemma 1 .

Original: For brevity, proofs are relegated to the Appendix D and E.
Revised: For brevity, proofs are relegated to the Appendix D and E.

Original: Corollary 1.
Revised: Corollary 1.

Original: For a robot scheduling problem with set of tasks t i ∈ T t , the random PGM representation for structure2vec in lemma 1 is (( T t , E TTt ) , { p ij } ) where { p ij } denotes the probability of a robot choosing task t i after serving t j .
Revised: For a robot scheduling problem with set of tasks t i ∈ T t , the random PGM representation for structure2vec in lemma 1 is (( T t , E TTt ) , { p ij } ) where { p ij } denotes the probability of a robot choosing task t i after serving t j .

Original: { p ij } inference procedure employed in this paper is as follows.
Revised: { p ij } inference procedure employed in this paper is as follows.

Original: Denote ages of task i, j as age i , age j .
Revised: Denote ages of task i, j as age i , age j .

Original: Note that if we generate M samples of (cid:15) ij as { e kij } Mk =1 , then 1 M (cid:80) Mk =1 f ( e kij , age i , age j ) is an unbiased and consistent estimator of E [ f ( (cid:15) ij , age i , age j )] .
Revised: Note that if we generate M samples of (cid:15) ij as { e kij } Mk =1 , then 1 M (cid:80) Mk =1 f ( e kij , age i , age j ) is an unbiased and consistent estimator of E [ f ( (cid:15) ij , age i , age j )] .

Original: For each sample k , for each task i and task j , we form a vector of u kij =
Revised: For each sample k , for each task i and task j , we form a vector of u kij =

Original: ( e kij , age i , age j ) and compute g ij = (cid:80) Mk =1 1 M W 1 ( relu ( W 2 u kij ) .
Revised: ( e kij , age i , age j ) and compute g ij = (cid:80) Mk =1 1 M W 1 ( relu ( W 2 u kij ) .

Original: We obtain { p ij } from { g ij } using softmax.
Revised: We obtain { p ij } from { g ij } using softmax.

Original: Algorithm details are in Appendix F.
Revised: Algorithm details are in Appendix F.

Original: 3. I NFERENCE OF Q- FUNCTION USING NEW STRUCTURE 2 VEC
Revised: 3. I NFERENCE OF Q- FUNCTION USING NEW STRUCTURE 2 VEC

Original: In this section, we show how Q ( s t , a t ) can be precisely and transeferably inferred using a two-step structure2vec inference method (For theoretical justiﬁcations on hierarchical variational inference, see Ranganath et al.
Revised: In this section, we show how Q ( s t , a t ) can be precisely and transeferably inferred using a two-step structure2vec inference method (For theoretical justiﬁcations on hierarchical variational inference, see Ranganath et al.

Original: We here assume that we are given ( T t , E TTt ) and inferred { p ij } so that
Revised: We here assume that we are given ( T t , E TTt ) and inferred { p ij } so that

Original: Corollary 1 can be applied.
Revised: Corollary 1 can be applied.

Original: For brevity, we illustrate the inference procedure for the special case when task completion time is deterministic (Appendix G illustrates how we can combine random sampling to inference procedure to deal with task completion times as a random variable).
Revised: For brevity, we illustrate the inference procedure for the special case when task completion time is deterministic (Appendix G illustrates how we can combine random sampling to inference procedure to deal with task completion times as a random variable).

Original: Step 1.
Revised: Step 1.

Original: Distance Embedding.
Revised: Distance Embedding.

Original: The output vectors { ˜ µ 1 k } of structure2vec embeds a local graph information around that vector node [Dai et al.
Revised: The output vectors { ˜ µ 1 k } of structure2vec embeds a local graph information around that vector node [Dai et al.

Original: We here focus on embedding information of robot locations around a task node and thus infer each task’s ‘relative graphical distance’ from robots around it.
Revised: We here focus on embedding information of robot locations around a task node and thus infer each task’s ‘relative graphical distance’ from robots around it.

Original: As the input of ﬁrst structure2vec ( { x k } in lemma 1), we only use robot assignment information (if t k is an assigned task, we set x k as ‘task completion time of assignment’; if t k is not an assigned task:, we set x k = 0 ).
Revised: As the input of ﬁrst structure2vec ( { x k } in lemma 1), we only use robot assignment information (if t k is an assigned task, we set x k as ‘task completion time of assignment’; if t k is not an assigned task:, we set x k = 0 ).

Original: This procedure is illustrated in Figure 1.
Revised: This procedure is illustrated in Figure 1.

Original: According to [Dai et al.
Revised: According to [Dai et al.

Original: (2016)], the output vectors { ˜ µ 1 k } of structure2vec will include sufﬁcient information about the relative graphical distance from all robots to each task.
Revised: (2016)], the output vectors { ˜ µ 1 k } of structure2vec will include sufﬁcient information about the relative graphical distance from all robots to each task.

Original: Step 2.
Revised: Step 2.

Original: Value Embedding.
Revised: Value Embedding.

Original: The second step is designed to infer ’How much value is likely in the local graph around each task’.
Revised: The second step is designed to infer ’How much value is likely in the local graph around each task’.

Original: Remind that vectors { ˜ µ 1 k } , output vectors of the ﬁrst step, carries information about the relative graphical distance from all robots to each task.
Revised: Remind that vectors { ˜ µ 1 k } , output vectors of the ﬁrst step, carries information about the relative graphical distance from all robots to each task.

Original: We concatenate ‘age’ of each tasks { age k } to each corresponding vector in { ˜ µ 1 k } and use the resulting graph as an input ( { x k } in lemma 1) of second structure2vec, as illustrated in Figure 1.
Revised: We concatenate ‘age’ of each tasks { age k } to each corresponding vector in { ˜ µ 1 k } and use the resulting graph as an input ( { x k } in lemma 1) of second structure2vec, as illustrated in Figure 1.

Original: Again, vectors { ˜ µ 2 k } of the output graph of second structure2vec operation embeds a local graph structure around each node.
Revised: Again, vectors { ˜ µ 2 k } of the output graph of second structure2vec operation embeds a local graph structure around each node.

Original: Our intuition is that { ˜ µ 2 k } includes sufﬁcient information about ’How much value is likely in the local graph around each task’.
Revised: Our intuition is that { ˜ µ 2 k } includes sufﬁcient information about ’How much value is likely in the local graph around each task’.

Original: Step 3.
Revised: Step 3.

Original: Computing Q ( s t , a t ) .
Revised: Computing Q ( s t , a t ) .

Original: To infer Q ( s t , a t ) , we aggregate the embedding vectors for all nodes, i.e., ˜ µ 2 = (cid:80) k
Revised: To infer Q ( s t , a t ) , we aggregate the embedding vectors for all nodes, i.e., ˜ µ 2 = (cid:80) k

Original: ˜ µ 2 k to get one vector ˜ µ 2 which embeds the ‘value likeliness’ of the global graph.
Revised: ˜ µ 2 k to get one vector ˜ µ 2 which embeds the ‘value likeliness’ of the global graph.

Original: We then use a layer of neural network to map ˜ µ 2 into Q ( s t , a t ) .
Revised: We then use a layer of neural network to map ˜ µ 2 into Q ( s t , a t ) .

Original: The detailed algorithm of above whole procedure (combined with random task completion times) is illustrated in Appendix G.
Revised: The detailed algorithm of above whole procedure (combined with random task completion times) is illustrated in Appendix G.

Original: Why are each inference steps transferable?
Revised: Why are each inference steps transferable?

Original: For the ﬁrst step, it is trivial; the inference problem is a scale-free task.
Revised: For the ﬁrst step, it is trivial; the inference problem is a scale-free task.

Original: In the second step, the ‘value likeliness’ will be underestimated or overestimated according to the ratio of (number of robots / number of tasks) in a local graph: underestimated if the ratio in training environment is smaller than the ratio in the testing environment; overestimated otherwise.
Revised: In the second step, the ‘value likeliness’ will be underestimated or overestimated according to the ratio of (number of robots / number of tasks) in a local graph: underestimated if the ratio in training environment is smaller than the ratio in the testing environment; overestimated otherwise.

Original: The key idea solving this problem is that this over/under-estimation does not matter in Q-function based action decision [van Hasselt et al.
Revised: The key idea solving this problem is that this over/under-estimation does not matter in Q-function based action decision [van Hasselt et al.

Original: (2015)] as long as the order of Q-function value among actions are the same.
Revised: (2015)] as long as the order of Q-function value among actions are the same.

Original: While analytic justiﬁcation of this order invariance is beyond this paper’s scope, the fact that there is no over/underestimation issue in the ﬁrst step inference problem helps this justiﬁcation.
Revised: While analytic justiﬁcation of this order invariance is beyond this paper’s scope, the fact that there is no over/underestimation issue in the ﬁrst step inference problem helps this justiﬁcation.

Original: 3. A CTION S ELECTION U SING H EURISTIC A UCTION
Revised: 3. A CTION S ELECTION U SING H EURISTIC A UCTION

Original: In Q-function based action choice, at each time-step t , we ﬁnd an action with largest Q ( s t , a t ) .
Revised: In Q-function based action choice, at each time-step t , we ﬁnd an action with largest Q ( s t , a t ) .

Original: We call this action choice operation ’max-operation’.
Revised: We call this action choice operation ’max-operation’.

Original: The problem in max-operation in the multi-robot setting is that the number of computation exponentially increases as the number of robots and tasks increases.
Revised: The problem in max-operation in the multi-robot setting is that the number of computation exponentially increases as the number of robots and tasks increases.

Original: In this section, we show that transferability of Q -function inference enables designing an efﬁcient heuristic auction that replaces max operation.
Revised: In this section, we show that transferability of Q -function inference enables designing an efﬁcient heuristic auction that replaces max operation.

Original: We call it auction-based policy(ADP) and denote it as φ Q θ , where Q θ indicates that we compute φ Q θ using current Q θ estimator.
Revised: We call it auction-based policy(ADP) and denote it as φ Q θ , where Q θ indicates that we compute φ Q θ using current Q θ estimator.

Original: At time-step t , a state s t is a graph G t =
Revised: At time-step t , a state s t is a graph G t =

Original: ( R t ∪T t , E t ) as deﬁned in section 2.1.
Revised: ( R t ∪T t , E t ) as deﬁned in section 2.1.

Original: Our ADP , φ Q θ , ﬁnds an action a t (which is a matching in bipartite graph (( R t ∪T t ) , E RTt ) of graph G t ) through iterations between two phases: the bidding phase and the consensus phase.
Revised: Our ADP , φ Q θ , ﬁnds an action a t (which is a matching in bipartite graph (( R t ∪T t ) , E RTt ) of graph G t ) through iterations between two phases: the bidding phase and the consensus phase.

Original: We start with a bidding phase.
Revised: We start with a bidding phase.

Original: All robots initially know the matching determined in previous iterations.
Revised: All robots initially know the matching determined in previous iterations.

Original: We denote this matching as Y , a bipartite subgraph of (( R t ∪T t ) , E RTt ) .
Revised: We denote this matching as Y , a bipartite subgraph of (( R t ∪T t ) , E RTt ) .

Original: When making a bid, a robot r i ignores all other unassigned robots.
Revised: When making a bid, a robot r i ignores all other unassigned robots.

Original: For example, suppose robot r i considers t j for bidding.
Revised: For example, suppose robot r i considers t j for bidding.

Original: For r i , Y ∪ (cid:15) ij is a proper action (according to deﬁnition in section 2.1) in a ‘unassigned robot-ignored’ problem.
Revised: For r i , Y ∪ (cid:15) ij is a proper action (according to deﬁnition in section 2.1) in a ‘unassigned robot-ignored’ problem.

Original: Robot r i thus can compute Q ( s t , Y ∪ (cid:15) r i t j ) of ‘unassigned robot-ignored’ problem for all unassigned task t j .
Revised: Robot r i thus can compute Q ( s t , Y ∪ (cid:15) r i t j ) of ‘unassigned robot-ignored’ problem for all unassigned task t j .

Original: If task t ∗ is with the highest value, robot r i bids { (cid:15) r i t ∗ , Q ( s t , Y ∪ (cid:15) r
Revised: If task t ∗ is with the highest value, robot r i bids { (cid:15) r i t ∗ , Q ( s t , Y ∪ (cid:15) r

Original: i t ∗ ) } to auctioneer.
Revised: i t ∗ ) } to auctioneer.

Original: Since number of robots ignored by r i is different at each iteration, transferability of Q-function inference plays key role.
Revised: Since number of robots ignored by r i is different at each iteration, transferability of Q-function inference plays key role.

Original: The consensus phase is simple.
Revised: The consensus phase is simple.

Original: The auctioneer ﬁnds the bid with the best value, say { (cid:15) ∗ , bid value with (cid:15) ∗ } .
Revised: The auctioneer ﬁnds the bid with the best value, say { (cid:15) ∗ , bid value with (cid:15) ∗ } .

Original: Then auctioneer updates everyone’s
Revised: Then auctioneer updates everyone’s

Original: Y as Y ∪ { (cid:15) ∗ } .
Revised: Y as Y ∪ { (cid:15) ∗ } .

Original: These bidding and consensus phases are iterated until we can’t add an edge to Y anymore.
Revised: These bidding and consensus phases are iterated until we can’t add an edge to Y anymore.

Original: Then the central decision maker chooses Y as φ Q θ ( s k ) .
Revised: Then the central decision maker chooses Y as φ Q θ ( s k ) .

Original: One can easily verify that the computational complexity of computing φ Q θ is O ( | L R | | L T | ) , which is only polynomial.
Revised: One can easily verify that the computational complexity of computing φ Q θ is O ( | L R | | L T | ) , which is only polynomial.

Original: While theoretical performance guarantee of this heuristic auction is out of this paper’s scope, in section 5 we show that empirically this heuristic achieves near-optimal performance.
Revised: While theoretical performance guarantee of this heuristic auction is out of this paper’s scope, in section 5 we show that empirically this heuristic achieves near-optimal performance.

Original:  L EARNING ALGORITHM 4. A UCTION - FITTED Q- ITERATION FRAMEWORK
Revised:  L EARNING ALGORITHM 4. A UCTION - FITTED Q- ITERATION FRAMEWORK

Original: In ﬁtted Q-iteration, we ﬁt θ of Q θ ( s t , a t ) with stored data using Bellman optimality equation.
Revised: In ﬁtted Q-iteration, we ﬁt θ of Q θ ( s t , a t ) with stored data using Bellman optimality equation.

Original: That is, chooses θ that makes E (cid:2) Q
Revised: That is, chooses θ that makes E (cid:2) Q

Original: θ ( s k , a k ) −
Revised: θ ( s k , a k ) −

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: small.
Revised: small.

Original: Note that every update of θ needs at least one max-operation.
Revised: Note that every update of θ needs at least one max-operation.

Original: To solve this issue, we suggest a learning framework we call auction-ﬁtted Q -iteration.
Revised: To solve this issue, we suggest a learning framework we call auction-ﬁtted Q -iteration.

Original: What we do is simple: when we update θ , we use auction-based policy(ADP) deﬁned in section 3.3 instead of max-operation.
Revised: What we do is simple: when we update θ , we use auction-based policy(ADP) deﬁned in section 3.3 instead of max-operation.

Original: That is, we seek the parameter θ that minimizes
Revised: That is, we seek the parameter θ that minimizes

Original: E (cid:2)
Revised: E (cid:2)

Original: [Equation]
Revised: [Equation]

Original: 4. E XPLORATION FOR A UCTION
Revised: 4. E XPLORATION FOR A UCTION

Original: How can we conduct exploration in Auction-ﬁtted Q-iteration framework?
Revised: How can we conduct exploration in Auction-ﬁtted Q-iteration framework?

Original: Unfortunately, we can’t use (cid:15) -greedy method since such randomly altered assignment is very likely to cause a catastrophic result in problems with combinatorial nature.
Revised: Unfortunately, we can’t use (cid:15) -greedy method since such randomly altered assignment is very likely to cause a catastrophic result in problems with combinatorial nature.

Original: In this paper, we suggest that parameter space exploration
Revised: In this paper, we suggest that parameter space exploration

Original: [Plappert et al.
Revised: [Plappert et al.

Original: (2017)] can be applied.
Revised: (2017)] can be applied.

Original: Recall that we use Q θ ( s k , a k ) to get policy
Revised: Recall that we use Q θ ( s k , a k ) to get policy

Original: φ Q θ ( s k ) .
Revised: φ Q θ ( s k ) .

Original: Note that θ denotes all neural network parameters used in the structure2vec iterations introduced in Section 5.
Revised: Note that θ denotes all neural network parameters used in the structure2vec iterations introduced in Section 5.

Original: Since Q θ ( s k , a k ) is parametrized by θ , exploration with φ Q θ ( s k ) can be performed by exploration with parameter θ .
Revised: Since Q θ ( s k , a k ) is parametrized by θ , exploration with φ Q θ ( s k ) can be performed by exploration with parameter θ .

Original: Such exploration in parameter space has been introduced in the policy gradient RL literature.
Revised: Such exploration in parameter space has been introduced in the policy gradient RL literature.

Original: While this method was originally developed for policy gradient based methods, exploration in parameter space can be particularly useful in auction-ﬁtted Q-iteration.
Revised: While this method was originally developed for policy gradient based methods, exploration in parameter space can be particularly useful in auction-ﬁtted Q-iteration.

Original: The detailed application is as follows.
Revised: The detailed application is as follows.

Original: When conducting exploration, apply a random perturbation on the neural network parameters θ in structure2vec.
Revised: When conducting exploration, apply a random perturbation on the neural network parameters θ in structure2vec.

Original: The resulting a perturbation in the Q-function used for decision making via the auction-based policy φ Q θ ( s k ) throughout that problem.
Revised: The resulting a perturbation in the Q-function used for decision making via the auction-based policy φ Q θ ( s k ) throughout that problem.

Original: Similarly, when conducting exploitation, the current surrogate Q-function is used throughout the problem.
Revised: Similarly, when conducting exploitation, the current surrogate Q-function is used throughout the problem.

Original: Updates for the surrogate Q-function may only occur after each problem is complete (and typically after a group of problems).
Revised: Updates for the surrogate Q-function may only occur after each problem is complete (and typically after a group of problems).

Original:  E XPERIMENT 5. MRRC
Revised:  E XPERIMENT 5. MRRC

Original: For MRRC, we conduct a simulation experiment for a discrete time, discrete state environment.
Revised: For MRRC, we conduct a simulation experiment for a discrete time, discrete state environment.

Original: We use maze (see Figure 1) generator of UC Berkeley CS188 Pacman project [Neller et al.
Revised: We use maze (see Figure 1) generator of UC Berkeley CS188 Pacman project [Neller et al.

Original: (2010)] to generate large size mazes.
Revised: (2010)] to generate large size mazes.

Original: We generated a new maze for every training and testing experiments.
Revised: We generated a new maze for every training and testing experiments.

Original: Under the deterministic environment, the robot succeeds its movement 100%.
Revised: Under the deterministic environment, the robot succeeds its movement 100%.

Original: Under stochastic environment, a robot succeeds its intended movement in 55% on the grid with dots and for every other direction 15% each; on the grid without dots, the rates are 70% and 10%.
Revised: Under stochastic environment, a robot succeeds its intended movement in 55% on the grid with dots and for every other direction 15% each; on the grid without dots, the rates are 70% and 10%.

Original: As described in section 2, routing problems are already solved.
Revised: As described in section 2, routing problems are already solved.

Original: That is, each robot knows how to optimally (in expectation) reach a task.
Revised: That is, each robot knows how to optimally (in expectation) reach a task.

Original: To ﬁnd an optimal routing policy, we use Dijkstra’s algorithm for deterministic environments and dynamic programming for stochastic environments.
Revised: To ﬁnd an optimal routing policy, we use Dijkstra’s algorithm for deterministic environments and dynamic programming for stochastic environments.

Original: The central assignment decision maker has enough samples of task completion time for every possible route.
Revised: The central assignment decision maker has enough samples of task completion time for every possible route.

Original: We consider two reward rules: Linearly decaying rewards obey f ( age ) = 200 − age until reaching 0, where age is the task age when served; For nonlinearly decaying rewards, f ( t ) =
Revised: We consider two reward rules: Linearly decaying rewards obey f ( age ) = 200 − age until reaching 0, where age is the task age when served; For nonlinearly decaying rewards, f ( t ) =

Original: λ t for λ = 0.99.
Revised: λ t for λ = 0.99.

Original: Initial age of tasks were uniformly distributed in the interval [0 , 100] .
Revised: Initial age of tasks were uniformly distributed in the interval [0 , 100] .

Original: Performance test.
Revised: Performance test.

Original: We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.
Revised: We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards.

Original: There are three baselines used for performance test: exact baseline, heuristic baseline, and indirect baseline.
Revised: There are three baselines used for performance test: exact baseline, heuristic baseline, and indirect baseline.

Original: For the experiment with deterministic with linearly decaying rewards, an exact optimal solution for mixed-integer exists and can be used as a baseline.
Revised: For the experiment with deterministic with linearly decaying rewards, an exact optimal solution for mixed-integer exists and can be used as a baseline.

Original: We solve this program using Gurobi with 60-min cut to get the baseline.
Revised: We solve this program using Gurobi with 60-min cut to get the baseline.

Original: We also implemented the most up-to-date heuristic for MRRC in [Ekici & Retharekar (2013)].
Revised: We also implemented the most up-to-date heuristic for MRRC in [Ekici & Retharekar (2013)].

Original: For any other experiments with nonlinearly decaying rewards or stochastic environment, such an exact optimal solution or other heuristics methods does not exist.
Revised: For any other experiments with nonlinearly decaying rewards or stochastic environment, such an exact optimal solution or other heuristics methods does not exist.

Original: In these cases, we should be conservative when talking about performance.
Revised: In these cases, we should be conservative when talking about performance.

Original: Our strategy is to construct a indirect baseline using a universally applicable algorithm called Sequential greedy algorithm (SGA) [Han-Lim Choi et al.
Revised: Our strategy is to construct a indirect baseline using a universally applicable algorithm called Sequential greedy algorithm (SGA) [Han-Lim Choi et al.

Original: SGA is a polynomial-time task allocation algorithm that shows decent scalable performance to both linear and non-linear rewards.
Revised: SGA is a polynomial-time task allocation algorithm that shows decent scalable performance to both linear and non-linear rewards.

Original: For stochastic environments, we use mean task completion time for task allocation and re-allocate the whole tasks at every timesteps.
Revised: For stochastic environments, we use mean task completion time for task allocation and re-allocate the whole tasks at every timesteps.

Original: We construct our indirect baseline as ‘ratio between our method and SGA for experiments with deterministic-linearly decaying rewards’.
Revised: We construct our indirect baseline as ‘ratio between our method and SGA for experiments with deterministic-linearly decaying rewards’.

Original: Showing that this ratio is maintained for stochastic environments in both linear/nonlinear rewards sufﬁces our purpose.
Revised: Showing that this ratio is maintained for stochastic environments in both linear/nonlinear rewards sufﬁces our purpose.

Original: Table 1 shows experiment results for (# of robots, # of tasks)
Revised: Table 1 shows experiment results for (# of robots, # of tasks)

Original: =
Revised: =

Original: For linear/deterministic rewards, our proposed method achieves nearoptimality (all above 95% optimality).
Revised: For linear/deterministic rewards, our proposed method achieves nearoptimality (all above 95% optimality).

Original: While there is no exact or comparable performance baseline for experiments under other environments, indirect baseline (%SGA) at least shows that our method does not lose %SGA for stochastic environments compared with %SGA for deterministic environments in both linear and nonlinear rewards.
Revised: While there is no exact or comparable performance baseline for experiments under other environments, indirect baseline (%SGA) at least shows that our method does not lose %SGA for stochastic environments compared with %SGA for deterministic environments in both linear and nonlinear rewards.

Original: Scalability test.
Revised: Scalability test.

Original: We count the training requirements for 93% optimality for seven problem sizes (# of robots N R , # of tasks N T ) =
Revised: We count the training requirements for 93% optimality for seven problem sizes (# of robots N R , # of tasks N T ) =

Original: (2 , 20) , (3 , 20) , (5 , 30) , (5 , 40) , (8 , 40) , (8 , 50) with deterministic/linearly decaying rewards (we can compare optimality only in this case).
Revised: (2 , 20) , (3 , 20) , (5 , 30) , (5 , 40) , (8 , 40) , (8 , 50) with deterministic/linearly decaying rewards (we can compare optimality only in this case).

Original: As we can see in Table 2, the training requirement shown not to scale as problem size increases.
Revised: As we can see in Table 2, the training requirement shown not to scale as problem size increases.

Original: Transferability test .
Revised: Transferability test .

Original: Suppose that we trained our learning algorithm with problems of three robots and 30 Tasks.
Revised: Suppose that we trained our learning algorithm with problems of three robots and 30 Tasks.

Original: We can claim transferability of our algorithm if our algorithm achieves similar performance for testing with problems of 8 robots and 50 tasks when compared with the algorithm speciﬁcally trained with problems of 8 robots and 50 tasks, the same size as testing.
Revised: We can claim transferability of our algorithm if our algorithm achieves similar performance for testing with problems of 8 robots and 50 tasks when compared with the algorithm speciﬁcally trained with problems of 8 robots and 50 tasks, the same size as testing.

Original: Table 3 shows our comprehensive experiment to test transeferability.
Revised: Table 3 shows our comprehensive experiment to test transeferability.

Original: The results in the diagonals (where training size and testing size is the same) becomes a baseline, and we can compare how the networks trained with different problem size did well compare to those results.
Revised: The results in the diagonals (where training size and testing size is the same) becomes a baseline, and we can compare how the networks trained with different problem size did well compare to those results.

Original: We could see that lower-direction transfer tests (trained with larger size problem and tested with smaller size problems) shows only a
Revised: We could see that lower-direction transfer tests (trained with larger size problem and tested with smaller size problems) shows only a

Original: [Table] [Table]
Revised: [Table] [Table]

Original: small loss in performance.
Revised: small loss in performance.

Original: For upper-direction transfer tests (trained with smaller size problem and tested with larger size problem),  performance loss was up 4 percent.
Revised: For upper-direction transfer tests (trained with smaller size problem and tested with larger size problem), the performance loss was up 4 percent.

Original: Ablation study.
Revised: Ablation study.

Original: There are three components in our proposed method: 1) a careful encoding of information using two-layers of structure2vec, 2) new structure2vec equation with random PGM and 3) an auction-based assignment.
Revised: There are three components in our proposed method: 1) a careful encoding of information using two-layers of structure2vec, 2) new structure2vec equation with random PGM and 3) an auction-based assignment.

Original: Each component was removed from the full method and tested to check the necessity of the component.
Revised: Each component was removed from the full method and tested to check the necessity of the component.

Original: We test the performance in a deterministic/linearly decaying rewards (so that there is an optimal solution available for comparison).
Revised: We test the performance in a deterministic/linearly decaying rewards (so that there is an optimal solution available for comparison).

Original: The experimental results are shown in Figure 2.
Revised: The experimental results are shown in Figure 2.

Original: While the full method requires more training steps, only the full method achieves near-optimal performance.
Revised: While the full method requires more training steps, only the full method achieves near-optimal performance.

Original: 5. IPMS
Revised: 5. IPMS

Original: For IPMS, we test it with continuous time, continuous state environment.
Revised: For IPMS, we test it with continuous time, continuous state environment.

Original: While there have been many learning-based methods proposed for (single) robot scheduling problems, to the best our knowledge our method is the ﬁrst learning method to claim scalable performance among machinescheduling problems.
Revised: While there have been many learning-based methods proposed for (single) robot scheduling problems, to the best our knowledge our method is the ﬁrst learning method to claim scalable performance among machinescheduling problems.

Original: Hence, in this case, we focus on showing comparable performance for large problems, instead of attempting to show the superiority of our method compared with heuristics speciﬁcally designed for IPMS (actually no heuristic was speciﬁcally designed to solve our exact problem (makespan minimization, sequence-dependent setup with no restriction on setup times))
Revised: Hence, in this case, we focus on showing comparable performance for large problems, instead of attempting to show the superiority of our method compared with heuristics speciﬁcally designed for IPMS (actually no heuristic was speciﬁcally designed to solve our exact problem (makespan minimization, sequence-dependent setup with no restriction on setup times))

Original: For each task, processing times is determined using uniform [16, 64].
Revised: For each task, processing times is determined using uniform [16, 64].

Original: For every (task i , task j ) ordered pair, a unique setup time is determined using uniform [0, 32].
Revised: For every (task i , task j ) ordered pair, a unique setup time is determined using uniform [0, 32].

Original: As illustrated in section 2, we want to minimize make-span.
Revised: As illustrated in section 2, we want to minimize make-span.

Original: As a benchmark for IPMS, we use Google OR-Tools library
Revised: As a benchmark for IPMS, we use Google OR-Tools library

Original: [Table]
Revised: [Table]

Original: 5R
Revised: 8R

Original: [Table]
Revised: [Table]

Original: 8R
Revised: 8R

Original: [Table]
Revised: [Table]

Original: 8R

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Google (2012).
Revised: Google (2012).

Original: This library provides metaheuristics such as Greedy Descent, Guided Local Search,
Revised: This library provides metaheuristics such as Greedy Descent, Guided Local Search,

Original: Simulated Annealing, Tabu Search.
Revised: Simulated Annealing, Tabu Search.

Original: We compare our algorithm’s result with the heuristic with the best result for each experiment.
Revised: We compare our algorithm’s result with the heuristic with the best result for each experiment.

Original: We consider cases with 3 , 5 , 7 , 10 machines and 50 , 75 , 100 jobs.
Revised: We consider cases with 3 , 5 , 7 , 10 machines and 50 , 75 , 100 jobs.

Original: The results are provided in Table 4.
Revised: The results are provided in Table 4.

Original: Makespan obtained by our method divided by the makespan obtained in the baseline is provided.
Revised: Makespan obtained by our method divided by the makespan obtained in the baseline is provided.

Original: Although our method has limitations in problems with a small number of tasks, it shows comparable performance to a large number of tasks and shows its value as the ﬁrst learning-based machine scheduling method that achieves scalable performance.
Revised: Although our method has limitations in problems with a small number of tasks, it shows comparable performance to a large number of tasks and shows its value as the ﬁrst learning-based machine scheduling method that achieves scalable performance.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We presented a learning-based method that achieves the ﬁrst success for multi-robot/machine scheduling problems in both challenges: scalable performance and tranferability.
Revised: We presented a learning-based method that achieves the ﬁrst success for multi-robot/machine scheduling problems in both challenges: scalable performance and tranferability.

Original: We identiﬁed that robot scheduling problems have an exact representation as random PGM.
Revised: We identiﬁed that robot scheduling problems have an exact representation as random PGM.

Original: We developed a meanﬁeld inference theory for random PGM and extended structure2vec method of Dai et al.
Revised: We developed a meanﬁeld inference theory for random PGM and extended structure2vec method of Dai et al.

Original: To overcome the limitations of ﬁtted Q-iteration, a heuristic auction that was enabled by transferability is suggested.
Revised: To overcome the limitations of ﬁtted Q-iteration, a heuristic auction that was enabled by transferability is suggested.

Original: Through experimental evaluation, we demonstrate our method’s success for
Revised: Through experimental evaluation, we demonstrate our method’s success for

Original: MRRC problems under a deterministic/stochastic environment.
Revised: MRRC problems under a deterministic/stochastic environment.

Original: Our method also claims to be the ﬁrst learning-based algorithm that achieves scalable performance among machine scheduling algorithms; our method achieves a comparable performance in a scalable manner.
Revised: Our method also claims to be the ﬁrst learning-based algorithm that achieves scalable performance among machine scheduling algorithms; our method achieves a comparable performance in a scalable manner.

Original: Our method for MRRC problems can be easily extended to ride-sharing problems or package delivery problems.
Revised: Our method for MRRC problems can be easily extended to ride-sharing problems or package delivery problems.

Original: Given a set of all user requests to serve, those problems can be formulated as a
Revised: Given a set of all user requests to serve, those problems can be formulated as a

Original: MRRC problem.
Revised: MRRC problem.

Original: For both ride-sharing and package delivery, it is reasonable to assume that the utility of a user depends on when she is completely serviced.
Revised: For both ride-sharing and package delivery, it is reasonable to assume that the utility of a user depends on when she is completely serviced.

Original: We can model how the utility of a user decreases over time since when it appears and set the objective function of problems as maximizing total collected user utility.
Revised: We can model how the utility of a user decreases over time since when it appears and set the objective function of problems as maximizing total collected user utility.

Original: Now consider a task ‘deliver user (or package) from A to B’.
Revised: Now consider a task ‘deliver user (or package) from A to B’.

Original: This is actually a task ”Move to location A and then move to location B”.
Revised: This is actually a task ”Move to location A and then move to location B”.

Original: If we know the completion time distribution of each move (as we did for MRRC), the task completion time is simply the sum of two random variables corresponding to task completion time distribution of the moves in the task.
Revised: If we know the completion time distribution of each move (as we did for MRRC), the task completion time is simply the sum of two random variables corresponding to task completion time distribution of the moves in the task.

Original: Indeed, ride-sharing or package delivery problems are of such tasks (We can ignore charging moves for simplicity, and also we don’t have to consider simple relocation of vehicles or robots since we don’t consider random customer arrivals).
Revised: Indeed, ride-sharing or package delivery problems are of such tasks (We can ignore charging moves for simplicity, and also we don’t have to consider simple relocation of vehicles or robots since we don’t consider random customer arrivals).

Original: Therefore, both ride-sharing problems and package delivery problems can be formulated as MRRC problems.
Revised: Therefore, both ride-sharing problems and package delivery problems can be formulated as MRRC problems.

Original: A MRRC WITH CONTINUOUS STATE / CONTINUOUS TIME SPACE FORMULATION , OR WITH SETUP TIME AND PROCESSING TIME
Revised: A MRRC WITH CONTINUOUS STATE / CONTINUOUS TIME SPACE FORMULATION , OR WITH SETUP TIME AND PROCESSING TIME

Original: In continuous state/continuous time space formulation, the initial location and ending location of robots and tasks are arbitrary on R 2 .
Revised: In continuous state/continuous time space formulation, the initial location and ending location of robots and tasks are arbitrary on R 2 .

Original: At every moment at least a robot ﬁnishes a task, we make assignment decision for a free robot(s).
Revised: At every moment at least a robot ﬁnishes a task, we make assignment decision for a free robot(s).

Original: We call this moments as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .
Revised: We call this moments as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .

Original: , t k , . . . ) .
Revised: , t k , . . . ) .

Original: Abusing this notation slightly, we use ( · ) t k = ( · ) k .
Revised: Abusing this notation slightly, we use ( · ) t k = ( · ) k .

Original: Task completion time can consist of three components: travel time, setup time and processing time.
Revised: Task completion time can consist of three components: travel time, setup time and processing time.

Original: While a robot in the travel phase or setup phase may be reassigned to other tasks, we can’t reassign a robot in the processing phase.
Revised: While a robot in the travel phase or setup phase may be reassigned to other tasks, we can’t reassign a robot in the processing phase.

Original: Under these assumptions, at each decision epoch robot r i is given a set of tasks it can assign itself: if it is in the traveling phase or setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.
Revised: Under these assumptions, at each decision epoch robot r i is given a set of tasks it can assign itself: if it is in the traveling phase or setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.

Original: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:
Revised: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:

Original: State.
Revised: State.

Original: State s k at decision epoch
Revised: State s k at decision epoch

Original: k is a directed graph G k
Revised: k is a directed graph G k

Original: =
Revised: =

Original: ( R k ∪ T k , E k ) :
Revised: ( R k ∪ T k , E k ) :

Original: R k is the set of all robots and T k is the set of all tasks;
Revised: R k is the set of all robots and T k is the set of all tasks;

Original: The set of directed edges E k
Revised: The set of directed edges E k

Original: =
Revised: =

Original: E RTk ∪E TTk where a directed edge (cid:15) r i t
Revised: E RTk ∪E TTk where a directed edge (cid:15) r i t

Original: j ∈ E RTk is a random variable which denotes task completion time of robot i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a robot which just ﬁnished serving task i in T k to service task j in T k .
Revised: j ∈ E RTk is a random variable which denotes task completion time of robot i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a robot which just ﬁnished serving task i in T k to service task j in T k .

Original: E RTk contains information about each robot’s possible assignments: E RTk =
Revised: E RTk contains information about each robot’s possible assignments: E RTk =

Original: ∪ i E r i k , where E r i t is a singleton set if robot
Revised: ∪ i E r i k , where E r i t is a singleton set if robot

Original: i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from robot r
Revised: i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from robot r

Original: i to remaining tasks that are not in the processing phase.
Revised: i to remaining tasks that are not in the processing phase.

Original: Action.
Revised: Action.

Original: The action a k at decision epoch k is the joint assignment of robots given the current state s k = G k .
Revised: The action a k at decision epoch k is the joint assignment of robots given the current state s k = G k .

Original: The feasible action should satisfy the two constraints:
Revised: The feasible action should satisfy the two constraints:

Original: No two robots can be assigned to a task; some robots may not be assigned when number of robots are more than remaining tasks.
Revised: No two robots can be assigned to a task; some robots may not be assigned when number of robots are more than remaining tasks.

Original: To best address those restrictions, we deﬁne an action a k at time t as a maximal bipartite matching in bipartite sub-graph (( R k ∪ T k ) , E RTk ) of graph G k .
Revised: To best address those restrictions, we deﬁne an action a k at time t as a maximal bipartite matching in bipartite sub-graph (( R k ∪ T k ) , E RTk ) of graph G k .

Original: For example, robot i in R k is matched with task j in T k in an action a k if we assign robot i to task j at decision epoch t .
Revised: For example, robot i in R k is matched with task j in T k in an action a k if we assign robot i to task j at decision epoch t .

Original: We denote the set of all possible actions at epoch k as A k .
Revised: We denote the set of all possible actions at epoch k as A k .

Original: Reward.
Revised: Reward.

Original: In MRRC, Each task has an arbitrarily determined initial age.
Revised: In MRRC, Each task has an arbitrarily determined initial age.

Original: At each decision epoch, the age of each task increases by one.
Revised: At each decision epoch, the age of each task increases by one.

Original: When a task is serviced, a reward is determined only by its age when serviced.
Revised: When a task is serviced, a reward is determined only by its age when serviced.

Original: Denote this reward rule as R ( k ) .
Revised: Denote this reward rule as R ( k ) .

Original: One can easily see that whether a task is served at epoch k is completely determined by s k , a k and s k
Revised: One can easily see that whether a task is served at epoch k is completely determined by s k , a k and s k

Original: +1 .
Revised: +1 .

Original: Therefore, we can denote the reward we get with s k , a k and s k
Revised: Therefore, we can denote the reward we get with s k , a k and s k

Original: +

Revised: +

Original: as R ( s k , a k , s k +1 ) .
Revised: as R ( s k , a k , s k +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .

Original: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that
Revised: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that

Original: [Equation]
Revised: [Equation]

Original: B I DENTICAL PARALLEL MACHINE SCHEDULING PROBLEM FORMULATION
Revised: B I DENTICAL PARALLEL MACHINE SCHEDULING PROBLEM FORMULATION

Original: As written in 2.2, IPMS is a problem deﬁned in continuous state/continuous time space.
Revised: As written in 2.2, IPMS is a problem deﬁned in continuous state/continuous time space.

Original: Machines are all identical, but processing times of tasks are all different.
Revised: Machines are all identical, but processing times of tasks are all different.

Original: In this paper, we discuss IPMS with ‘sequence-dependent setup time’.
Revised: In this paper, we discuss IPMS with ‘sequence-dependent setup time’.

Original: A machine’s setup time required for servicing a task i is determined by its previously served task j .
Revised: A machine’s setup time required for servicing a task i is determined by its previously served task j .

Original: In this case, the task completion time is the sum of setup time and processing time.
Revised: In this case, the task completion time is the sum of setup time and processing time.

Original: Under this setting, we solve IPMS problem for make-span minimization objective discussed in [Kurz et al.
Revised: Under this setting, we solve IPMS problem for make-span minimization objective discussed in [Kurz et al.

Original: (The constraints are different in this problem though);
Revised: (The constraints are different in this problem though);

Original: That is, minimizing total time spent from start to end to ﬁnish all tasks.
Revised: That is, minimizing total time spent from start to end to ﬁnish all tasks.

Original: Every time there is a ﬁnished task, we make assignment decision for a free machine.
Revised: Every time there is a ﬁnished task, we make assignment decision for a free machine.

Original: We call this times as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .
Revised: We call this times as ‘decision epochs’ and express them as an ordered set ( t 1 , t 2 , . . .

Original: , t k , . . . ) .
Revised: , t k , . . . ) .

Original: Abusing this notation slightly, we use ( · ) t k = ( · ) k .
Revised: Abusing this notation slightly, we use ( · ) t k = ( · ) k .

Original: Task completion time for a machine to a task consists of two components: processing time and setup time.
Revised: Task completion time for a machine to a task consists of two components: processing time and setup time.

Original: While a machine in setup phase may be reassigned to another task, we can’t reassign a machine in the processing phase.
Revised: While a machine in setup phase may be reassigned to another task, we can’t reassign a machine in the processing phase.

Original: Under these assumptions, at each epoch, a machine r i is given a set of tasks it can assign: if it is in the setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.
Revised: Under these assumptions, at each epoch, a machine r i is given a set of tasks it can assign: if it is in the setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unﬁnished task.

Original: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:
Revised: This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are deﬁned as follows:

Original: State.
Revised: State.

Original: State s k at decision epoch
Revised: State s k at decision epoch

Original: k is a directed graph G k
Revised: k is a directed graph G k

Original: =
Revised: =

Original: ( R k ∪ T k , E k ) :
Revised: ( R k ∪ T k , E k ) :

Original: R k is the set of all machines and T k is the set of all tasks;
Revised: R k is the set of all machines and T k is the set of all tasks;

Original: The set of directed edges E k
Revised: The set of directed edges E k

Original: =
Revised: =

Original: E RTk ∪ E TTk where a directed edge (cid:15) r i t j ∈ E RTk is a random variable which denotes task completion time of machine i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a machine which just ﬁnished serving task i in T k to service task j in T k .
Revised: E RTk ∪ E TTk where a directed edge (cid:15) r i t j ∈ E RTk is a random variable which denotes task completion time of machine i in R k to service task j in T k and a directed edge (cid:15) t i t j ∈ E TTk denotes a task completion time of a machine which just ﬁnished serving task i in T k to service task j in T k .

Original: E RTk contains information about each robot’s possible assignments: E RTk =
Revised: E RTk contains information about each robot’s possible assignments: E RTk =

Original: ∪ i E r i k , where E r i k is a singleton set if machine i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from machine r i to remaining tasks that are not in the processing phase.
Revised: ∪ i E r i k , where E r i k is a singleton set if machine i is in the processing phase and it must be assigned to its unﬁnished task, and otherwise it is the set of possible assignments from machine r i to remaining tasks that are not in the processing phase.

Original: Action.
Revised: Action.

Original: Deﬁned the same as MRRC with continuous state/time space.
Revised: Deﬁned the same as MRRC with continuous state/time space.

Original: Reward.
Revised: Reward.

Original: In IPMS, time passes between decision epoch t and decision epoch t +1 .
Revised: In IPMS, time passes between decision epoch t and decision epoch t +1 .

Original: Denote this time as T t .
Revised: Denote this time as T t .

Original: One can easily see that T t is completely determined by s k , a k and s k
Revised: One can easily see that T t is completely determined by s k , a k and s k

Original: +1 .
Revised: +1 .

Original: Therefore, we can denote the reward we get with s k , a k and s k +1 as T ( s k , a k , s k +1 ) .
Revised: Therefore, we can denote the reward we get with s k , a k and s k +1 as T ( s k , a k , s k +1 ) .

Original: Objective.
Revised: Objective.

Original: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .
Revised: We can now deﬁne an assignment policy φ as a function that maps a state s k to action a k .

Original: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that
Revised: Given s 0 initial state, an MRRC problem can be expressed as a problem of ﬁnding an optimal assignment policy φ ∗ such that

Original: [Equation]
Revised: [Equation]

Original: C B AYESIAN N ETWORK REPRESENTATION
Revised: C B AYESIAN N ETWORK REPRESENTATION

Original: Here we analytically show that robot scheduling problem randomly induces a random Bayesian
Revised: Here we analytically show that robot scheduling problem randomly induces a random Bayesian

Original: Network from state s t .
Revised: Network from state s t .

Original: Given starting state s t and action a t , a person can repeat a random experiment of “sequential decision making using policy φ ”.
Revised: Given starting state s t and action a t , a person can repeat a random experiment of “sequential decision making using policy φ ”.

Original: In this random experiment, we can deﬁne events ‘How robots serve all remaining tasks in which sequence’.
Revised: In this random experiment, we can deﬁne events ‘How robots serve all remaining tasks in which sequence’.

Original: We call such an event a ‘scenario’.
Revised: We call such an event a ‘scenario’.

Original: For example, suppose that at time-step t we are given robots { A, B } , tasks { 1 , 2 , 3 , 4 , 5 } , and policy φ .
Revised: For example, suppose that at time-step t we are given robots { A, B } , tasks { 1 , 2 , 3 , 4 , 5 } , and policy φ .

Original: One possible scenario S ∗ can be { robot A serves task 3 → 1 → 2 and robot B serves task 5 → 4 } .
Revised: One possible scenario S ∗ can be { robot A serves task 3 → 1 → 2 and robot B serves task 5 → 4 } .

Original: Deﬁne random variable
Revised: Deﬁne random variable

Original: X k a task characteristic, e.g. ‘The time when task k is serviced’.
Revised: X k a task characteristic, e.g. ‘The time when task k is serviced’.

Original: The question is, ‘Given a scenario S ∗ , what is the relationship among random variables { X k } ’?
Revised: The question is, ‘Given a scenario S ∗ , what is the relationship among random variables { X k } ’?

Original: Recall that in our sequential decision making formulation we are given all the ‘task completion time’ information in the s t description.
Revised: Recall that in our sequential decision making formulation we are given all the ‘task completion time’ information in the s t description.

Original: Note that, task completion time is only dependent on the previous task and assigned task.
Revised: Note that, task completion time is only dependent on the previous task and assigned task.

Original: In our example above, under scenario S ∗ ‘when task 2 is served’ is only dependent on ‘when task 1 is served’.
Revised: In our example above, under scenario S ∗ ‘when task 2 is served’ is only dependent on ‘when task 1 is served’.

Original: That is, P ( X 2 | X 1 , X 3 , S ∗ ) =
Revised: That is, P ( X 2 | X 1 , X 3 , S ∗ ) =

Original: P ( X 2 | X 1 , S ∗ ) .
Revised: P ( X 2 | X 1 , S ∗ ) .

Original: This relationship is called ‘conditional independence’.
Revised: This relationship is called ‘conditional independence’.

Original: Given a scenario S ∗ , every relationship among { X i | S ∗ } can be expressed using this kind of relationship among random variables.
Revised: Given a scenario S ∗ , every relationship among { X i | S ∗ } can be expressed using this kind of relationship among random variables.

Original: A graph with this special relationship is called ‘Bayesian Network’
Revised: A graph with this special relationship is called ‘Bayesian Network’

Original: [Koller & Friedman (2009)], a probabilistic graphical model.
Revised: [Koller & Friedman (2009)], a probabilistic graphical model.

Original: D P ROOF OF T HEOREM 1.
Revised: D P ROOF OF T HEOREM 1.

Original: We ﬁrst deﬁne necessary deﬁnitions for our proof.
Revised: We ﬁrst deﬁne necessary deﬁnitions for our proof.

Original: In a random PGM, a PGM is chosen among all possible PGMs on { X k } and semi-cliques C .
Revised: In a random PGM, a PGM is chosen among all possible PGMs on { X k } and semi-cliques C .

Original: Denote the set of all possible factorization as
Revised: Denote the set of all possible factorization as

Original: F = {S 1 , S 2 , ..., S N } where a factorization with index k is denoted as S k ⊆ C .
Revised: F = {S 1 , S 2 , ..., S N } where a factorization with index k is denoted as S k ⊆ C .

Original: Suppose we are given P ( {S = S m } ) .
Revised: Suppose we are given P ( {S = S m } ) .

Original: For each semi-clique D i in C , deﬁne a binary random variable V
Revised: For each semi-clique D i in C , deﬁne a binary random variable V

Original: i : F (cid:55)→ { 0 , 1 } with valuefor the factorization that does not include semi-clique D i and value 1 for the factorization that include semi-clique D i .
Revised: i : F (cid:55)→ { 0 , 1 } with valuefor the factorization that does not include semi-clique D i and value 1 for the factorization that include semi-clique D i .

Original: Let V be a random vector V
Revised: Let V be a random vector V

Original: = (cid:0) V 1 , V 2 , . .
Revised: = (cid:0) V 1 , V 2 , . .

Original: . , V | C | (cid:1) .
Revised: . , V | C | (cid:1) .

Original: Then we can express
Revised: Then we can express

Original: P ( X 1 , . . .
Revised: P ( X 1 , . . .

Original: , X n | V ) ∝ (cid:81) | C | i =1 (cid:2)
Revised: , X n | V ) ∝ (cid:81) | C | i =1 (cid:2)

Original: φ i (cid:0) D i (cid:1)(cid:3) V i .
Revised: φ i (cid:0) D i (cid:1)(cid:3) V i .

Original: We denote (cid:2)
Revised: We denote (cid:2)

Original: φ i (cid:0) D i (cid:1)(cid:3) V i as ψ ( D i ) .
Revised: φ i (cid:0) D i (cid:1)(cid:3) V i as ψ ( D i ) .

Original: Given { P ( {S = S m } ) } , each semi-clique D
Revised: Given { P ( {S = S m } ) } , each semi-clique D

Original: i ’s presence probability p
Revised: i ’s presence probability p

Original: i can be simply calculated; clique i ’s presence probability p
Revised: i can be simply calculated; clique i ’s presence probability p

Original: i is simply the sum of probabilities of all factorizations which include clique i , that is, p i = (cid:80) m : D i ∈S m P ( {S = S m } ) .
Revised: i is simply the sum of probabilities of all factorizations which include clique i , that is, p i = (cid:80) m : D i ∈S m P ( {S = S m } ) .

Original: Now we prove Theorem 1.
Revised: Now we prove Theorem 1.

Original: In mean-ﬁeld inference, we want to ﬁnd a distribution Q ( X 1 , . . .
Revised: In mean-ﬁeld inference, we want to ﬁnd a distribution Q ( X 1 , . . .

Original: , X n ) = (cid:81)
Revised: , X n ) = (cid:81)

Original: ni =1 Q i ( X i ) such that the cross-entropy between it and a target distribution is minimized.
Revised: ni =1 Q i ( X i ) such that the cross-entropy between it and a target distribution is minimized.

Original: Following the notation in Koller & Friedman (2009), the mean ﬁeld inference problem can written as the following optimization problem.
Revised: Following the notation in Koller & Friedman (2009), the mean ﬁeld inference problem can written as the following optimization problem.

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: Here D ( (cid:81)
Revised: Here D ( (cid:81)

Original: i Q i | P
Revised: i Q i | P

Original: ( X 1 , . .
Revised: ( X 1 , . .

Original: , X n | V )) can be expressed as D ( (cid:81)
Revised: , X n | V )) can be expressed as D ( (cid:81)

Original: i Q i | P
Revised: i Q i | P

Original: ( X 1 , . .
Revised: ( X 1 , . .

Original: , X n | V )) =
Revised: , X n | V )) =

Original: E
Revised: E

Original: Q [ln ( (cid:81) i Q i )] − E Q [ln ( P ( X 1 , . . .
Revised: Q [ln ( (cid:81) i Q i )] − E Q [ln ( P ( X 1 , . . .

Original: , X n | V ))] .
Revised: , X n | V ))] .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: the above optimization problem can be written as
Revised: the above optimization problem can be written as

Original: [Equation]
Revised: [Equation]

Original: In Koller & Friedman (2009), the ﬁxed point equation is derived by solving an analogous equation to (1) without the presence of the p i .
Revised: In Koller & Friedman (2009), the ﬁxed point equation is derived by solving an analogous equation to (1) without the presence of the p i .

Original: Theorem 1 follows by proceeding as in Koller & Friedman (2009) with straightforward accounting for p i .
Revised: Theorem 1 follows by proceeding as in Koller & Friedman (2009) with straightforward accounting for p i .

Original: E P ROOF OF L EMMA 1.
Revised: E P ROOF OF L EMMA 1.

Original: Since we assume semi-cliques are only between two random variables, we can denote C
Revised: Since we assume semi-cliques are only between two random variables, we can denote C

Original: = D ij and presence probabilities as { p ij } where i, j are node indexes.
Revised: = D ij and presence probabilities as { p ij } where i, j are node indexes.

Original: Denote the set of nodes as V .
Revised: Denote the set of nodes as V .

Original: From here, we follow the approach of Dai et al.
Revised: From here, we follow the approach of Dai et al.

Original: and assume that the joint distribution of random variables can be written as
Revised: and assume that the joint distribution of random variables can be written as

Original: [Equation]
Revised: [Equation]

Original: Expanding the ﬁxed-point equation for the mean ﬁeld inference from Theorem 1, we obtain:
Revised: Expanding the ﬁxed-point equation for the mean ﬁeld inference from Theorem 1, we obtain:

Original: [Equation]
Revised: [Equation]

Original: This ﬁxed-point equation for Q k ( h k ) is a function of { Q j ( h j ) } j (cid:54) =
Revised: This ﬁxed-point equation for Q k ( h k ) is a function of { Q j ( h j ) } j (cid:54) =

Original: k such that
Revised: k such that

Original: [Equation]
Revised: [Equation]

Original: As in Dai et al.
Revised: As in Dai et al.

Original: (2016), this equation can be expressed as a Hilbert space embedding of the form
Revised: (2016), this equation can be expressed as a Hilbert space embedding of the form

Original: [Equation]
Revised: [Equation]

Original: where ˜ µ k indicates a vector that encodes Q k ( h k ) .
Revised: where ˜ µ k indicates a vector that encodes Q k ( h k ) .

Original: In this paper, we use the nonlinear mapping ˜ T (based on a neural network form ) suggested in Dai et al.
Revised: In this paper, we use the nonlinear mapping ˜ T (based on a neural network form ) suggested in Dai et al.

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: F P RESENCE PROBABILITY INFERENCE
Revised: F P RESENCE PROBABILITY INFERENCE

Original: Let V denote the set of nodes.
Revised: Let V denote the set of nodes.

Original: In lines 1 and 2 , the likelihood of the existence of a directed edge from each node m to node n is computed by calculating W 1 (cid:0) relu (cid:0)
Revised: In lines 1 and 2 , the likelihood of the existence of a directed edge from each node m to node n is computed by calculating W 1 (cid:0) relu (cid:0)

Original: W 2 u kmn (cid:1)(cid:1) and averaging over the M samples.
Revised: W 2 u kmn (cid:1)(cid:1) and averaging over the M samples.

Original: In lines 3 and 4 , we use the soft-max function to obtain p m,n .
Revised: In lines 3 and 4 , we use the soft-max function to obtain p m,n .

Original: 1 For m, n ∈ V do
Revised: 1 For m, n ∈ V do

Original: [Equation]
Revised: [Equation]

Original: 3 For m, n ∈ V do
Revised: 3 For m, n ∈ V do

Original: [Equation]
Revised: [Equation]

Original: G T ASK COMPLETION TIME AS A RANDOM VARIABLE
Revised: G T ASK COMPLETION TIME AS A RANDOM VARIABLE

Original: We combine random sampling and inference procedure suggested in section 3.2 and Figure 1.
Revised: We combine random sampling and inference procedure suggested in section 3.2 and Figure 1.

Original: Denote the set of task with a robot assigned to it as T A .
Revised: Denote the set of task with a robot assigned to it as T A .

Original: Denote a task in T A as t i and the robot assigned to t i as r t i .
Revised: Denote a task in T A as t i and the robot assigned to t i as r t i .

Original: The corresponding edge in E RT for this assignment is (cid:15) r
Revised: The corresponding edge in E RT for this assignment is (cid:15) r

Original: ti t i .
Revised: ti t i .

Original: The key idea is to use samples of (cid:15) r ti t i to generate N number of sampled Q ( s, a ) value and average them to get the estimate of E ( Q ( s, a )) .
Revised: The key idea is to use samples of (cid:15) r ti t i to generate N number of sampled Q ( s, a ) value and average them to get the estimate of E ( Q ( s, a )) .

Original: First, for l = 1 . . .
Revised: First, for l = 1 . . .

Original: N we conduct the following procedure.
Revised: N we conduct the following procedure.

Original: For each task t i in T A , we sample one data e
Revised: For each task t i in T A , we sample one data e

Original: lr ti t i .
Revised: lr ti t i .

Original: Using those samples and { p ij } , we follow the whole procedure illustrated in section 3.2 to get Q ( s, a ) l .
Revised: Using those samples and { p ij } , we follow the whole procedure illustrated in section 3.2 to get Q ( s, a ) l .

Original: Second, we get the average of { Q ( s, a ) l } l = N l =1 to get the estimate of E ( Q ( s, a )) , 1 N (cid:80) l = N l =1 Q ( s, a ) l .
Revised: Second, we get the average of { Q ( s, a ) l } l = N l =1 to get the estimate of E ( Q ( s, a )) , 1 N (cid:80) l = N l =1 Q ( s, a ) l .

Original: The complete algorithm of section 3.2 with task completion time as a random variable is given as below.
Revised: The complete algorithm of section 3.2 with task completion time as a random variable is given as below.

Original: γ
Revised: γ

Original: Q l = W i ∈V γ
Revised: Q l = W i ∈V γ

Original: 20 Q avg = 1 N (cid:80) Nl
Revised: 20 Q avg = 1 N (cid:80) Nl

Original: =
Revised: =

Original: Q
Revised: Q