L EARNING A T RANSFERABLE S CHEDULING P OLICY FOR V ARIOUS V EHICLE R OUTING P ROBLEMS BASED ON G RAPH - CENTRIC R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning has been used to learn to solve various routing problems.
however, most of the algorithm is restricted to ﬁnding an optimal routing strategy for only a single vehicle.
In addition, the trained policy under a speciﬁc target routing problem is not able to solve different types of routing problems with different objectives and constraints.
This paper proposes an reinforcement learning approach to solve the min-max capacitated multi vehicle routing problem (mCVRP), the problem seeks to minimize the total completion time for multiple vehicles whose one-time traveling distance is constrained by their fuel levels to serve the geographically distributed customer nodes.
The method represents the relationships among vehicles, customers, and fuel stations using relationship-speciﬁc graphs to consider their topological relationships and employ graph neural network (GNN) to extract the graph’s embedding to be used to make a routing action.
We train the proposed model using the random mCVRP instance with different numbers of vehicles, customers, and refueling stations.
We then validate that the trained policy solve not only new mCVRP problems with different complexity (weak transferability but also different routing problems (CVRP, mTSP, TSP) with different objectives and constraints (storing transferability).
I NTRODUCTION
The Vehicle Routing Problem (VRP), a well-known NP-hard problem, has been enormously studied since it appeared by Dantzig & Ramser (1959).
There have been numerous attempts to compute the exact (optimal) or approximate solutions for various types of vehicle routing problems by using mixed integer linear programming (MILP), which uses mostly a branch-and-price algorithm appeared in Desrochers et al.
(1992) or a column generation method (Chabrier, 2006), or heuristics ((Cordeau et al., 2002; Clarke & Wright, 1964; Gillett & Miller, 1974; Gendreau et al., 1994)).
However, these approaches typically require huge computational time to ﬁnd the near optimum solution.
For more information for VRP, see good survey papers (Cordeau et al., 2002; Toth & Vigo, 2002).
There have been attempts to solve such vehicle routing problems using learning based approaches.
These approaches can be categorized into supervised-learning based approaches and reinforcementlearning based approaches (Bengio et al., 2020); supervised learning approaches try to map a target
VRP with a solution or try to solve sub-problems appears during optimization procedure, while reinforcement learning (RL) approaches seek to learn to solve routing problems without supervision (i.e, solution) but using only repeated trials and the associated reward signal.
Furthermore, the
RL approaches can be further categorized into improvement heuristics and construction heuristics (Mazyavkina et al., 2020); improvement heuristics learn to modify the current solution for a better solution, while construction heuristics learn to construct a solution in a sequential decision making framework.
The current study focuses on the RL-based construction heuristic for solving various routing problems.
Various RL-based solution construction approaches have been employed to solve the traveling salesman problem (TSP) (Bello et al., 2016;
Khalil et al., 2017; Nazari et al., 2018; Kool et al., 2018) or the capacitated vehicle routing problem (CVRP) (Nazari et al., 2018; Kool et al., 2018).
(Bello et al., 2016; Nazari et al., 2018; Kool et al., 2018) has used the encoder-decoder structure to sequentially generate routing schedules, and (Khalil et al., 2017) uses graph based embedding to determine the next assignment action.
Although these approaches have shown the potential that the
RL based approaches can learn to solve some types of routing problems, these approaches have the major two limitations: (1) only focus on routing a single vehicle over cities for minimizing the total traveling distance (i.e., min-sum problem) and (2) the trained policy for a speciﬁc routing problem cannot be used for solving other routing problems with different objective and constraints (they show that trained policy can be used to solve the same type of the routing problems with different problem sizes).
In this study, We proposed the Graph-centric RL-based Transferable Scheduler (GRLTS) for various vehicle routing problems.
GRLTS is composed of graph-centric representation learning and RLbased scheduling policy learning.
GRLTS is mainly designed to solve min-max capacititated multi vehicle routing problems (mCVRP); the problem seeks to minimize the total completion time for multiple vehicles whose one-time traveling distance is constrained by their fuel levels to serve the geographically distributed customer nodes.
The method represents the relationships among vehicles, customers, and fuel stations using relationship-speciﬁc graphs to consider their topological relationships and employ graph neural network (GNN) to extract the graph’s embedding to be used to make a routing action.
To effectively train the policy for minimizing the total completion time while satisfying the fuel constraints, we use the specially designed reward signal in RL framework.
The representation learning for graph and the decision making policy are trained in an end-to-end fashion in an MARL framework.
In addition, to effectively explore the joint combinatorial action space, we employ curriculum learning while controlling the difﬁculty (complexity) of a target problem.
The proposed GRLTS resolves the two issues raised in other RL-based routing algorithms:
• GRLTS learns to coordinate multiple vehicles to minimize the total completion time (makespan).
It can resolve the ﬁrst issue of other RL-based routing algorithms and can be used to solve practical routing problems of scheduling multiple vehicles simultaneously.
(Kang et al., 2019) also employed the graph based embedding (random graph embedding) to solve identical parallel machine scheduling problem, the problem seeking to minimize the makespan by scheduling multiple machines.
However, our approach is more general in that it can consider capacity constraint and more fast and scalable node embedding strategies.
• GRLTS transfers the trained scheduling policy with random mCVRP instances to be used for solving not only new mCVRP problems with different complexity but also different routing problems (CVRP, mTSP, TSP) with different objectives and constraints.
F ORMULATION 2. MIN - MAX SOLUTION FOR M CVRP
We deﬁne the set of vehicles V V = 1 , ..., N V , the set of customers V C = 1 , ..., N C , and the set of refueling stations V R = 1 , ..., N R , where N A , N C , and N R are the numbers of vehicles, customers, and refueling stations, respectively.
The objective of min-max mCVRP is minimizing the makespan that is the longest distance among all vehicle’s traveling distance, i.e., min max i ∈ V V L i with L i being the traveling distance of vehicle i , while each vehicle’s one-time traveling distance is constrained by its remaining fuel.
The detailed mathematical formulation using mixed integer linear programming (MILP) is provided in Appendix.
Figure (1) (left) shows a snapshot of a mCVRP state and Figure (1) (right) represents a feasible solution of the mCVRP.
2. D EC -MDP FORMULATION FOR M CVRP
We seek to sequentially construct an optimum solution.
Thus, we frame the solution construction procedure as a decentralized Markov decision problem (Dec-MDP) as follows.
2.2. S TATE
We deﬁne the vehicle state s vt , ∀ v ∈ V V , the customer state s ct , ∀ c ∈ V C , and the refueling station state s rt , ∀ r ∈ V R as follows:
• State of vehicle v , s vt
= ( x vt , f vt , q vt ) .
x vt is the allocated node that vehicle v to visit; f vt is the current fuel level; and q vt is the number of customers served by the vehicle v so far.
• State of a customer c , s c t =
( x c , v c ) .
x c is the location of customer node c (static).
Visit indicator v c ∈ { 0 , 1 } becomes 1 if the customer c is visited and 0 , otherwise.
• State of a refueling station r , s rt =
x r .
x r is the location of the refueling station r (static).
The global state s t then becomes s t
= ( { s vt } N v v =1 , { s ct } N C c =1 , { s rt } N R r =1 ) .
2.2. A CTIONS & S TATE TRANSITION
Action a vt for vehicle v at time t is indicating a node to be visited by vehicle v at time t + 1 , that is, a vt = x vt +1 ∈ { V C ∪ V R } .
Therefore, the next state of vehicle v becomes s vt +
=
( x vt +1 , f vt +1 , q vt +1 ) where f vt +1 and q vt +1 are determined deterministically by an action a vt as follows:
• Fuel capacity update: f vt +1 = (cid:26) F v , if a vt ∈ V R f tv − d ( x vt , a vt ) , otherwise .
• Customer visit number update: q vt +1 = (cid:26) q vt , if a vt ∈ V R q vt + 1 , otherwise .
R EWARDS
The goal of mCVRP is to force all agents to coordinate to ﬁnish the distributed tasks quickly while satisfying the fuel constraints.
To achieve this global goal in a distributed manner, we use the specially designed independent reward for each agent as:
• visiting reward : To encourage vehicles to visit the customer nodes faster, in turn, minimizing makespan, we deﬁne customer visit reward r vvisit =
q vt .
This reward is provided when an agent visits a customer; the more customer nodes a vehicle agent n visits, the greater reward it can earn.
• Refueling reward : To induce a strategic refueling, we introduce refuel reward r vrefuel =
q vt × ( ( F v − f vt ) /
( F v − 1) )
α .
We deﬁne the refuel reward as an opportunity cost.
That is, vehicles with sufﬁcient fuel are not necessary to refuel (small reward).
In contrast, refueling vehicles with a lack of fuel is worth as much as visiting customers.
In this study, we set F v = 10 (which is the equivalent to the total traveling distance that vehicle v can travel with the fuel tank fully loaded) and α = 2 .
2. R ELATIONSHIPS WITH OTHER CLASS OF VRP S
mCVRP, the target problem of this study, has three key properties: 1) the problem seeks to minimize the total completion time of vehicles by forcing all vehicles to coordinate (in a distributed manner), 2) the problem employs fuel capacity constraints requiring the vehicles to visit the refueling stations strategically, 3) the problem considers multiple refueling depots (revisit allowed).
If some of these requirements are relaxed, min-max mCVRP can be degenerated into simpler conventional routing problems:
• TSP is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance.
The agent needs or needs not come back to the depot.
This problem does not have capacity constraints. • CVRP (capacity-constrained TSP) is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance and satisfying the fuel constraint.
The vehicles need to comeback depot to charge.
• mTSP (multi-agent TSP) is the problem where multiple vehicles should serve all the customers as quickly as possible.
This problem does not have capacity constraints.
• mCVRP (multi-agent, capacity-constrained TSP) is our target problem having the properties of both mTSP and CVRP.
Additionally, we add more than one refueling depot.
The mathematical formulations for these problems are provided in Appendix.
We train the policy using random mCVRP instances with varying numbers of agents and customers and employ the trained policy without parameter changes to solve TSP, CVRP and mTSP to test its domain transferability.
M ETHOD
This section explains how the proposed model, given a state (a partial solution), assigns an idle vehicle to next node to visit under the sequential decision-making framework (see Figure 2).
3. S TATE REPRESENTATION USING RELATIONSHIP - SPECIFIC MULTIPLE GRAPHS
The proposed model represents the global state s t using as a weighted graph G t =
G ( V, E, w ) where
V =
{ V V , V C , V R } , and E is the set of edges between node i, j ∈ V and w is weight for edge ( i, j ) (here, distance d ij ).
Each node corresponding to vehicle, customer, and refueling station will be initialized with its associated states deﬁned earlier.
Although we can assume that all nodes are connected with each other regardless of types and distance, we restrict the edge connection to its neighboring nodes to reduce the computational cost.
Speciﬁcally, each type of node can deﬁne its connectivity range and connect an edge if any node is located within its range as follows (see Figure 2):
[Equation]
That is, the vehicle node v ∈ V v connects the edges with nodes that are located within its traveling distance (i.e., the current fuel level f vt ).
In addition, the customer nodes c ∈ V c connects the edges with nodes that are located within the constant range R C .
We set R C = 5 while following a typical hyperparameter selection procedure.
Finally, the refueling node r ∈ V R connects the edges with nodes within maximum distance that the vehicle with the largest fuel capacity can travel with the full loaded fuel (in this study F v = 10 for all vehicles).
Note that the target node j that can be connected to each node can be any types of nodes.
3. N ODE EMBEDDING USING GNN
The proposed model employs Graph neural network (GNN) (Scarselli et al., 2008) to compute the node embeddings for all nodes.
The node embedding procedure starts with constructing the graph G t out of the global state s t =
( { s vt } N v v =1 , { s ct } N C c =1 , { s rt } N R r =1 ) .
The method then compute the initial node embeddings h i and edge embeddings h ij for all the nodes and edges of G t by employing encoder network.
The sequel will explain how the GNN layer update these node and edge embeddings.
3.2. EDGE UPDATE
The edge feature h τij at τ iteration is updated using the edge updater function φ E as
[Equation]
(4) where h τ − 1 i and h τ − 1 j are node embedding vectors of node i and node j at τ − 1 iteration.
3.2. EDGE FEATURE AGGREGATION
The updated edge feature h τij can be thought of as an message sent from node j to node i .
Node i aggregates these messages from its neighboring nodes j ∈ {N V ( i ) , N C ( i ) , N R ( i ) } , where N V ( i ) is the neighboring vehicle nodes of node i , as
[Equation]
where the attention weight α ij = sofmax i ( e ij ) , where e ij =
f e ( s i , s j ; w e ) , scores the signiﬁcance of node j to node i .
Note that message aggregation is separately conducted for different types of noes and the aggregated messages per type are concatenated.
3.2. NODE FEATURE UPDATE
The aggregated edge node embeddings (¯ h τi,V , ¯ h τi,C , ¯ h τi,R ) per its neighborhood type are then used to update the node embedding vector h τi using node update function φ V as
[Equation]
[Equation]
The node embedding procedure is repeated H (hop) times for all nodes, and the ﬁnal node embeddings h H
= { h Hi } Ni
=1 is used to determine the next assignment action of an idle vehicle.
3. D ECISION MAKING USING NODE EMBEDDING
When an vehicle node i reaches the assigned customer node, an event occurs and the vehicle node i computes its node embedding h Hi and selects one of its feasible actions, choosing one of unvisited customer nodes or refueling nodes around vehicle node i .
The probability for agent i to choose node j , a i t = j , is computed by the parameterized actor π ( a i t =
j | s t ; φ ) as:
[Equation]
where F ( h Hi , h Gj ; φ ) is the ﬁtness function evaluating the goodness for agent i to choose node j as the next action (i.e., a it = j ).
3. T RAINING GRLTS
We train the proposed model using random mCVRP instances with varying numbers of vehicles, customers.
We employ the actor-critic algorithm to train the parameters for the GNN, the critic, and the actor (GRLTS).
We ﬁrst approximate the action-value function Q π ( s , a )
≈ ˆ Q π ( s , a ; θ ) using a neural network with the critic parameter θ .
The parameter θ for the centralized critic is optimized to minimize the loss L : [Equation] where y = r + γ ˆ Q π (cid:48) ( h N hop ; θ (cid:48) ) is the target value evaluated with the the actor π (cid:48)
=
π ( a | s ; φ (cid:48) ) and the critic ˆ Q π ( s , a ; θ (cid:48) )
using the target parameters φ (cid:48) and θ (cid:48) .
In addition, D is a state transition memory.
To train the actor network, we use PPO method (Schulman et al., 2017) to maximize J ( φ ) . PPO aims to maximize the clipped surrogate objective function as follow: [Equation] where R t ( φ ) =
π φ
[Equation]
a t | s t )
[Equation]
φ old ( a t | o t ) .
We compute the advantage estimator ˆ A t by running the policy for T time steps as
[Equation] In addition, equation (9) is added by a value function error and an entropy bonus for sufﬁcient exploration because critic and actor share parameters as follow: [Equation] where V θ ( s t )
≈ V ( h N hop t ; θ ) from the centralized critic; H denotes an entropy bonus; and c 1 and care hyperparameters.
On updating the centralized critic parameter θ and the decentralized (but, shared) actor parameter φ , we use Monte-Carlo simulation.
To sequentially update both parameters, we follow an update rule as follow:
[Equation]
Algorithm (1) focus on sequence of the parameter update using above equations.
Algorithm 1 Parameter update in decentralized actors with PPO and Monte-Carlo simulation 1: for Agent i = 1 , 2 , . .
, N do 2: for Update iteration = 1 , 2 , . . .
, K do 3:
Evaluate policy π θ old in environment for an experienced episode 4:
Compute r t ( φ ) in Equation (9) 5:
Compute advantage estimates ˆ A 1 , . . .
, ˆ A T in Equation (10) 6:
Optimize surrogate objective in Equation (11) with batch size T 7: θ old ← θ , φ old ← φ followed by (12) 8: end for 9: end for
E XPERIMENTS
The trained policy is used to solve various types of vehicle routing problems, mCVRP, mTSP, CVRP, and TSP without changing the parameters.
For each type except mCVRP, we use two types of data sets: random instances with varying numbers of vehicles and customers whose locations are sampled randomly, and (2) benchmark problems obtained from the library (mTSP library, CVRP library, and
TSP library).
To validate the proposed GRLTS [Table] we develop mixed integer linear programming (MILP) formulations and compute the solutions using CPLEX 12.9 (all the optimization formulations are provided in the Appendix).
We also use Google OR-Tools(Perron & Furnon) as the representative heuristic solvers.
For the problems where other deep RL based approaches tried to solve (TSP and CVRP), we compare the performance of the proposed approach to those of deep RL baselines.
All experiments are conducted on Windows 10, Intel(R) Core(TM) i [Table] 9900K
[Table]
3.6 GHz [Table] and 32GB RAM. GPU acceleration is not used in testing.
4. P ERFORMANCE COMPARISON OF M CVRP S
[Table] [Table]
We apply the trained policy (GRLTS) to solve mCVRP with different numbers of vehicles N v ∈ 2 , 3 ,and the numbers of customers N c ∈ 25 , 50 , 100 .
The number of refueling stations are set to be 4, 5 and 10 in case of N c
= 25 , N c = 50 and N c = 100 .
For every combination of N v and N c , we randomly generate 100 mCVRP instances, each of which has randomly located customer nodes and refueling nodes.
The x and y coordinates of each node is randomly sampled from the uniform distributions; x ∼ U [0 , 1] and y ∼ U [0 , 1] , respectively. We employ GRLTS and ORT to solve the same 100 random mCVRP instances and compute the average makespan and the computational time required to solve the mCVRP instance with the policies.
Table 1 compares the average makespan and computational times.
ORT (Google OR-tool) produces the best results for the small-sized problems with reasonable computational time; however, it requires extensive time or even fails to compute any feasible solution for the large scale problem ( ∞ means that ORT cannot ﬁnd a feasible solution).
Notably, the GRLTS achieves a better performance than ORT for large-scale problems with a signiﬁcantly shorter computational time.
To validate the scalability of the proposed model, we further conduct test experiments with the four cases: N c = 25 with N v
=
5 , N c = 100 with N v = 10 , N c = 400 with N v = [-
-]20 , and N c
= 2 ,with N v = 50 .
Table 2 shows how the makespan and the computational time increases with the size of the problem.
By comparing the ratio between the number of customers that each vehicle need to serve, (5: 10: 20: 50), and the makespan , (1.20: 1.88: 4.09: 11.71), we can roughly conﬁrms that the trained model can perform reasonably well even in large sized problem that have never been experience during the training.
For all different sizes of the mCVRP problem, we also compare the performance of the proposed method with the solution computed by CPLEX solver.
Because it typically takes a long time to compute the near-optimum solution by CPLEX, we only compare the single mCVRP case.
The result is provided in the Appendix.
4. P ERFORMANCE COMPARISON ON M TSP
We apply the trained network (without parameter changes) to solve mTSP, which seeks to minimize the total completion time of multiple vehicles (minmax mTSP).
This problem is a relaxed version of mCVRP in that it does not require the capacity constraints (we set the fuel level to be constant during execution).
We solve the randomly generated 100 mTSP instances for every combination of N c ∈ 50 , 100 , 400 and N v ∈ 2 , 4 , 6 , 8 (or N v ∈ 10 , 20 , 30 , 40 ).
Table 7 compares the average total completion time and computational time.
The trained model outperforms ORT in the large-size problems (see N c = 100 and N c = 400 cases in Table 7) in terms of both makespan and the computational time.
For large scale problem, GRLTS ac hives roughly 28% shorter makespan than ORT on average
[Table] [Table] Similarly, for all the sizes of mTSP case
[Table]
we compare the performance of GRLTS to the solution computed by CPLEX in the Appendix.
We also employed the trained model to solve mTSP benchmark problems in MTSPLib 1 solving MinMax mTSP.
The performance results on 124 instances (four different number of vehicles per 31 different maps) are provided in Table 9 in Appendix
. 4. P ERFORMANCE COMPARISON ON CVRP ( PAYLOAD CAPACITY )
[Table]
We employ the trained model to solve randomly generated 100 CVRP instances with N c = 20 , 50 ,with a single capacitated vehicle.
All nodes are randomly scattered in the unit square of [0 , 1] ×
To make the trained policy compatible with the CVRP settings [Table] we ﬁx the number of vehicles to be one and use a single refueling node as if it were the depot in CVRP environment.
As a result, the single vehicle needs to revisit the depot due to payload capacity (which is equivalent to fuel constraint).
The payload capacity is set to be 30, 40 and 50 for CVRP20, CVRP50 and CVRP100, respectively.
Demand are uniformly sampled from { 1 , . . .
These test cases have been widely used by the studies seeking to develop RL-based solvers for CVRP problems.
Table 4 summaries results for the random CVRP environment.
L2I and LKH3 are well-known best performing heuristic algorithms developed from the OR community, thus can serve as an oracle for comparing the performance.
We also consider other RL-based approaches (Kool et al., 2018;
Nazari et al., 2018; Chen & Tian, 2019).
In general, our model is not outperforming other RL-based approaches and OR-tool.
However, all RL-based approaches, except our model, are trained under the same environment of CVRP cases (the training and test instances are sampled from distribution).
However, our model is trained with the complete different mCVRP environment and tested with the
CVRP environment (strong transferability).
We also employed the trained policy to solve the CVRP benchmark problem instances from CVRPLib (Uchoa et al., 2017).
Table 10 in Appendix provides all the performance results compared with other RL-baseline models.
The results show that our model has better performance than one of the state-of-art RL-based approach (Kool et al., 2018) [Table]
4. P ERFORMANCE COMPARISON OF TSP
[Table]
Lastly
[Table]
we employ the trained model to solve 100 randomly generated TSP with different number of customers N c
= 20 , 50 , 100 and a single vehicle.
Table 5 shows summarized the results.
Although
GRLTS is not outperforming other RL-based scheduling methods, it shows the reasonably performance that is comparable to some of well known heuristic algorithms.
Given that the GRLTS is trained with mCVRP environment and have never seen TSP instances, this result can validate that
GRLTS can be transferred to TSP as well.
We also employed the trained policy to solve the TSP benchmark problem instances fromTSPLib (Reinhelt, 2014).
Table 8 in Appendix provides all the performance results compared with other
RL-baseline models.
The results show that our model has comparable performance with the-state-ofart RL-based approaches (GPN (Ma et al., 2019) and S2V-DQN (Khalil et al., 2017)) and heuristic algorithms, especially for large-scale TSP problems.
C ONCLUSION
We proposed the Graph-centric RL-based Transferable Scheduler for various vehicle routing problems using graph-centric state presentation (GRLTS) that can solve any types of vehicle routing problems such as mCVRP, mTSP, CVRP, and TSP.
The transferability is achieved by graph-centric representation learning that can generalize well over various relationships among vehicles, customers, and refuel stations (depot).
GRLTR is computationally efﬁcient for solving very large-scale vehicle routing problems with complex constraints, which provides potential that such RL-based scheduler can be used for large-scale realistic applications in logistics, transportation, and manufacturing
A A PPENDIX A. MILP FORMULATIONS FOR R OUTING P ROBLEMS
The MILP formulations for mCVRP, mTSP, and TSP are provided in this section.
All problems are formulated to ﬁnd the optimum routes of vehicles to minimize the makespan.
For TSP formulation, the makespan minimization is equal to total distance minimization due to single-vehicle operation.
Although mTSP here is same as multi-VRP by its deﬁnition, we denote the problem as mTSP because we focus on the makespan minimization problem with multi-vehicle operation.
A.1. TSP
A fundamental formulation in the context of routing problems is based on TSP formulation by
Miller-Tucker-Zemlin (MTZ) formulation (Miller et al., 1960): minimize (cid:88)
i ∈ V (cid:88)
j ∈ V,i (cid:54) =
j d ij x ij subject to.
(cid:88) i ∈
V x ij = 1 , ∀ j ∈ V : i (cid:54) =
j, (1) (cid:88)
j ∈ V x ij = 1 , ∀ i ∈ V : i (cid:54) =
j, (2) u i − u j + | V | x ij ≤ | V | − 1 , ∀ i, j ∈ V \ vStart, (3) 0 ≤ u i ≤ | V | − 1 , ∀ i ∈
V \ vStart, (4) x ij ∈ { 0 , 1 } , ∀ i, j ∈ V, (5) u i ∈
Z , ∀ i ∈ V \ vStart.
The routing problem is deﬁned in a graph G ( V, E, w ) where V , E and w are nodes, edges, and weight (distance d or cost c ).
x ij is 1 if the edge between node i and node j exists, otherwise, 0 .
Here, corresponding distance of the edge is d ij .
Constraint (3,4) with dummy variable u i is for subtour elimination.
As setting d ij = 0 , ∀ j ∈ vStart , the problem can be arbitrary end assumption.
A.1. M TSP
The goal of mTSP is to ﬁnd the sub-routes of multiple vehicles to minimize the makespan.
Thus, decision variables are expanded to multi-vehicle settings, and the objective function is modiﬁed from
A.1.1 to consider the makespan minimization setting (MinMax problem).
minimize
Q subject to
[Equation]
Here, we denoted Q as the makespan, which is the longest traveling distance among multiple vehicles.
By minimizing this maximum traveling distance, the above formulation can minimize the makespan.
Therefore, the above formulation is to minimize the makespan.
We also allow a vehicle to start a tour at any staring node vStart a (constraint (2)).
A.1. M CVRP
We extend the above mTSP formulation in A.1.2 to include the fuel constraint (i.e., allowable traveling distance per sub-tour).
In this formulation, vehicles can start their tour at an arbitrary location.
minimize
Q subject to.
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Constraint (9) indicates that a vehicle can charge fuel as much as the vehicle’s maximum fuel capacity
F a .
In addition, constraint (10) requires that a vehicle must have enough remaining fuel to visit a refueling node.
Constraint (11) indicates that a vehicle cannot consume (travel) more than its fuel capacity.
We allow vehicles to visit refueling nodes as many times as possible by introducing a sufﬁcient number of dummy variables.
A.1. CVRP
Similar to the above mCVRP with fuel constraint, we also solve CVRP constrained by payload capacity (not fuel constraint) and with single vehicle ( m = 1 so that | A | = 1 ).
We consider a single depot to start a mission and unload some burdens.
Therefore, vRefuel in the MILP of mCVRP becomes same as vStart a .
Fuel consumption as much as distance d ij between node i and node j is equivalent to payload capacity consumption as much as demand d j at node j .
As a result, constraint (9-11) of the mCVRP model becomes:
[Equation]
The constraint (10) in the mCVRP model is dropped because payload capacity is independent in returning to the starting depot.
A. C OMPARISON WITH CPLEX SOLUTIONS
For a single mCVRP and mTSP instances, we compare the performance of the proposed approach with the exact solution computed by CPLEX.
This experiment shows how the solution computed from the proposed method is comparable to the near-optimum solution calculated by the powerful optimization solver.
Note that the mCVRP instance for this experiment was generated from the grid environment.
The distance between customer nodes are different from the experiments for Table 1.
A.2. M CVRP
[Table]
We apply the already trained model to solve mCVRP with different numbers of vehicles N v ∈ 2 , 3 ,and the numbers of customers N c ∈ 25 , 50 , 100 .
The number of refueling stations are set to be 4,and 10 in case of N c
= 25 , N c = 50 and N c = 100 .
For all cases, we compute the total completion time and the computational time (the number in the parenthesis) required to construct a scheduling.
We set the limit of the computational time to be 18,000 (sec) for all cases.
The symbol ∞ indicate the case where the computational time is reached and the solution at that moment is used in the table.
The blanks with the hyphen (-) indicate the case where the algorithm could not ﬁnd any feasible solution.
First, CPX (CPLEX) can only solve for mCVRPs with N c = 25 ; for large scale problems, it cannot compute any feasible solution.
ORT (Google OR-tool) produces the best results for the small sized problem with reasonable computational time; however, for the large scale problem it requires extensive amount of time or even fail to compute any feasible solution.
Notably, the proposed GRLTS achieves the good performance with the least amount of computational time.
A.2. M TSP
For the grid environment, a single mTSP instance was generated and used for comparing the performances of GRLTS to CPLEX and ORT.
[Table]
. E XPERIMENT R ESULTS ON B ENCHMARK P ROBLEMS
We apply the trained network to solve benchmark problems for TSP, mTSP and CVRP.
A.3. TSP
Table 8 shows results of TSPLib (Reinhelt, 2014) with some RL-based approaches (GPN (Ma et al., 2019) and S2V-DQN (Khalil et al., 2017) and heuristics.
The average performance of the proposed model is not outperforming the state of the art RL approaches.
However, we observed that the proposed method is scalable in that the performance on the large-scale problem does not degrade compared to other approaches.
We compare the performance by problem instance size (eil51 ∼ tsp225 Vs.
pr226 ∼ pch442).
Our model showed 1.1% reduction in the optimality gap (15.4% → 14.3%).
However, the other RL-based approaches show much higher increases in the optimality gap (Drori et al.
(2020): 3.1% → 10.0%, GPN: 14.1% → 33.1%, Drori et al.
(2020): 4.7% → 10.6%).
[Table]
Table 9 shows results from mTSPLib 2 solving MinMax TSP.
Our model shows 3% longer results than ORT.
However, computational time is, on average, about 30 seconds faster, which is about 45%.
In addition, we calculate difference of the optimality gap by N v = 2 , 3 , 5 , 7 .
Our model shows better performance as N v increases ( N v = 2 case: 9.8%, N v = 3 case: 8.9%, N v = 5 case: 2.8%, N v =case: -7.5%)
[Table]
ch   31.
rat   205.
pr   34.
d   170.
pr   23.
kroA   154.
ch   43.
kroB   92.
kroA   40.
ts   230.
kroB   54.
tsp   277.
[Table]
A.3. CVRP
Table 10 shows results from CVRPLib (Uchoa et al., 2017).
Those values are the total traveling distance.
Although our model is trained to minimize makespan (MinMax), our model shows better performance than AM in the total distance (MinSum).
[Table]
A. G RAPHICAL SOLUTIONS EXAMPLE
This section provides the visual results for several vehicle routing problems solved by the trained policy.
A.4. T RAINING RESULTS ON M CVRP
[Figure]
Figure 3 depicts the evaluation case ( N c = 100 , N v = 5 case of mCVRP) during training process.
This test environment is designed to mimic the search and rescue problems, operating multiple drones to search victims distributed over a map.
Green grids are for refueling nodes, blue circles are for vehicles (color becomes black as fuel capacity diminishes), and grey grids are for unvisited customers.
If a customer is visited by a vehicle, the color becomes red or white; the red girds indicate the victims, while the white indicates no victim in that cell.
These red grids do not have any effect on performance measure, because visiting all customers (grey grids) is top-priority.
[Figure] [Figure]
At the very beginning of the training process (ﬁgure 3 (b)), the trained model cannot have vehicles search over the cells and make them die due to lack of fuel.
At the end of the training process (ﬁgure 3 (c)), however, all customers are visited by the vehicles.
A.4. M VRP
[Equation]
Figure 4 depicts an example solutions of mTSP using CPLEX [Table] OR-Tools [Table]
and RL. The RL-based solver is trained under mCVRP environment but we solve mTSP with the trained model
. A.4. TSP
Figure 5 depicts an example solutions of TSP using CPLEX, OR-Tools
[Table]
and RL.
The RL-based solver is trained under mCVRP environment but we solve TSP with the trained model.
A. N ETWORK A RCHITECTURE AND H YPERPARAMETERS
Table 11 is summary of hyperparameters used in training process.
[Table]
[Figure]
A. T RAINING PERFORMANCE
Figure (6) shows how the trained policy perform on the thee set of validation problems while the policy is being trained by the random mCVRP inc stances.
The three validation problems are 1)customers covered by 5 vehicles, 2) 100 customers by 10 vehicles and 3) 400 customers covered by 20 vehicles.
The ﬁrst row show the performance on the random training instances while the second, third, and fourth row show the performances on the three validation problems.
As training processes progress, the trained model gradually becomes more efﬁcient, that is, a ﬂeet of vehicles visits more customers faster.
After about 200 training epochs, the trained model converges, but there are some randomness due to the nature of policy gradient structure.
Although the makespan curves in the second column of training case seems relatively constant (the ﬁrst row row), the level of difﬁculty is increasing as curriculum becomes harder (The number of refueling is also increasing as curriculum becomes harder).
Comparison of case #1 and case #2 give us two interpretations: 1) case #1 shows the model is trained to visit all customers and 2) in case #2, the model achieves consistent performance as training iterations progress in spite of same visit ratio as 1 around 100 epoch and 350 epoch.
Besides, the makespans of both training epochs are very close.
Case #2 is relatively easier environment than case #1 in that more vehicles are deployed to serve same number of customers.
[Figure]
[Figure]
Simulation: Problem Instance generation We generate random mCVRP instances and use these training instances to train the proposed model.
For training, we randomly select the number of vehicles, customers, and refueling stations.
We randomly assign these entities over the grid world with the distance between each grid cell being 1.
When transforming the snapshot of the mCVRP to a graph, we use the Manhattan distance between the two cells as the edge weight.
In addition, the speed of each vehicle is set to be 1; thus, each vehicle moves over one cell during a single time-tic.
Although the world is represented as discrete gird, the mCVRP problem’s state transition is event-based.
Whenever the agent reaches the assigned customer, the event occurs, and all the edge distance is updated.
Initially, this grid world is developed to use the trained policy for search and rescue problems, seeking distributed victims over the grid-world.
Because each vehicle (drone) can search a speciﬁc zone within a certain amount of time, using a cell-based grid world is a reasonable choice.
Once the policy is trained, it can be used for both a discrete world or a continuous world because these environments can be transformed into a graph without any difference.
We validated the proposed method using CVRP and TSP instances deﬁned over the continuous state to compare the performance of the proposed model to other deep baseline line models.
Other deep baseline line models all use the continuous state.
To boost training, we vary the difﬁculty level of random mCVRP instances during training process (see Algorithm (2)).
For the curriculum instance generation in line 9, we compute N curr , the number of customers which are assumed to be visited already, as N curr =
N c × (1 − 2 × currLevel/ 10) .
Then, we choose N curr tasks randomly and mark as visited and distribute N curr for all agents’ visit number q t .
1: Generate random problem instances N v , N c , N r , x 0 v , x 0 r , 2: currLevel ←3: for i training = 1 , . .
. , N training do 4: for each 100 training iterations do 5: currLevel ← currLevel +6: end for 7:
Generate a random number p curr ∼ U [0 , 1] 8: if p curr ≥ 0 .
5 then 9:
Curriculum instance generation 10: else 11:
Non-curriculum instance generation (followed by line 1) 12: end if 13: end for
Simulation: Episode generation As a medium for the interaction of the two components such as environment and the GRLTS, simulation conducts the agents’ actions and stores the state transitions which are used to update the GRLTS.
Starting from the problem instance generation (Algorithm (2)), simulation assigns the agents’ assignment as a tagt using the action probability π ( ·| o tagt ) computed from the proposed GRLTS.
Simulation computes the agent agt ’s transition time ∆ agt as follow:
[Equation]
(13) where velocity of agents are constant as 1 and distance between i and j Dist ( i, j ) can be the
Manhattan distance or Euclidean distance.
1: Initialize problem instances (based on Algorithm 2) 2: termination, done ← False , t ← 0 , memory ← ∅ 3: idleAgts ← { 1 , 2 , . . .
, N v } , actingAgts ← ∅ 4: while termination do 5: for agt ∈ idleAgts do 6:
Send an obervation Π agt o tagt to a GRLTS 7:
Choose an action a tagt ∼ π ( ·| o tagt ) where an action probability is from GRLTS 8:
Assign the action a tagt to the agent 9: end for 10:
Compute and update transition time ∆ agt for agt ∈ idleAgts (Equation (13)) 11: actingAgts ← { agt | ∆ agt = min agt ∆ agt } for all agents 12: t ← t + min agt ∆ agt 13: for agt ∈ actingAgts do 14:
Perform the recently assigned action a t (cid:48) agt assigned by π ( ·| o t (cid:48) agt ) 15:
Observe the observation o tagt , reward r tagt and done 16:
Append a state transition tuple < t (cid:48) , o t (cid:48) agt , a t (cid:48) agt , r tagt , o tagt > to memory 17: end for 18: if done then 19: termination ← True 20: else 21: idleAgts ← actingAgts 22: end if 23: end while
RL networks: GRLTS Most of algorithms in the RL model is explained the paper so that we focus on the interaction between simulation and GRLTS networks and internal message exchanges.
agt o tagt time simulation
G t an input observations Π
i o ti embedding do 4:
Compute connectivity f n , ∀ j ∈ V T ∪ V R ∞ , ∀ j ∈ V A .
For refueling nodes r , C rj = F, ∀ j ∈ V
For task nodes t , C tj = 5 , ∀ j ∈ V 5:
Store the node embedding vector h Ai at memeory i 6:
Update the connected edges following Equation (3) 7:
Aggregate the incoming edge featured at each nodes following Equation (4) 8:
Update the node feature with the node updator function φ v 9: end for 10: Compute Q ( h Ai , a ) for agent i where a ∈ A ( o i ) = { v ∈ { V R ∪ V T }| C iv = 1 } 11: Compute π ( a | o i ) =
exp ( Q ( h Ai ,a )) (cid:80) a ∈ A ( oi ) exp
( Q ( h Ai ,a )) 12: Send the action probability π ( a | o i ) to simulation