Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Abstract
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown significant merit over the conventional CO solvers as DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervised learning method).
This paper presents a novel training scheme, Sym-NCO, which is a regularizer-based training scheme that leverages universal symmetricities in various CO problems and solutions.
Leveraging symmetricities such as rotational and reflectional invariance can greatly improve the generalization capability of
DRL-NCO because it allows the learned solver to exploit the commonly shared symmetricities in the same CO problem class.
Our experimental results verify that our Sym-NCO greatly improves the performance of DRL-NCO methods in four CO tasks, including the traveling salesman problem (TSP), capacitated vehicle routing problem (CVRP), prize collecting TSP (PCTSP), and orienteering problem (OP), without utilizing problem-specific expert domain knowledge.
Remarkably, SymNCO outperformed not only the existing DRL-NCO methods but also a competitive conventional solver, the iterative local search (ILS), in PCTSP at 240 × faster speed.
Our source code is available at https://github.com/alstn12088/Sym-NCO .
Introduction
Combinatorial optimization problems (COPs), mathematical optimization problems on discrete input space, have been used to solve numerous valuable applications, including vehicle routing problems (VRPs) [1, 2], drug discovery [3, 4] and semi-conductor design [5, 6, 7, 8, 9].
However, finding an optimal solution to COP is difficult due to its NP-hardness.
Therefore, computing near-optimal solutions fast is essential from a practical point of view.
Conventionally, COPs are solved by integer program (IP) solvers or hand-crafted (meta) heuristics.
Recent advances in computing infrastructures and deep learning have conceived the field of neural combinatorial optimization (NCO), a deep learning-based COP solving strategy.
Depending on the training scheme, NCO methods are generally classified into supervised learning [10, 11, 12, 13, 14] and reinforcement learning (RL) [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].
Depending on the solution generation scheme, NCO methods are also classified into improvement [17, 16, 15, 28, 18, 19, 25] and constructive heuristics [20, 21, 22, 23, 24, 26, 27].
Among the NCO approaches, deep RL (DRL)-based constructive heuristics (i.e., DRL-NCO) are favored over conventional approaches for two major reasons.
First, RL can be applied to train the NCO model in less explorered CO tasks because training RL does not require domain expert knowledge and supervised labels from a verified solver.
Second, it is easy to produce qualified feasible solutions because the constructive process can easily avoid constraint-violated actions [21].
Despite the strength of DRL-NCO, there exists a performance gap between the state-of-the-art
[Figure]
conventional heuristics and DRL-NCO.
In an effort to close the gap, there have been attempts to employ problem-specific heuristics to existing DRL-NCO methods [23, 29].
However, devising a general training scheme to improve the performance of DRL-NCO still remains challenging.
[Figure]
Symmetric Neural
Combinatorial Optimization (Sym-NCO), a general training scheme to universal CO problems.
Sym-NCO is a regularization-based training scheme that leverages the symmetricities commonly found in COPs to increase the performance of existing DRL-NCO methods.
Sym-NCO leverages two types of symmetricities innate in COP that are defined on the Euclidean graph.
First, the problem symmetricity is derived from the rotational invariance of the solution; the rotated graph must exhibit the same optimal solution as the original graph as shown in Fig.
2a.
Second, the solution symmetricity refers to the property that solutions have identical output values (See Fig. 2b).
To train an effective NCO solver while leveraging the symmetricities, we employ REINFORCE algorithm with the baselines terms specially designed to impart solution and problem symmetricities.
REINFORCE algorithm is used because the well-known effective NCO solvers are trained by REINFORCE; thus, we can improve such solvers by just modifying their baseline with our symmetricity-considered baseline terms.
Specifically, we sample multiple solutions from the transformed problems and use the average return of them.
Then, REINFORCE pushes each solution sampled by the solver to excel this baseline during training, thus improving the policy and making all the solutions the same, i.e., the problem and solution symmetricities are realized.
Motivation for learning symmetricity.
Leveraging symmetricity is important to train CO models for two major reasons.
Firstly, symmetricity is a strong inductive bias that can support the training process of DRL by making compact training space as shown in Fig.
Secondly, learning symmetricity is beneficial to increasing generalization capability for unseen CO problems because symmetricity induces the invariant representation that every COP contains.
Novelty.
The major novelty of the proposed learning strategy is that it can easily improve existing powerful CO models; existing equivariant neural network schemes must be re-designed at the architecture level for adapting to CO.
Symmetricity in Combinatorial Optimization Markov Decision Process
This section presents several symmetric characteristics found in combinatorial optimization, which is formulated in the Markov decision process.
The objective of NCO is to train the θ -parameterized solver F θ by solving the following problem:
[Equation]
where P =
( x , f ) is a problem instance with the N node coordinates x = { x i } Ni =1 and corresponding
N features f = { f i } Ni =1 .
The ρ is a problem generating distribution.
The π = { π i } Ni =1 is a solution where each element is index value π i ∈ { 1 , ..., N } for coordinates x
= { x i } Ni =1 .
The R ( π ; P ) is objective value for π on problem P .
For example, assume that we solve TSP with five cities (i.e. N = 5 ).
Then the problem instance
P contains five city coordinates x
= { x i } 5 i =1 where the salesman must visit.
The solution π is a sequence of city indices; if π = { 1 , 3 , 2 , 5 , 4 } , the salesman visits city coordinates as x 1 → x 3 → x 2 → x 5 → x 4 → x 1 (the salesman must go back to first visited city to complete a tour).
In TSP, each city contains the homogeneous features f .
Thus, objective R of the TSP is defined as the negative of tour length:
R ( π ; P ) = − (cid:0) (cid:80)
[Equation]
=
||
x
π
[Equation]
1 − x π i ||
+ || x
[Equation]
5 − x π 1 || (cid:1) .
This combinatorial decision process can be expressed as Markov decision process, and the solver F θ ( π | P ) can be expressed as instance conditioned policy .
To this end, we can utilize deep reinforcement learning for training solver F θ ( π | P ) .
We formally define the Markov decision process for CO in the below chapter.
2.1 Combinatorial optimization Markov decision process
We define the combinatorial optimization Markov decision process (CO-MDP) as the sequential construction of a solution of COP.
For a given P , the components of the corresponding CO-MDP are defined as follows:
• State.
The state s t =
( a 1: t , x , f ) is the t -th (partially complete) solution, where a 1: t represents the previously selected nodes.
The initial and terminal states s 0 and s T are equivalent to the empty and completed solution, respectively.
In this paper, we denote the solution π ( P ) as the completed solution.
• Action.
The action a t is the selection of a node from the un-visited nodes (i.e., a t ∈
A t = {{ 1 , ..., N } \ { a 1: t − 1 }} ).
• Reward.
The reward function R ( π ; P ) maps the objective value from given π of problem P .
We assume that the reward is a function of a 1: T (solution sequences), || x i − x
j || i,j ∈{ 1 ,...N } (relative distances) and f (nodes features).
In TSP, capacitated VRP (CVRP), and prize collecting TSPs (PCTSP), the reward is the negative of the tour length.
In orienteering problem (OP), the reward is the sum of the prizes.
Having defined CO-MDP, we define the solution mapping as follows:
π
[Equation]
where p θ ( a t | s t ) is the policy that produces a t at s t , and T is the maximum number of states in the solution construction process.
2.2 Symmetricities in CO-MDP
Symmetricities are found in various COPs.
We conjecture that imposing those symmetricities on F θ improves the generalization and sample efficiency of F θ .
We define the two identified symmetricities that are commonly found in various COPs:
Definition 2.1 ( Problem Symmetricity ) .
Problem P i and P j are problem symmetric ( P i sym ←−→
P j ) if their optimal solution sets are identical.
Definition 2.2 ( Solution Symmetricity ) .
Two solutions π i and π j are solution symmetric ( π i sym ←−→ π j ) on problem P if R ( π i ; P )
=
R ( π j ; P ) .
An exemplary problem symmetricity found in various COPs is the rotational symmetricity:
Theorem 2.1 ( Rotational symmetricity ) .
For any orthogoanl matrix Q , the problem P and Q ( P ) ≜ {{ Qx i } Ni =1 , f } are problem symmetric: i.e., P sym ←−→ Q ( P ) .
See Appendix A for the proof.
Rotational problem symmetricity is identified in every Euclidean COPs.
On the other hand, solution symmetricity cannot be identified easily as the properties of the solutions are distinct for every COP.
Symmetric Neural Combinatorial Optimization
This section presents Sym-NCO, an effective training scheme that leverages the symmetricities of
COPs.
Sym-NCO learns a solve F θ by minimizing the total loss function:
[Equation]
where L Sym-RL is REINFORCE loss term supported by symmetricity and L inv is the regularization loss to induce that invariant representation.
The
L Sym-RL is composed with L ss , the REINFORCE loss term of Eq.
(1) with solution symmetricity regularizing mechanism, and
L ps , the REINFORCE loss term of Eq.
(1) with both solution and problem symmetricity regularization.
α, β ∈ [0 , 1] are the weight coefficients.
In the following subsections, we explain each loss term in detail.
3.1 Regularizing REINFORCE with problem and solution symmetricities via L Sym-RL
As discussed in Section 2.2, COPs have problem and solution symmetricities.
We explain how to learn solver F θ using REINFORCE with the specially designed baseline to approximately impose symmetricities.
We provide the policy gradients to L ss ( π ( P )) and L ps ( π ( P )) in the context of the
REINFORCE algorithm [30] with the proposed baseline schemes.
Leveraging solution symmetricity.
As defined in Definition 2.2, the symmetric solutions must have the same objective values.
We propose the REINFORCE loss L ss with the baseline specially designed to exploit the solution symmetricity of CO as follows:
[Equation] [Equation] [Equation]
[Equation] [Equation]
[Equation]
where { π k } Kk =1 are the solutions of P sampled from F θ ( π | P ) , log F θ is the log-likelihood of F θ , K is the number of sampled solutions, b ( P ) is a shared baseline which is the average reward from K solutions for the identical problem P .
The L ss approximately imposes solution symmetricity using REINFORCE algorithm with a novel baseline b ( P ) .
The sum of advantage in the solution group { π k } Kk =1 is always zero:
[Equation]
The L ss induces competition among the rewards, R ( π 1 ; P ) , ..., R ( π K ; P ) which can be seen as a zero-sum game.
Therefore, L ss improves the overall reward quality of the solution group using the proposed competitive REINFORCE scheme, making the solver generate high-rewarded solutions but a small reward deviation between solutions.
The small reward-deviation indicates L ss approximately imposes solution symmetricity to solver F θ .
The POMO [23] employed a similar training technique with our L ss , that finds symmetric solutions by forcing F θ to visit all possible initial cities when solving TSP and CVRP.
However, the reward of
COPs, including CVRP, PCTSP, and OP, is usually sensitive to first-city selection.
Therefore, POMO can be an excellent standalone method for TSP but can be further improved using our L ss loss term in other tasks including CVRP.
Leveraging problem symmetricity.
As discussed in Section 2.2, the rotational problem symmetricity is common in various COPs.
We propose the REINFORCE loss L ps which is equipped with problem symmetricity:
[Equation]
where Q is the distribution of random orthogonal matrices, Q l is the l th sampled rotation matrix, and π l,k is the k th sample solution of the l th rotated problem.
We construct L problem symmetric problems, Q 1 ( P ) , ..., Q L ( P ) , by using the sampled rotation matrices, and smaple K symmetric solutions from each of the L problems.
Then, the shared baseline b ( P , Q ) is constructed by averaging
L × K solutions.
Similar to the regularization scheme of L ss , the advantage term of L ps also induces competition between solutions sampled from rotationally symmetric problems.
Since the rotational symmetricity is defined such that x and Q l ( x ) have the same solution, the negative advantage value forces the solver to find a better solution.
As mentioned in Section 2.2, problem symmetricity in COPs is usually pre-identified (i.e. there is provable guaranteed symmetricity such as rotational symmetricity
Theorem 2.1)
; L ps are applicable to general COPs.
Moreover, multiple solutions are sampled for each symmetric problem so that L ps can also identify and exploit the solution symmetricity with a similar approach taken for L ss .
We provide detailed implementation and design guides regarding the integration strategy of L ss and L ps in Appendix C.2.
3.2 Learning invariant representation with Pre-identified Symmetricity: L inv .
By Theorem 2.1, the original problem x and its rotated problem Q ( x ) have identical solutions.
Therefore the encoder of F θ can be enforced to have invariant representation by leveraging the pre-identified symmetricity: rotation symmetricity.
We denote h ( x ) and h ( Q ( x )) as the hidden representations of x and Q ( x ) , respectively.
To impose the rotational invariant property on h ( x ) , we train solver F θ with an additional regularization loss term L inv defined as:
[Equation]
where S cos ( a, b ) is the cosine similarity between a and b .
g is the MLP-parameterized projection head.
For learning the invariant representation on rotational symmetricity, we penalize the difference between the projected representation g ( h ( x )) and g ( h ( Q ( x ))) , instead of directly penalizing the difference between h ( x ) and h ( Q ( x ) .
This penalizing scheme allows the use of an arbitrary encoder network architecture while maintaining the diversity of h [31].
We empirically verified that this approach attains stronger solvers as described in Section 6.1.
Related Works
Deep construction heuristics.
Bello et al.
[20] propose one of the earliest DRL-NCO methods, based on pointer network (PointerNet) [10], and trained it with an actor-critic method.
Attention model (AM) [21] successfully extends [20] by swapping PointerNet with Transformer [32], and it is currently the de-facto standard method for NCO.
Notably, AM verifies its problem agnosticism by solving several classical routing problems and their practical extensions [7, 19].
The multi-decoder
AM (MDAM)
[26] extends AM by employing an ensemble of decoders.
However, such an extension is inapplicable for stochastic routing problems.
The policy optimization for multiple optimal (POMO) [23] extends AM by exploiting the solution symmetricities in TSP and CVRP.
Even though POMO shows significant improvements from AM, it relies on problem-specific solution symmetricities for TSP.
Our method can be seen as a general-purpose symmetric learning scheme, extended from
POMO, which can be applied in more general CO tasks.
Equivariant deep learning.
In deep learning, symmetricities are often enforced by employing specific network architectures.
Niu et al.
[33] proposes a permutation equivariant graph neural network (GNN) that produces equivariant outputs to the input order permutations.
The SE (3) -Transformer
[34] restricts the Transformer so that it is equivariant to SE (3) group input transformation.
Similarly, equivariant GNN (EGNN) [35] proposes a GNN architecture that produces O ( n ) group equivariant output.
These network architectures can dramatically reduce the search space of the model parameters.
Some research applies equivariant neural networks to RL tasks to improve sample efficiency [36].
Also, there are several works that exploited the symmetric nature of CO.
Ouyang et al.
[37] proposed equivariant encoding scheme by giving rule-based input transformation to input graph.
Hudson et al.
[38] suggested a line-graph embedding scheme, which is beneficial to process CO graphs with rotational equivariant.
Our Sym-NCO is a regularization method that is capable of learning existing powerful CO models without giving rule-based hard constraints to the structure.
We empirically study the benefit of Sym-NCO over other symmetricity-based approaches in Section 6.1 and Appendix D.5.
Experiments
This section provides the experimental results of Sym-NCO for TSP, CVRP, PCTSP, and OP.
Focusing on the fact that Sym-NCO can be applied to any encoder-decoder-based NCO method, we implement
Sym-NCO on top of POMO [23] to solve TSP and CVRP, and AM [21] to solve PCTSP and OP, respectively.
We additionally validate the effectiveness of Sym-NCO on PointerNet [10] at TSP ( N = 100 ).
5.1 Tasks and baseline selections
TSP aims to find the Hamiltonian cycle with a minimum tour length.
We employ Concorde [39] and LKH-3 [40] as the non-learnable baselines, and PointerNet [10], the structured-to-vector deepQ-network (S2V-DQN) [41], AM [21], POMO [23] and MDAM [26] as the neural constructive baselines.
CVRP is an extension of TSP that aims to find a set of tours with minimal total tour lengths while satisfying the capacity limits of the vehicles.
We employ LKH-3 [40] as the non-learnable baselines, and Nazari et al.
[22], AM [21], POMO [23] and MDAM [26] as the constructive neural baselines.
PCTSP is a variant of TSP that aims to find a tour with minimal tour length while satisfying the prize constraints.
We employ the iterative local search (ILS) [21] as the non-learnable baseline, and AM [21] and MDAM [26] as the constructive neural baselines.
OP is a variant of TSP that aims to find the tour with maximal total prizes while satisfying the tour length constraint.
We employ compass [42] as the non-learnable baseline, and AM [21] and MDAM [26] as the constructive neural baselines.
5.2 Experimental setting
Problem size.
We provide the results of problems with N = 100 for the four problem classes, and real-world TSP problems with 50 < N < 250 from TSPLIB.
Hyperparameters.
We apply Sym-NCO to POMO, AM, and PointerNet.
To make fair comparisons, we use the same network architectures and training-related hyperparameters from their original papers to train their Sym-NCO-augmented models.
Please refer to Appendix Appendix C.1 for more details.
Dataset and Computing Resources.
We use the benchmark dataset [21] to evaluate the performance of the solvers.
To train the neural solvers, we use Nvidia A100 GPU.
To evaluate the inference speed, we use an Intel Xeon E5-2630 CPU and Nvidia RTX2080Ti GPU to make fair comparisons with the existing methods as proposed in [26].
5.3 Performance metrics
This section provides detailed performance metrics:
Average cost.
We report an average cost of 10,000 benchmark instances which is proposed by [21].
[Table] [Table]
.45% 3m
Evaluation speed.
We report the evaluation speeds of solvers in a out-of-the-box manner as they are used in practice.
In that regard, the execution time of non-neural and neural methods are measured on
CPU and GPU, respectively.
Greedy/Multi-start performance.
For neural solvers, it is a common practice to measure multi-start performance as its final performance.
However, when those are employed in practice, such resource consuming multi-start may not be possible.
Hence, we discuss greedy and multi-start separately.
5.4 Experimental results
Results of TSP and CVRP.
As shown in Table 1, Sym-NCO outperforms the NCO baselines in both the greedy rollout and multi-start settings with the fastest inference speed.
Remarkably,
Sym-NCO achieves a 0 .
95% gap in TSP using the greedy rollout.
In the TSP greedy setting, it solves
TSP 10,000 instances in a few seconds.
Results of PCTSP and OP.
As shown in Table 2, Sym-NCO outperforms the NCO baselines in both the greedy rollout and multi-start settings.
In the multi-start setting, Sym-NCO outperforms the classical PCTSP baseline (i.e., ILS) with the 43200180 ≈ 240 × faster speed.
[Figure] [Figure]
Gap
POMO 1.87%
Sym
[Table]
Results of the real-world TSP.
We evaluate POMO and Sym-NCO on
TSPLib [43].
Table 3 shows that Sym-NCO outperforms POMO.
Please refer to Appendix D.2 for the full benchmark results.
Application to various DRL-NCO methods.
As discussed in Section 3,
Sym-NCO can be applied to various various DRL-NCO methods.
We validate that Sym-NCO significantly improves the existing DRL-NCO methods as shown in Fig.
Time-performance analysis for multi-starts.
Multi-starts is a common method that improves the solution qualities while requiring a longer time budget.
We use the rotation augments [23] to produce multiple inputs (i.e., starts).
As shown in Fig.
5, Sym-NCO achieves the Pareto frontier for all benchmark datasets.
In other words, Sym-NCO exhibits the best solution quality among the baselines within the given time consumption.
Discussion 6.1 Discussion of Regularization based Symmetricity Learning
Ablation Study of L inv .
As shown in Fig.
6b, L inv increases the cosine similarity of the projected representation (i.e., g ( h ) ).
We can conclude that L inv contributes to the performance improvements (see Fig. 6a).
We further verify that imposing similarity on h degrades the performance as demonstrated in Fig.
6c.
This proves the importance of maintaining the expression power of the encoder as we mentioned in Section 3.2.
Comparison with EGNN.
EGNN [35] provably guarantees to process coordinate graph with rotational equivalency.
Also, EGNN empirically verified its’ high performance in point cloud tasks.
Therefore, we implemented a simple EGNN-based CO policy (simply termed EGNN in this paper), to check the feasibility of CO.
We leverage six EGNN layers with 128 hidden dimensions, to replace the POMO encoder where the POMO decoder is unchanged.
In the experimental results, we observed that EGNN significantly underperforms Sym-NCO and fails to converge as shown Fig.
7b.
This is because the euclidian CO has a fully connected input graph containing informative coordinates, and we believe the equivariant neural network should be carefully crafted to consider such unique input graph structures of CO tasks.
On the other hand, Sym-NCO could leverage the existing powerful NCO model without fine modification of the neural network.
These numerical results conform well with our hypothesis that equivariance is necessary but not
[Figure] [Figure]
sufficient for obtaining the optimal parameters for the solver.
Therefore
[Figure]
we expect our approach can be simply extended to other domains requiring symmetricity and geometricity as positioned to leverage the existing legacy of powerful non-equivariant neural networks
[Figure]
6.2 Limitations & future directions
Extended problem symmetricities.
In this work
[Figure]
we employ the rotational symmetricity (Theorem 2.1) as the problem symmetricity.
However, for some COPs, different problem symmetricities, such as scaling and translating P , can also be considered.
Employing these additional symmetricities may further enhance the performance of Sym-NCO.
We leave this for future research.
Large scale adaptation.
Large scale applicability is essential to NCO.
In this work we simply present scale adaptation capability of Sym-NCO using the effective active search (EAS) [44]; see
Appendix D.3.
We expect curriculum-
[45], and meta-learning approaches may improve the generalizability of NCO to larger-sized problems.
Extension to the graph COP.
This work finds the problem symmetricity that is universally applicable for Euclidean COPs.
However, some COPs are defined in non-Euclidean spaces such as asymmetric TSP.
Furthermore, there are also a bunch of existing neural combinatorial optimization models that can solve graph COP [46, 47, 48, 49, 50, 51], where we can improve with a symmetricity regularization scheme.
We also leave finding the universal symmetricities of non-Euclidean COPs and applying them to existing graph COP models for future research.
6.3 Social Impacts
Design automation through NCO research affects various industries including logistics and transportation industries.
From a negative perspective, this automation process may have some concerns to lead to unemployment in certain jobs.
However, automation of logistics, transportation, and design automation can increase the efficiency of industries, reducing CO2 emissions (by reducing total tour length) and creating new industries and jobs.
Acknowledgments and Disclosure of Funding
We thank Jiwoo Son, Hyeonah Kim, Haeyeon Rachel Kim, Fangying Chen and anonymous reviews for proving helpful feedback for preparing our manuscripts.
This work was supported by a grant of the
KAIST-KT joint research project through AI2XL Laboratory, Institute of convergence Technology, funded by KT [Project No. G01210696, Development of Multi-Agent Reinforcement Learning
Algorithm for Efficient Operation of Complex Distributed Systems]. A Proof of Theorem 2.
In this section, we prove the Theorem 2.1, which states a problem P and its’ orthogonal transformed problem Q ( P ) =
{{ Qx i } Ni =1 , f } have identical optimal solutions if Q is orthogonal matrix: QQ T =
Q T Q
= I .
As we mentioned in Section 2.2, reward R is a function of a 1: T (solution sequences),
|| x i − x
j || i,j ∈{ 1 ,...N } (relative distances) and f (nodes features).
For simple notation, let denote
|| x
i − x
j || i,j ∈{ 1 ,...N } as || x i − x j || .
And Let R ∗ ( P ) is optimal value of problem P : i.e.
[Equation]
Where π ∗ is an optimal solution of problem P .
Then the optimal value of transformed problem
Q ( P ) , R ∗ ( Q ( P )) is invariant:
[Equation]
Therefore, problem transformation of orthogonal matrix Q does not change the optimal value.
Then, the remaining proof is to show Q ( P ) has an identical solution set with P .
Let optimal solution set Π ∗ ( P ) =
{ π i ( P ) } Mi =1 , where π i ( P ) indicates optimal solution of P and
M is the number of heterogeneous optimal solution.
For any π i ( Q ( P )) ∈
Π ∗ ( Q ( P )) , they have same optimal value with P :
[Equation]
Thus, π i ( Q ( P )) ∈
Π ∗ ( P ) .
Conversely, For any π i ( P ) ∈
Π ∗ ( P ) , they have sample optimal value with Q ( P ) :
[Equation]
Thus, π i ( P ) ∈
Π ∗ ( Q ( P )) .
Therefore, Π ∗ ( P ) =
Π ∗ ( Q ( P )) , i.e., P sym ←−→ Q ( P ) .
B Implementation of Baselines
We directly reproduce competitive DRL-NCO methods: POMO [23] and AM [21] and PointerNet [10, 20].
PointerNet.
The PointerNet is early work of DRL-NCO using LSTM-based encoder-decoder architecture trained with actor-critic manner.
We follow the instruction of open source code 1 by [21] following hyperparmeters.
[Table]
AM.
The AM is a general-purpose DRL-NCO, a transformer-based encoder-decoder model that solves various routing problems such as TSP, CVRP, PCTSP
[Table]
and OP.
We follow the instruction of open source code, same with the PointerNet with the following hyperparameters.
[Table]
POMO.
The POMO is a high-performance DRL-NCO for TSP and CVRP, implemented on the top of the AM.
We follow the instruction of open source code 2 with the following hyperparameters.
[Table]
[Table]
[Figure]
C Implementation Details of Proposed Method C.1 Training Hyperparameters
Sym-NCO is a training scheme that is attached to the top of the existing DRL-NCO model.
We set the same hyperparameters with PointerNet, AM, and POMO Appendix B except REINFORCE baseline (we set the proposed Sym-NCO baseline introduced in Section 3).
Sym-NCO has additional hyperparameters.
First of all, we set identical hyperparameters for PointerNet and AM for all tasks:
[Table]
Note that the design choice of β = 0 is to show high applicability of L ps , and is because AM with
L ss is just similar to the POMO.
For POMO, we set β = 1 to force solution symmetricity on the top of POMO’s baseline.
Note that we follow POMO’s first node restriction only in TSP, which is a reasonable bias as we mentioned in
Section 3.
The hyperparameter setting is as follows:
[Table]
Note that the design choice of β = 1 and K = 100 is based on POMO’s baseline setting.
We just set
L = 2 , because of training efficiency.
We suggest to set L > 4 if training resources and time-budget is sufficient;; it may increase performance further.
C.2 Integration of L ss and L ps
The L ps is an extension of L ss where it can both leverage problem symmetricity and solution symmetricity.
Therefore, we can simply use
L RL-Sym = L ps .
However, some specific CO problem such as TSP has cyclic nature, which contains pre-identifiable solution symmetricity, and some method already exploit the cyclic nature.
For example, the POMO [23] which is a powerful NCO model already utilizes pre-identified solution symmetricity in the training process for specific CO tasks.
Therefore, we provide a general loss term L RL-Sym = L ps + β L ss that can be used with POMO or similar methods for specific CO problems (TSP and CVRP).
If the problem has pre-identified solution symmetricity (TSP) or has a strong cyclic nature (CVRP), we can set β = 1 to leverage solution-symmetricity more.
If we do not have specific domain knowledge for the target task, then we leave β = 0 , to leverage problem-symmetricity and solution-symmetricity simultaneously using only L ps .
C.3 Multi-start Post-processing
To sample multi solutions from one solver F θ we suggest instance augmentation method following [23].
As suggested in [23], we can generate multiple samples to ablate first node selection of decoding step by N .
Moreover we can generate 8 samples to rotate with 0 , 90 , 180 , 270 degrees with reflection: 4 × 2 = 8 .
To comparison with Sym-NCO and POMO as shwon in Fig.
5 (three markers), we conduct these multi state post processing with sampling width: 1 , 100 , 100 × 8 .
We, on the other hand, suggest an extended version of the instance augmentation method of [23], using random orthogonal matrix Q .
By transforming input problem P with Q 1 , ..., Q M which are orthogonal matrices, we can sample multiple sample solutions from the M symmetric problems.
We used these strategies in PCTSP and OP by setting M = 200 .
C.4 Details of Projection Head
The projection head introduced in Section 3.2 is a simple two-layer perception with the ReLU activation function, where input/output/hidden dimensions are equals to encoder’s embedding dimension (i.e. 128).
C.5 Computing Resources and Computing Time
For training Sym-NCO, we use NVIDIA A100 GPU.
Because POMO implementation does not support GPU parallelization, we use a single GPU for the POMO + Sym-NCO.
It takes approximately two weeks to finish training POMO + Sym-NCO.
For training AM + Sym-NCO, we use 4 × GPU, which takes approximately three days to finish training.
As mentioned in Section 5.2, we use NVIDIA RTX2080Ti single GPU at the test time.
D Additional Experiments D.1 Hyperparameter Tuning of α in CVRP
We did not tune hyperparameter much in this work because training resources were limited where
Sym-NCO must be verified on several tasks and DRL-NCO architectures.
Therefore, we only contain simple hyperparameter ablation for α in CVRP (POMO + SymNCO setting).
[Figure]
c t i v e
[Figure]
This validation results shows α = 0 .
2 give slightly better performances than α = 0 . 1 , but tuning of α semms to be not sensitive.
D.2 Performance Evaluation on TSPLIB
This section gives Sym-NCO performance evaluation in the TSPLIB ( N < 250) .
Sym-NCO and the POMO is pre-trained model in N = 100 that is evaluated in Table 1.
In this experiment, we conduct multi-start sampling with sample
width M = N × 20 where the N indicates multi initial city sampling of problem size (ex., the “eil51" has N = 51 ).
The 20 indicates multi-sampling using random orthogonal matrix as we introduced in Appendix C.5.
As shown in the below table, our
Sym-NCO outperforms POMO, having a 1 . 62% optimal gap, which is extremely high performance in real-world TSPLIB evaluation compared with other NCO evaluations [19]
[Table]
[Table]
D.3 Performance Evaluation of Transferability to Large Scale Problems
This section verifies that the pre-trained model using the Sym-NCO has powerful transferability on large-scale problems.
We use the efficient-active-search (EAS) [44] as a transfer learning algorithm for large-scale TSP 3 .
In transfer learning, the number of iterations is an important factor for adaptation.
We set the iteration K = 200 as default; we provide an ablation study for few-shot learning
K ∈ { 1 , 2 , 5 , 10 } to show Sym-NCO’s few-shot adaptation capability.
Note that the pre-trained model is trained on CVRP ( N = 100 ).
As shown in Table
[Table]
our method outperforms the POMO [23] in large-scale CVRP, having only a small performance gap with LKH3.
Furthermore, our method increase few shot adaptation capabilities for large-scale tasks, where our model achieved better performances than POMO with 5 × reduced training shot K .
To sum up, our Sym-NCO can be positioned with an effective pretraining scheme that approximately imposes symmetricity and is further transferred to larger-scale tasks
[Table]
[Table]
[Table]
[Table]
D.4 Comparison with Deep Improvement Heuristic Methods
In this section, we provide performance comparison with state-of-the-art deep improvement heuristics.
As shown in Table 12, our method outperformed state-of-the-art deep improvement heuristics with the fastest speed.
Note that constructive heuristics (which include our method) and improvement heuristics are complementary and can support each other.
[Table]
We remark that the speed evaluation of Wu et al.
[16] and DACT [25] is referred to by [25] where the speed is evaluated with NVIDIA TITAN RTX.
The speed of our method is evaluated with NVIDIA
RTX 2080Ti.
D.5 Comparison with Symmetric NCO models
Previous Symmetricitcy Considered NCO methods vs. Sym-NCO.
Several studies exploited the symmetric nature of CO.
Ouyang et al.
[37] have a similar purpose to Sym-NCO in that both are
DRL-based constructive heuristics, but they give rule-based input transformation (relative position from first visited city) to satisfy equivariance.
However, our method learns to impose symmetricity approximately into the neural network with regularization loss term.
We believe our approach is a more general approach to tackling symmetricity (see Table 14) because not every task can be represented as a relative position with the first visited city.
The Hudson et al.
[38] is the extended work of Joshi et al.
[11] where graph neural network (GNN) makes a sparse graph from a fully connected input graph, and the search method figures out the feasible solution from the sparse graph.
This method is based on the supervised learning scheme that requires expert labels.
Moreover, this method does not guarantee to generate feasible solutions in hard-constraint CO tasks because the pruning process of the GNN may eliminate feasible trajectory (In TSP, it may work, but in other tasks, this method must address feasibility issues).
Regardless of this limitation, we view the line graph transformation suggested by Hudson et al.
[38] as novel and helpful in terms of symmetricity.
Ma et al.
[25] proposed a DRL-based improvement heuristic, exploiting the cyclic nature of TSP and CVRP.
The purpose of Ma et al.[25], and our Sym-NCO is different: the objective of Sym-NCO is approximately imposing symmetricity nature, but the objective of Ma et al.
[25] is to improve the iteration process of improvement heuristic with fined designed positional encoding for TSP and
CVRP.
Note that Sym-NCO (constructive method) and Ma et al.
[25] (Improvement method) are complementary and can support each other.
For example, pretrained constructive model can generate an initial high-quality solution, whereas an improvement method can iteratively improve solution quality.
Experimental Comparison.
Our Sym-NCO outperforms all the relevant related baselines as shown in Table 13.
Furthermore, shows our method covers the widest arrange of CO tasks, where it does not needs labeled data
[Table]
[Table]
[Table]