Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Abstract
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown significant merit over the conventional CO solvers as DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervised learning method).
This paper presents a novel training scheme, Sym-NCO, which is a regularizer-based training scheme that leverages universal symmetricities in various CO problems and solutions.
Leveraging symmetricities such as rotational and reflectional invariance can greatly improve the generalization capability of
DRL-NCO because it allows the learned solver to exploit the commonly shared symmetricities in the same CO problem class.
Our experimental results verify that our Sym-NCO greatly improves the performance of DRL-NCO methods in four CO tasks, including the traveling salesman problem (TSP), capacitated vehicle routing problem (CVRP), prize collecting TSP (PCTSP), and orienteering problem (OP), without utilizing problem-specific expert domain knowledge.
Remarkably,
Sym-NCO outperformed not only the existing DRL-NCO methods but also a competitive conventional solver, the iterative local search (ILS), in PCTSP at 240 × faster speed.
Introduction
Combinatorial optimization problems (COPs), mathematical optimization problems on discrete input space, have been used to solve numerous valuable applications, including vehicle routing problems (VRPs) [1, 2], drug discovery [3, 4] and semi-conductor chip design [5, 6, 7].
However, finding an optimal solution to COP is difficult due to its NP-hardness.
Therefore, computing near-optimal solutions fast is essential from a practical point of view.
Conventionally, COPs are solved by integer program (IP) solvers or hand-crafted (meta) heuristics.
Recent advances in computing infrastructures and deep learning have conceived the field of neural combinatorial optimization (NCO), a deep learning-based COP solving strategy.
Depending on the training scheme, NCO methods are generally classified into supervised learning [8, 9, 10, 11, 12] and reinforcement learning (RL) [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Depending on the solution generation scheme, NCO methods are also classified into improvement [15, 14, 13, 26, 16, 17, 23] and constructive heuristics [18, 19, 20, 21, 22, 24, 25].
Among the NCO approaches, deep RL (DRL)-based constructive heuristics (i.e., DRL-NCO) are favored over conventional approaches for two major reasons.
First, RL can be applied to train the NCO model in less explorered CO tasks because training RL does not require domain expert knowledge and supervised labels from a verified solver.
Second, it is easy to produce qualified feasible solutions because the constructive process can easily avoid constraint-violated actions [19].
Despite the strength of DRL-NCO, there exists a performance gap between the state-of-the-art [Figure] conventional heuristics and DRL-NCO.
In an effort to close the gap, there have been attempts to employ problem-specific heuristics to existing DRL-NCO methods [21, 27].
However, devising a general training scheme to improve the performance of DRL-NCO still remains challenging.
[Figure]
Symmetric Neural Combinatorial Optimization (Sym-NCO), a general scheme to universal CO problems.
Sym-NCO is a regularization-based training scheme that leverages the symmetricities commonly found in COPs to increase the performance of existing DRL-NCO methods.
Sym-NCO leverages two types of symmetricities innate in COP that are defined on the Euclidean graph.
First, the problem symmetricity is derived from the rotational invariance of the solution; the rotated graph must exhibit the same optimal solution as the original graph as shown in Fig.
2a.
Second, the solution symmetricity refers to the property that solutions have identical output values (See Fig. 2b).
To train an effective NCO solver while leveraging the symmetricities, we employ REINFORCE algorithm with the baselines terms specially designed to impart solution and problem symmetricities.
REINFORCE algorithm is used because the well-known effective NCO solvers are trained by REINFORCE; thus, we can improve such solvers by just modifying their baseline with our symmetricity-considered baseline terms.
Specifically, we sample multiple solutions from the transformed problems and use the average return of them.
Then, REINFORCE pushes each solution sampled by the solver to excel this baseline during training, thus improving the policy and making all the solutions the same, i.e., the problem and solution symmetricities are realized.
Motivation for learning symmetricity.
Leveraging symmetricity is important to train CO models for two major reasons.
Firstly, symmetricity is a strong inductive bias that can support the training process of DRL by making compact training space as shown in Fig.
Secondly, learning symmetricity is beneficial to increasing generalization capability for unseen CO problems because symmetricity induces the invariant representation that every COP contains.
Novelty.
The major novelty of the proposed learning strategy is that it can easily improve existing powerful CO models; existing equivariant neural network schemes must be re-designed at the architecture level for adapting to CO.
Symmetricity in Combinatorial Optimization Markov Decision Process
This section presents several symmetric characteristics found in combinatorial optimization, which is formulated in the Markov decision process.
The objective of NCO is to train the θ -parameterized solver F θ by solving the following problem:
[Equation]
where P =
( x , f ) is a problem instance with the N node coordinates x = { x i } Ni =1 and corresponding
N features f = { f i } Ni =1 .
The ρ is a problem generating distribution. The π = { π i } Ni =1 is a solution where each element is index value π i ∈ { 1 , ..., N } for coordinates x
= { x i } Ni =1 .
The R ( π ; P ) is objective value for π on problem P .
For example, assume that we solve TSP with five cities (i.e. N = 5 ).
Then the problem instance
P contains five city coordinates x
= { x i } 5 i =1 where the salesman must visit. The solution π is a sequence of city indices; if π = { 1 , 3 , 2 , 5 , 4 } , the salesman visits city coordinates as x 1 → x 3 → x 2 → x 5 → x 4 → x 1 (the salesman must go back to first visited city to complete a tour). In TSP, each city contains the homogeneous features f . Thus, objective R of the TSP is defined as the negative of tour length:
R ( π ; P ) = − (cid:0) (cid:80)
[Equation]
{+=
+}
||
x
π
[Equation]
1 − x π i ||
+ || x [Equation] 5 − x π 1 || (cid:1) . This combinatorial decision process can be expressed as Markov decision process, and the solver F θ ( π | P ) can be expressed as instance conditioned policy .
To this end, we can utilize deep reinforcement learning for training solver F θ ( π | P ) .
We formally define the Markov decision process for CO in the below chapter.
2.1 Combinatorial optimization Markov decision process
We define the combinatorial optimization Markov decision process (CO-MDP) as the sequential construction of a solution of COP.
For a given P , the components of the corresponding CO-MDP are defined as follows:
• State.
The state s t =
( a 1: t , x , f ) is the t -th (partially complete) solution, where a 1: t represents the previously selected nodes.
The initial and terminal states s 0 and s T are equivalent to the empty and completed solution, respectively.
In this paper, we denote the solution π ( P ) as the completed solution. • Action.
The action a t is the selection of a node from the un-visited nodes (i.e., a t ∈
A t = {{ 1 , ..., N } \ { a 1: t − 1 }} ).
• Reward.
The reward function R ( π ; P ) maps the objective value from given π of problem P .
We assume that the reward is a function of a 1: T (solution sequences), || x i − x
j || i,j ∈{ 1 ,...N } (relative distances) and f (nodes features).
In TSP, capacitated VRP (CVRP), and prize collecting TSPs (PCTSP), the reward is the negative of the tour length.
In orienteering problem (OP), the reward is the sum of the prizes.
Having defined CO-MDP, we define the solution mapping as follows:
π
[Equation]
where p θ ( a t | s t ) is the policy that produces a t at s t , and T is the maximum number of states in the solution construction process.
2.2 Symmetricities in CO-MDP
Symmetricities are found in various COPs.
We conjecture that imposing those symmetricities on F θ improves the generalization and sample efficiency of F θ .
We define the two identified symmetricities that are commonly found in various COPs:
Definition 2.1 ( Problem Symmetricity ) .
Problem P i and P j are problem symmetric ( P i sym ←−→
P j ) if their optimal solution sets are identical.
Definition 2.2 ( Solution Symmetricity ) .
Two solutions π i and π j are solution symmetric ( π i sym ←−→ π j ) on problem P if R ( π i ; P )
=
R ( π j ; P ) .
An exemplary problem symmetricity found in various COPs is the rotational symmetricity:
Theorem 2.1 ( Rotational symmetricity ) .
For any orthogoanl matrix Q , the problem P and Q ( P ) ≜ {{ Qx i } Ni =1 , f } are problem symmetric: i.e., P sym ←−→ Q ( P ) .
See Appendix A for the proof.
Rotational problem symmetricity is identified in every Euclidean COPs.
On the other hand, solution symmetricity cannot be identified easily as the properties of the solutions are distinct for every COP.
Symmetric Neural Combinatorial Optimization
This section presents Sym-NCO, an effective training scheme that leverages the symmetricities of
COPs.
Sym-NCO learns a solve F θ by minimizing the total loss function:
[Equation]
where L Sym-RL is REINFORCE loss term supported by symmetricity and L inv is the regularization loss to induce that invariant representation.
The
L Sym-RL is composed with L ss , the REINFORCE loss term of Eq.
(1) with solution symmetricity regularizing mechanism, and L ps , the REINFORCE loss term of Eq.
(1) with both solution and problem symmetricity regularization.
α, β ∈ [0 , 1] are the weight coefficients.
In the following subsections, we explain each loss term in detail.
3.1 Regularizing REINFORCE with problem and solution symmetricities via L Sym-RL
As discussed in Section 2.2, COPs have problem and solution symmetricities.
We explain how to learn solver F θ using REINFORCE with the specially designed baseline to approximately impose symmetricities.
We provide the policy gradients to L ss ( π ( P )) and L ps ( π ( P )) in the context of the
REINFORCE algorithm [28] with the proposed baseline schemes.
Leveraging solution symmetricity.
As defined in Definition 2.2, the symmetric solutions must have the same objective values.
We propose the REINFORCE loss L ss with the baseline specially designed to exploit the solution symmetricity of CO as follows:
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
where { π k } Kk =1 are the solutions of P sampled from F θ ( π | P ) , log F θ is the log-likelihood of F θ , K is the number of sampled solutions, b ( P ) is a shared baseline which is the average reward from K solutions for the identical problem P .
The L ss approximately imposes solution symmetricity using REINFORCE algorithm with a novel baseline b ( P ) .
The sum of advantage in the solution group { π k } Kk =1 is always zero:
[Equation]
The L ss induces competition among the rewards, R ( π 1 ; P ) , ..., R ( π K ; P ) which can be seen as a zero-sum game.
Therefore, L ss improves the overall reward quality of the solution group using the proposed competitive REINFORCE scheme, making the solver generate high-rewarded solutions but a small reward deviation between solutions.
The small reward-deviation indicates L ss approximately imposes solution symmetricity to solver F θ .
The POMO [21] employed a similar training technique with our L ss , that finds symmetric solutions by forcing F θ to visit all possible initial cities when solving TSP and CVRP.
However, the reward of
COPs, including CVRP, PCTSP, and OP, is usually sensitive to first-city selection.
Therefore, POMO can be an excellent standalone method for TSP but can be further improved using our L ss loss term in other tasks including CVRP.
Leveraging problem symmetricity.
As discussed in Section 2.2, the rotational problem symmetricity is common in various COPs.
We propose the REINFORCE loss L ps which is equipped with problem symmetricity:
[Equation]
where Q is the distribution of random orthogonal matrices, Q l is the l th sampled rotation matrix, and π l,k is the k th sample solution of the l th rotated problem.
We construct L problem symmetric problems, Q 1 ( P ) , ..., Q L ( P ) , by using the sampled rotation matrices, and smaple K symmetric solutions from each of the L problems.
Then, the shared baseline b ( P , Q ) is constructed by averaging
L × K solutions.
Similar to the regularization scheme of L ss , the advantage term of L ps also induces competition between solutions sampled from rotationally symmetric problems.
Since the rotational symmetricity is defined such that x and Q l ( x ) have the same solution, the negative advantage value forces the solver to find a better solution.
As mentioned in Section 2.2, problem symmetricity in COPs is usually pre-identified (i.e. there is provable guaranteed symmetricity such as rotational symmetricity
Theorem 2.1)
; L ps are applicable to general COPs.
Moreover, multiple solutions are sampled for each symmetric problem so that L ps can also identify and exploit the solution symmetricity with a similar approach taken for L ss .
We provide detailed implementation and design guides regarding the integration strategy of L ss and L ps in Appendix C.2.
3.2 Learning invariant representation with Pre-identified Symmetricity: L inv .
By Theorem 2.1, the original problem x and its rotated problem Q ( x ) have identical solutions.
Therefore the encoder of F θ can be enforced to have invariant representation by leveraging the pre-identified symmetricity: rotation symmetricity.
We denote h ( x ) and h ( Q ( x )) as the hidden representations of x and Q ( x ) , respectively.
To impose the rotational invariant property on h ( x ) , we train solver F θ with an additional regularization loss term L inv defined as:
[Equation]
where S cos ( a, b ) is the cosine similarity between a and b .
g is the MLP-parameterized projection head.
For learning the invariant representation on rotational symmetricity, we penalize the difference between the projected representation g ( h ( x )) and g ( h ( Q ( x ))) , instead of directly penalizing the difference between h ( x ) and h ( Q ( x ) .
This penalizing scheme allows the use of an arbitrary encoder network architecture while maintaining the diversity of h [29].
We empirically verified that this approach attains stronger solvers as described in Section 6.1.
Related Works
Deep construction heuristics. Bello et al.
[18] propose one of the earliest DRL-NCO methods, based on pointer network (PointerNet) [8], and trained it with an actor-critic method.
Attention model (AM) [19] successfully extends [18] by swapping PointerNet with Transformer [30], and it is currently the de-facto standard method for NCO.
Notably, AM verifies its problem agnosticism by solving several classical routing problems and their practical extensions [7, 17].
The multi-decoder AM (MDAM)
[24] extends AM by employing an ensemble of decoders.
However, such an extension is inapplicable for stochastic routing problems.
The policy optimization for multiple optimal (POMO) [21] extends AM by exploiting the solution symmetricities in TSP and CVRP. Even though POMO shows significant improvements from AM, it relies on problem-specific solution symmetricities for TSP. Our method can be seen as a general-purpose symmetric learning scheme, extended from POMO, which can be applied in more general CO tasks.
Equivariant deep learning. In deep learning, symmetricities are often enforced by employing specific network architectures.
Niu et al.
[31] proposes a permutation equivariant graph neural network (GNN) that produces equivariant outputs to the input order permutations.
The SE (3) -Transformer [32] restricts the Transformer so that it is equivariant to SE (3) group input transformation.
Similarly, equivariant GNN (EGNN) [33] proposes a GNN architecture that produces O ( n ) group equivariant output.
These network architectures can dramatically reduce the search space of the model parameters.
Some research applies equivariant neural networks to RL tasks to improve sample efficiency [34].
Also, there are several works that exploited the symmetric nature of CO.
Ouyang et al.
[35] proposed equivariant encoding scheme by giving rule-based input transformation to input graph.
Hudson et al.
[36] suggested a line-graph embedding scheme, which is beneficial to process CO graphs with rotational equivariant.
Our Sym-NCO is a regularization method that is capable of learning existing powerful CO models without giving rule-based hard constraints to the structure.
We empirically study the benefit of Sym-NCO over other symmetricity-based approaches in Section 6.1 and Appendix [-D.
-]D.5.
Experiments
This section provides the experimental results of Sym-NCO for TSP, CVRP, PCTSP, and OP.
Focusing on the fact that Sym-NCO can be applied to any encoder-decoder-based NCO method, we implement
Sym-NCO on top of POMO [21] to solve TSP and CVRP, and AM [19] to solve PCTSP and OP, respectively.
We additionally validate the effectiveness of Sym-NCO on PointerNet [8] at TSP ( N = 100 ).
5.1 Tasks and baseline selections
TSP aims to find the Hamiltonian cycle with a minimum tour length.
We employ Concorde [37] and LKH-3 [38] as the non-learnable baselines, and PointerNet [8], the structured-to-vector deepQ-network (S2V-DQN) [39], AM [19], POMO [21] and MDAM [24] as the neural constructive baselines.
CVRP is an extension of TSP that aims to find a set of tours with minimal total tour lengths while satisfying the capacity limits of the vehicles.
We employ LKH-3 [38] as the non-learnable baselines, and Nazari et al.
[20], AM [19], POMO [21] and MDAM [24] as the constructive neural baselines.
PCTSP is a variant of TSP that aims to find a tour with minimal tour length while satisfying the prize constraints.
We employ the iterative local search (ILS) [19] as the non-learnable baseline, and AM [19] and MDAM [24] as the constructive neural baselines.
OP is a variant of TSP that aims to find the tour with maximal total prizes while satisfying the tour length constraint.
We employ compass [40] as the non-learnable baseline, and AM [19] and MDAM [24] as the constructive neural baselines.
5.2 Experimental setting
Problem size.
We provide the results of problems with N = 100 for the four problem classes, and real-world TSP problems with 50 < N < 250 from TSPLIB.
Hyperparameters. We apply Sym-NCO to POMO, AM, and PointerNet.
To make fair comparisons, we use the same network architectures and training-related hyperparameters from their original papers to train their Sym-NCO-augmented models.
Please refer to Appendix Appendix C.1 for more details.
Dataset and Computing Resources. We use the benchmark dataset [19] to evaluate the performance of the solvers.
To train the neural solvers, we use Nvidia A100 GPU.
To evaluate the inference speed, we use an Intel Xeon E5-2630 CPU and Nvidia RTX2080Ti GPU to make fair comparisons with the existing methods as proposed in [24].
5.3 Performance metrics
This section provides detailed performance metrics:
Average cost.
We report an average cost of 10,000 benchmark instances which is proposed by [19]. [Table] [Table]
Evaluation speed.
We report the evaluation speeds of solvers in a out-of-the-box manner as they are used in practice.
In that regard, the execution time of non-neural and neural methods are measured on
CPU and GPU, respectively.
Greedy/Multi-start performance.
For neural solvers, it is a common practice to measure multi-start performance as its final performance.
However, when those are employed in practice, such resource consuming multi-start may not be possible.
Hence, we discuss greedy and multi-start separately.
5.4 Experimental results
Results of TSP and CVRP. As shown in Table 1, Sym-NCO outperforms the NCO baselines in both the greedy rollout and multi-start settings with the fastest inference speed.
Remarkably,
Sym-NCO achieves a 0 .
95% gap in TSP using the greedy rollout.
In the TSP greedy setting, it solves
TSP 10,000 instances in a few seconds.
Results of PCTSP and OP. As shown in Table 2, Sym-NCO outperforms the NCO baselines in both the greedy rollout and multi-start settings.
In the multi-start setting, Sym-NCO outperforms the classical PCTSP baseline (i.e., ILS) with the 43200180 ≈ 240 × faster speed. [Figure] [Figure] Gap
POMO 1.87%
Sym
[Table]
Results of the real-world TSP. We evaluate POMO and Sym-NCO on
TSPLib [41].
Table 3 shows that Sym-NCO outperforms POMO.
Please refer to Appendix D.2 for the full benchmark results.
Application to various DRL-NCO methods.
As discussed in Section 3,
Sym-NCO can be applied to various various DRL-NCO methods.
We validate that Sym-NCO significantly improves the existing DRL-NCO methods as shown in Fig.
Time-performance analysis for multi-starts. Multi-starts is a common method that improves the solution qualities while requiring a longer time budget.
We use the rotation augments [21] to produce multiple inputs (i.e., starts).
As shown in Fig.
5, Sym-NCO achieves the Pareto frontier for all benchmark datasets.
In other words, Sym-NCO exhibits the best solution quality among the baselines within the given time consumption.
Discussion 6.1 Discussion of Regularization based Symmetricity Learning
Ablation Study of L inv .
As shown in Fig.
6b, L inv increases the cosine similarity of the projected representation (i.e., g ( h ) ).
We can conclude that L inv contributes to the performance improvements (see Fig. 6a).
We further verify that imposing similarity on h degrades the performance as demonstrated in Fig.
6c.
This proves the importance of maintaining the expression power of the encoder as we mentioned in Section 3.2.
Comparison with EGNN. EGNN [33] provably guarantees to process coordinate graph with rotational equivalency.
Also, EGNN empirically verified its’ high performance in point cloud tasks.
Therefore, we implemented a simple EGNN-based CO policy (simply termed EGNN in this paper), to check the feasibility of CO.
We leverage six EGNN layers with 128 hidden dimensions, to replace the POMO encoder where the POMO decoder is unchanged.
In the experimental results, we observed that EGNN significantly underperforms Sym-NCO and fails to converge as shown Fig.
7b.
This is because the euclidian CO has a fully connected input graph containing informative coordinates, and we believe the equivariant neural network should be carefully crafted to consider such unique input graph structures of CO tasks.
On the other hand, Sym-NCO could leverage the existing powerful NCO model without fine modification of the neural network.
These numerical results conform well with our hypothesis that equivariance is necessary but not [Figure] [Figure] sufficient for obtaining the optimal parameters for the solver.
Therefore
[Figure]
we expect our approach can be simply extended to other domains requiring symmetricity and geometricity as positioned to leverage the existing legacy of powerful non-equivariant neural networks [Figure]
6.2 Limitations & future directions
Extended problem symmetricities.
In this work [Figure] we employ the rotational symmetricity (Theorem 2.1) as the problem symmetricity.
However, for some COPs, different problem symmetricities, such as scaling and translating P , can also be considered.
Employing these additional symmetricities may further enhance the performance of Sym-NCO.
We leave this for future research.
Large scale adaptation.
Large scale applicability is essential to NCO. In this work we simply present scale adaptation capability of Sym-NCO using the effective active search (EAS) [42]; see Appendix D.3.
We expect curriculum- [43], and meta-learning approaches may improve the generalizability of NCO to larger-sized problems.
Extension to the graph COP.
This work finds the problem symmetricity that is universally applicable for Euclidean COPs.
However, some COPs are defined in non-Euclidean spaces such as asymmetric TSP.
Furthermore, there are also a bunch of existing neural combinatorial optimization models that can solve graph COP [44, 45, 46, 47, 48, 49], where we can improve with a symmetricity regularization scheme.
We also leave finding the universal symmetricities of non-Euclidean COPs and applying them to existing graph COP models for future research. 6.3 Social Impacts
Design automation through NCO research affects various industries including logistics and transportation industries.
From a negative perspective, this automation process can lead to unemployment in certain jobs.
However, automation of logistics, transportation, and design automation can increase the efficiency of industries, reducing CO2 emissions (by reducing total tour length) and creating new industries and jobs.
Acknowledgments and Disclosure of Funding
We thank Jiwoo Son, Hyeonah Kim, Haeyeon Rachel Kim, Fangying Chen and anonymous reviews for proving helpful feedback for preparing our manuscripts. This work was supported by a grant of the KAIST-KT joint research project through AI2XL Laboratory, Institute of convergence Technology, funded by KT [Project No. G01210696, Development of Multi-Agent Reinforcement Learning Algorithm for Efficient Operation of Complex Distributed Systems].