Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown significant merit over the conventional CO solversas DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervisedlearning method).
This paper presents a novel training scheme, Sym-NCO, whichis a regularizer-based training scheme that leverages universal symmetricities invarious CO problems and solutions.
Leveraging symmetricities such as rotationaland reflectional invariance can greatly improve the generalization capability of
DRL-NCO because it allows the learned solver to exploit the commonly sharedsymmetricities in the same CO problem class.
Our experimental results verifythat our Sym-NCO greatly improves the performance of DRL-NCO methods infour CO tasks, including traveling salesman problem (TSP), capacitated vehiclerouting problem (CVRP), prize collecting TSP (PCTSP), and orienteering problem(OP), without utilizing problem-specific expert domain knowledge.
Remarkably,
Sym-NCO outperformed not only the existing DRL-NCO methods but also a competitive conventional solver, the iterative local search (ILS), in PCTSP at 240 ×faster speed.
Source code will be available after the decision is made.
Introduction
Combinatorial optimization problems (COPs), mathematical optimization problems on discrete inputspace, have been used to solve numerous valuable applications, including vehicle routing problems(VRPs)
[1, 2], drug discovery [3, 4] and semi-conductor chip design [5, 6, 7].
However, findingan optimal solution to COP is difficult due to its NP-hardness.
Therefore, computing near-optimalsolutions fast is essential from a practical point of view.
Conventionally, COPs are solved by integer program (IP) solvers or hand-crafted (meta) heuristics.
Recent advances in computing infrastructures and deep learning have conceived the fieldof neural combinatorial optimization (NCO), a deep learning-based COP solving strategy.
Depending on the training scheme, NCO methods are generally classified into supervised learning[8, 9, 10, 11, 12] and reinforcement learning (RL) [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Depending on the solution generation scheme, NCO methods are also classified into improvement [15, 14, 13, 26, 16, 17, 23] and constructive heuristics [18, 19, 20, 21, 22, 24, 25].
Amongthe NCO approaches, deep RL (DRL)-based constructive heuristics (i.e., DRL-NCO) are favoredover conventional approaches for two major reasons.
First, RL can be applied to train the NCOmodel in less explorered CO tasks because training RL does not require domain expert
knowl-
[Figure]
edge and supervised labels from a verified solver.
Second, it is easy to produce qualified feasible solutions because the constructive process can easily avoid constraint-violated actions [19].
Despite the strength of DRL-NCO, there exists a performance gap between the state-of-the-artconventional heuristics and DRL-NCO.
In an effort to close the gap, there have been attempts toemploy problem-specific heuristics to existing DRL-NCO methods [21, 27].
However, devisinga general training scheme to improve the performance of DRL-NCO still remains challenging.
[Figure]
Sym-NCO makes a more compact training space compared with previous DRLNCO
In this study, we propose the Symmetric Neural Combinatorial Optimization (Sym-NCO), a general trainingscheme applicable to universal CO problems.
Sym-NCOis a regularization-based training scheme that leverages thesymmetricities commonly found in COPs to increase theperformance of existing DRL-NCO methods.
Sym-NCOleverages two types of symmetricities innate in COP thatare defined on the Euclidean graph.
First, the problemsymmetricity is derived from the rotational invariance ofthe solution; the rotated graph must exhibit the same optimal solution as the original graph as shown in Fig.
2a.
Second, the solution symmetricity refers to the propertythat solutions have identical output values (See Fig. 2b).
To train an effective NCO solver while leveraging the symmetricities, we employ REINFORCE algorithm with thebaselines terms specially designed to impart solution andproblem symmetricities.
REINFORCE algorithm is usedbecause the well-known effective NCO solvers are trainedby REINFORCE; thus, we can improve such solvers byjust modifying their baseline with our symmetricity-considered baseline terms.
Specifically, wesample multiple solutions from the transformed problems and use the average return of them.
Then, REINFORCE pushes each solution sampled by the solver to excel this baseline during training,thus improving the policy and making all the solutions the same, i.e., the problem and solutionsymmetricities are realized.
Motivation for learning symmetricity.
Leveraging symmetricity is important to train CO models fortwo major reasons.
Firstly, symmetricity is a strong inductive bias that can support the training processof DRL by making compact training space as shown in Fig.
Secondly, learning symmetricityis beneficial to increasing generalization capability for unseen CO problems because symmetricityinduces the invariant representation that every COP contains.
Novelty.
The major novelty of the proposed learning strategy is that it can easily improve existingpowerful CO models such as AM and POMO, both performances on specific problem scales (Tables 1and 2 and generalization capability on larger scale problems (Tables 10 and 11).
Symmetricity in Combinatorial Optimization Markov Decision Process
This section presents several symmetric characteristics found in combinatorial optimization, which isformulated in the Markov decision process.
The objective of NCO is to train the θ -parameterized solver F θ by solving the following problem:
[Equation]
(1) where P =
( x , f ) is a problem instance with the N node coordinates x = { x i } Ni =1 and corresponding
N features f = { f i } Ni =1 .
The ρ is a problem generating distribution, π ( P ) is a solution of P , and
R ( π ( P )) is the objective value of π ( P ) .
2.1 Combinatorial optimization Markov decision process
We define the combinatorial optimization Markov decision process (CO-MDP) as the sequentialconstruction of a solution of COP.
For a given P , the components of the corresponding CO-MDP aredefined as follows:
• State.
The state s t =
( a 1: t , x , f ) is the t -th (partially complete) solution, where a 1: t representsthe previously selected nodes.
The initial and terminal states s 0 and s T are equivalent to theempty and completed solution, respectively.
In this paper, we denote the solution π ( P ) as thecompleted solution.• Action.
The action a t is the selection of a node from the un-visited nodes (i.e., a t ∈
A t ={{ 1 , ..., N } \ { a 1: t − 1 }} ).•
Reward.
The reward R ( π ( P )) is the objective of COP.
We assume that the reward is a functionof a 1: T (solution sequences), || x i − x
j || i,j ∈{ 1 ,...N } (relative distances) and f (nodes features).
InTSP, capacitated VRP (CVRP), and prize collecting TSPs (PCTSP), the reward is the negative ofthe tour length.
In orienteering problem (OP), the reward is the sum of the prizes.
Having defined CO-MDP, we define the solution mapping as follows:
π ( P [Equation] ∼ F θ ( P )
= 93 (cid:81) Tt =1 p θ ( a t | s t ( P ))
where
p
θ ( a t | s t (
P
)) is the
policy
that produces a t at s t , and T is the maxi mum number of states in the solution construction process.
2.2 Symmetricities in CO-MDP
Symmetricities are found in various COPs.
We conjecture that imposing those symmetricities on F θimproves the generalization and sample efficiency of F θ .
We define the two identified symmetricitiesthat are commonly found in various COPs:
Definition 2.1 ( Problem Symmetricity ) .
Problem P i and P j are problem symmetric ( P i sym ←−→
P j )if their optimal solution sets are identical.
Definition 2.2 ( Solution Symmetricity ) .
Two solutions of problem P ( π i ( P ) and π j ( P ) ) aresolution symmetric ( π i sym ←−→ π j ) if R ( π i ) =
R ( π j ) .
An exemplary problem symmetricity found in various COPs is the rotational symmetricity:
Theorem 2.1 ( Rotational symmetricity ) .
For any orthogoanl matrix Q , the problem P and Q ( P ) ≜{{ Qx i } Ni =1 , f } are problem symmetric: i.e., P sym ←−→ Q ( P ) .
See Appendix A for the proof.
Rotational problem symmetricity is identified in every Euclidean COPs.
On the other hand, solutionsymmetricity cannot be identified easily as the properties of the solutions are distinct for every COP.
Symmetric Neural Combinatorial Optimization
This section presents Sym-NCO, an effective training scheme that leverages the symmetricities of
COPs.
Sym-NCO learns a solve F θ by minimizing the total loss function:
[Equation]
where L Sym-RL is REINFORCE loss term supported by symmetricity and L inv is the regularizationloss to induce that invariant representation.
The L Sym-RL is composed with L ss , the REINFORCE loss term of Eq.
(1) with solution symmetricity regularizing mechanism, and L ps , the REINFORCEloss term of Eq.
(1) with both solution and problem symmetricity regularization.
α, β ∈ [0 , 1] are theweight coefficients.
In the following subsections, we explain each loss term in detail.
3.1 Regularizing REINFORCE with problem and solution symmetricities via L Sym-RL
As discussed in Section 2.2, COPs have problem and solution symmetricities.
We explain how tolearn solver F θ using REINFORCE with the specially designed baseline to approximately imposesymmetricities.
We provide the policy gradients to L ss ( π ( P )) and L ps ( π ( P )) in the context of the
REINFORCE algorithm [28] with the proposed baseline schemes.
Leveraging solution symmetricity.
As defined in Definition 2.2, the symmetric solutions must havethe same objective values.
We propose the REINFORCE loss L ss with the baseline specially designedto exploit the solution symmetricity of CO as follows:
[Equation]
[Equation] [Equation] (5) where { π k } Kk =1 are the solutions of P sampled from F θ ( π | P ) , log F θ is the log-likelihood of F θ , Kis the number of sampled solutions, b ( P ) is a shared baseline which is the average reward from Ksolutions for the identical problem P .
The L ss approximately imposes solution symmetricity using REINFORCE algorithm with a novelbaseline b ( P ) .
The sum of advantage in the solution group { π k } Kk =1 is always zero:
[Equation]
The L ss induces competition among the rewards, R ( π 1 ) , ..., R ( π K ) which can be seen as a zero-sumgame.
Therefore, L ss improves the overall reward quality of the solution group using the proposedcompetitive REINFORCE scheme, making the solver generate high rewarded solutions but a smallreward deviation between solutions.
The small reward-deviation indicates L ss approximately imposessolution symmetricity to solver F θ .
The POMO [21] employed a similar training technique with our L ss , that finds symmetric solutionsby forcing F θ to visit all possible initial cities when solving TSP and CVRP.
However, the reward of
COPs, including CVRP, PCTSP, and OP, is usually sensitive to first city selection.
Therefore, POMOcan be an excellent standalone method for TSP but can be further improved using our L ss loss term.
Leveraging problem symmetricity.
As discussed in Section 2.2, the rotational problem symmetricityis common in various COPs.
We propose the REINFORCE loss L ps which is equipped with problem symmetricity:
[Equation]
where Q is the distribution of random orthogonal matrices, Q l is the l th sampled rotation matrix,and π l,k is the k th sample solution of the l th rotated problem.
We construct L problem symmetricproblems, Q 1 ( P ) , ..., Q L ( P ) , by using the sampled rotation matrices, and smaple K symmetricsolutions from each of the L problems.
Then, the shared baseline b ( P , Q ) is constructed by averaging
L × K solutions.
Similar to the regularization scheme of L ss , the advantage term of L ps also induces competitionbetween solutions sampled from rotationally symmetric problems.
Since the rotational symmetricityis defined such that x and Q l ( x ) have the same solution, the negative advantage value forces thesolver to find a better solution.
As mentioned in Section 2.2, problem symmetricity in COPs isusually pre-identified (i.e. there is provable guaranteed symmetricity such as rotational symmetricity
Theorem 2.1)
; L ps are applicable to general COPs.
Moreover, multiple solutions are sampled foreach symmetric problem so that L ps can also identify and exploit the solution symmetricity with asimilar approach taken for L ss .
We provide detailed implementation and design guides regarding theintegration strategy of L ss and L ps in Appendix C.2.
3.2 Learning invariant representation with Pre-identified Symmetricity: L inv .
By Theorem 2.1, the original problem x and its rotated problem Q ( x ) have identical solutions.
Therefore the encoder of F θ can be enforced to have invariant representation by leveraging thepre-identified symmetricity: rotation symmetricity.
We denote h ( x ) and h ( Q ( x )) as the hidden representations of x and Q ( x ) , respectively.
To imposethe rotational invariant property on h ( x ) , we train solver F θ with an additional regularization lossterm L inv defined as:
[Equation]
where S cos ( a, b ) is the cosine similarity between a and b .
g is the MLP-parameterized projectionhead.
For learning the invariant representation on rotational symmetricity, we penalize the differencebetween the projected representation g ( h ( x )) and g ( h ( Q ( x ))) , instead of directly penalizing thedifference between h ( x ) and h ( Q ( x ) .
This penalizing scheme allows the use of an arbitrary encodernetwork architecture while maintaining the diversity of h [29].
We empirically verified that thisapproach attains stronger solvers as described in Section 6.1.
Related Works
Deep construction heuristics Bello et al.
[18] propose one of the earliest DRL-NCO methods,based on pointer network (PointerNet) [8], and trained it with an actor-critic method.
Attention model(AM) [19] successfully extends [18] by swapping PointerNet with Transformer [30], and it is currentlythe de-facto standard method for NCO.
Notably, AM verifies its problem agnosticism by solvingseveral classical routing problems and their practical extensions [7, 17].
The policy optimizationfor multiple optimal (POMO) [21] extends AM by exploiting the solution symmetricities in TSPand CVRP.
Even though POMO shows significant improvements from AM, it relies on problemspecific solution symmetricities (i.e., not problem agnostic).
The multi-decoder AM (MDAM) [24] extends AM by employing an ensemble of decoders.
However, such an extension is inapplicablefor stochastic routing problems (i.e., not problem agnostic).
Among these promising DRL-NCOmethods, Sym-NCO achieved SOTA performances with problem-agnostic properties.
Equivariant deep learning In deep learning, symmetricities are often enforced by employing specific network architectures.
Niu et al. [31] proposes a permutation equivariant graph neural network(GNN) that produces equivariant outputs to the input order permutations.
The SE (3) -Transformer[32] restricts the Transformer so that it is equivariant to SE (3) group input transformation.
Similarly,equivariant GNN (EGNN) [33] proposes a GNN architecture that produces O ( n ) group equivariantoutput.
These network architectures can dramatically reduce the search space of the model parameters.
Some research applies equivariant neural networks to RL tasks to improve sample efficiency [34].
Also, there are several works that exploited the symmetric nature of CO.
Ouyang et al.
[35] proposedequivariant encoding scheme by giving rule-based input transformation to input graph.
Hudson etal.
[36] suggested a line-graph embedding scheme, which is beneficial to process CO graphs withrotational equivariant.
Our Sym-NCO is a regularization method that is capable of learning existingpowerful CO models without giving rule-based hard constraints to the structure.
We empirically studythe benefit of Sym-NCO over other symmetricity-based approaches in Section 6.1 and Appendix D.
Experiments
This section provides the experimental results of Sym-NCO for TSP, CVRP, PCTSP, and OP.
Focusingon the fact that Sym-NCO can be applied to any encoder-decoder-based NCO method, we implement
Sym-NCO on top of POMO [21] to solve TSP and CVRP, and AM [19] to solve PCTSP and OP,respectively.
We additionally validate the effectiveness of Sym-NCO on PointerNet [8] at TSP( N = 100 ).
5.1 Tasks and baseline selections
TSP aims to find the Hamiltonian cycle with a minimum tour length.
We employ Concorde [-[37] and LKH-
-][37]and LKH-3 [38] as the non-learnable baselines, and PointerNet [8], the structured-to-vector deepQ-network (S2V-DQN) [39], AM [19], POMO [21] and MDAM [24] as the neural constructivebaselines.
CVRP is an extension of TSP that aims to find a set of tours with minimal total tour lengths whilesatisfying the capacity limits of the vehicles.
We employ LKH-3 [38] as the non-learnable baselines,and Nazari et al. [20], AM [19], POMO [21] and MDAM [24] as the constructive neural baselines.
PCTSP is a variant of TSP that aims to find a tour with minimal tour length while satisfying the prizeconstraints.
We employ the iterative local search (ILS) [19] as the non-learnable baseline, and AM[19] and MDAM [24] as the constructive neural baselines.
OP is a variant of TSP that aims to find the tour with maximal total prizes while satisfying the tourlength constraint.
We employ compass [40] as the non-learnable baseline, and AM [19] and MDAM[24] as the constructive neural baselines.
5.2 Experimental setting
Problem size.
We provide the results of problems with N = 100 for the four problem classes, andreal-world TSP problems with 50 < N < 250 from TSPLIB.
Hyperparameters We apply Sym-NCO to POMO, AM, and PointerNet.
To make fair comparisons,we use the same network architectures and training-related hyperparameters from their original papersto train their Sym-NCO-augmented models.
Please refer to Appendix Appendix C.1 for more details.
Dataset and Computing Resources We use the benchmark dataset [19] to evaluate the performanceof the solvers.
To train the neural solvers, we use Nvidia A100 GPU.
To evaluate the inference speed,
[Table]
we use an Intel Xeon E5-2630 CPU and Nvidia RTX2080Ti GPU to make fair comparisons with theexisting methods as proposed in [24].
5.3 Performance metrics
This section provides detailed performance metrics:
Average cost.
We report an average cost of 10,000 benchmark instances which is proposed by [19].
Evaluation speed.
We report the evaluation speeds of solvers in a out-of-the-box manner as they areused in practice.
In that regard, the execution time of non-neural and neural methods are measured on
CPU and GPU, respectively.
Greedy/Multi-start performance.
For neural solvers, it is a common practice to measure multi-startperformance as its final performance.
However, when those are employed in practice, such resourceconsuming multi-start may not be possible.
Hence, we discuss greedy and multi-start separately.
5.4 Experimental results
Results of TSP and CVRP As shown in Table 1, Sym-NCO outperforms the NCO baselinesin both the greedy rollout and multi-start settings with the fastest inference speed.
Remarkably,
Sym-NCO achieves a 0 .
95% gap in TSP using the greedy rollout.
In the TSP greedy setting, it solves
TSP 10,000 instances in a few seconds.
Results of PCTSP and OP As shown in Table 2, Sym-NCO outperforms the NCO baselines inboth the greedy rollout and multi-start settings.
In the multi-start setting, Sym-NCO outperforms theclassical PCTSP baseline (i.e., ILS) with the 43200180 ≈ 240 × faster speed.
[Table]
Results of the real-world TSP We evaluate POMO and Sym-NCO on
TSPLib [41].
Table 3 shows that Sym-NCO outperforms POMO.
Please referto Appendix D.2 for the full benchmark results.
Application to various DRL-NCO methods As discussed in Section 3,
Sym-NCO can be applied to various various DRL-NCO methods.
We validate [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
that Sym-NCO significantly improves the existing DRL-NCO methods asshown in Fig.
Time-performance analysis for multi-starts Multi-starts is a common method that improvesthe solution qualities while requiring a longer time budget.
We use the rotation augments [21] toproduce multiple inputs (i.e., starts).
As shown in Fig.
5, Sym-NCO achieves the Pareto frontierfor all benchmark datasets.
In other words, Sym-NCO exhibits the best solution quality among thebaselines within the given time consumption.
Discussion
6.1 Discussion of Regularization based Symmetricity LearningAblation Study of L inv
As shown in Fig.
6b, L inv increases the cosine similarity of the projectedrepresentation (i.e., g ( h ) ).
We can conclude that L inv contributes to the performance improvements(see Fig.
6a).
We further verify that imposing similarity on h degrades the performance as demonstrated in Fig.
6c.
This proves the importance of maintaining the expression power of the encoder aswe mentioned in Section 3.2.
Comparison with EGNN EGNN [33] provably guarantees to process coordinate graph withrotational equivalency.
Also, EGNN empirically verified its’ high performance in point cloud tasks.
However, we observed that EGNN significantly underperforms than Sym-NCO and fails to convergeas shown Fig.
7b.
This is because the euclidian CO has a fully connected input graph containinginformative coordinates, and we believe the equivariant neural network should be carefully craftedto consider such unique input graph structures of CO tasks.
On the other hand, Sym-NCO couldleverage the existing powerful NCO model without fine modification of the neural network.
Thesenumerical results conform well with our hypothesis that equivariance is necessary but not sufficientfor obtaining the optimal parameters for the solver (see Fig. 1).
Therefore, we expect our approach
[Figure] [Figure]
can be simply extended to other domains requiring symmetricity and geometricity as positioned toleverage the existing legacy of powerful non-equivariant neural networks.
6.2 Limitations & future directions
Extended problem symmetricities.
In this work, we employ the rotational symmetricity (Theorem 2.1) as the problem symmetricity.
However, for some COPs, different problem symmetricities,such as scaling and translating P , can also be considered.
Employing these additional symmetricitiesmay further enhance the performance of Sym-NCO.
We leave this for future research.
Large scale adaptation.
Large scale applicability is essential to NCO (this work solves N < 250) .
Hence, we expect that the transfer- [42], curriculum- [43], and meta-learning approaches may improvethe generalizability of NCO to larger-sized problems.
Extension to the graph COP.
This work finds the problem symmetricity that is universallyapplicable for Euclidean COPs.
However, some COPs are defined in non-Euclidean spaces such asasymmetric TSP.
Furthermore, there are also a bunch of existing neural combinatorial optimizationmodels that can solve graph COP [44, 45, 46, 47, 48, 49], where we can improve with a symmetricityregularization scheme.
We also leave finding the universal symmetricities of non-Euclidean COPsand applying them to existing graph COP models for future research.
[Figure]