Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown signiﬁcant merit over the conventional CO solversas DRL-NCO is capable of learning CO solvers less relying on problem-speciﬁc expert domain knowledge (heuristic method) and supervised labeled data (supervisedlearning method).
This paper presents a novel training scheme, Sym-NCO, whichis a regularizer-based training scheme that leverages universal symmetricities invarious CO problems and solutions.
Leveraging symmetricities such as rotationaland reﬂectional invariance can greatly improve the generalization capability of
DRL-NCO because it allows the learned solver to exploit the commonly sharedsymmetricities in the same CO problem class.
Our experimental results verifythat our Sym-NCO greatly improves the performance of DRL-NCO methods infour CO tasks, including traveling salesman problem (TSP), capacitated vehiclerouting problem (CVRP), prize collecting TSP (PCTSP), and orienteering problem(OP), without utilizing problem-speciﬁc expert domain knowledge.
Remarkably, Sym-NCO outperformed not only the existing DRL-NCO methods but also a competitive conventional solver, the iterative local search (ILS), in PCTSP at 240 ⇥faster speed.
Source code will be available after the decision is made.
Introduction
Combinatorial optimization problems (COPs), mathematical optimization problems on discrete inputspace, have been used to solve numerous valuable applications, including vehicle routing problems(VRPs) [1, 2], drug discovery [3, 4] and semi-conductor chip design [5, 6, 7].
However, ﬁndingan optimal solution to COP is difﬁcult due to its NP-hardness.
Therefore, computing near-optimalsolutions fast is essential from a practical point of view.
Conventionally, COPs are solved by integer program (IP) solvers or hand-crafted (meta) heuristics.
Recent advances in computing infrastructures and deep learning have conceived the ﬁeldof neural combinatorial optimization (NCO), a deep learning-based COP solving strategy.
Depending on the training scheme, NCO methods are generally classiﬁed into supervised learning[8, 9, 10, 11, 12] and reinforcement learning (RL) [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Depending on the solution generation scheme, NCO methods are also classiﬁed into improvement [15, 14, 13, 26, 16, 17, 23] and constructive heuristics [18, 19, 20, 21, 22, 24, 25].
Amongthe NCO approaches, deep RL (DRL)-based constructive heuristics (i.e., DRL-NCO) are favoredover conventional approaches for two major reasons. First, RL can be applied to train the NCOmodel in less explorered CO tasks because training RL does not require domain expert knowl-
[Figure]
edge and supervised labels from a veriﬁed solver. Second, it is easy to produce qualiﬁed feasible solutions because the constructive process can easily avoid constraint-violated actions [19].
Despite the strength of DRL-NCO, there exists a performance gap between the state-of-the-artconventional heuristics and DRL-NCO.
In an effort to close the gap, there have been attempts toemploy problem-speciﬁc heuristics to existing DRL-NCO methods [21, 27].
However, devisinga general training scheme to improve the performance of DRL-NCO still remains challenging.
[Figure]
Sym-NCO makes a more compact training space compared with previous DRLNCO and supports the NCO model efﬁciently converges in near-optimal pa
Equivariant neural network (ENN) scheme provably guarantees to trained in symmetric space; more expression power is needed for CO tasks.
In this study, we propose the Symmetric Neural Combinatorial Optimization (Sym-NCO), a general trainingscheme applicable to universal CO problems.
Sym-NCOis a regularization-based training scheme that leverages thesymmetricities commonly found in COPs to increase theperformance of existing DRL-NCO methods.
Sym-NCOleverages two types of symmetricities innate in COP thatare deﬁned on the Euclidean graph.
First, the problemsymmetricity is derived from the rotational invariance ofthe solution; the rotated graph must exhibit the same optimal solution as the original graph as shown in Fig.
2a.
Second, the solution symmetricity refers to the propertythat solutions have identical output values (See Fig. 2b).
To train an effective NCO solver while leveraging the symmetricities, we employ REINFORCE algorithm with thebaselines terms specially designed to impart solution andproblem symmetricities.
REINFORCE algorithm is usedbecause the well-known effective NCO solvers are trainedby REINFORCE; thus, we can improve such solversby just modifying their baseline with our symmetricityconsidered baseline terms. Speciﬁcally, we sample multiple solutions from the transformed problems and use theaverage return of them.
Then, REINFORCE pushes each solution sampled by the solver to excel thisbaseline during training, thus improving the policy and making all the solutions the same, i.e., theproblem and solution symmetricities are realized. Motivation for learning symmetricity.
Leveraging symmetricity is important to train CO models fortwo major reasons.
Firstly, symmetricity is a strong inductive bias that can support the training processof DRL by making compact training space as shown in Fig.
Secondly, learning symmetricityis beneﬁcial to increasing generalization capability for unseen CO problems because symmetricityinduces the invariant representation that every COP contains.
Novelty.
The major novelty of the proposed learning strategy is that it can be easily used to boost theperformances of the existing powerful CO models such as AM and POMO.
The existing equivariantgraph neural network (EGNN) [28] scheme also can leverage symmetricity; however, it requires a lotof additional effort to design high expressive model for solving CO tasks (See Fig. 1 and Section 6.1).
Symmetricity in Combinatorial Optimization Markov Decision Process
This section presents several symmetric characteristics found in combinatorial optimization, which isformulated in the Markov decision process.
The objective of NCO is to train the ✓ -parameterized solver F ✓ by solving the following problem:
[Equation]
[Equation]
where P =
( x , f ) is a problem instance with the N node coordinates x = { x i } Ni =1 and corresponding
N features f = { f i } Ni =1 .
The ⇢ is a problem generating distribution, ⇡ ( P ) is a solution of P , and
R ( ⇡ ( P )) is the objective value of ⇡ ( P ) .
2.1 Combinatorial optimization Markov decision process
We deﬁne the combinatorial optimization Markov decision process (CO-MDP) as the sequentialconstruction of a solution of COP.
For a given P , the components of the corresponding CO-MDP aredeﬁned as follows:
• State.
The state s t =
( a 1: t , x , f ) is the t -th (partially complete) solution, where a 1: t representsthe previously selected nodes.
The initial and terminal states s 0 and s T are equivalent to theempty and completed solution, respectively.
In this paper, we denote the solution ⇡ ( P ) as thecompleted solution.• Action.
The action a t is the selection of a node from the un-visited nodes (i
e., a t 2 A t ={{ 1 , ..., N } \ { a 1: t  1 }} ).•
Reward.
The reward R ( ⇡ ( P )) is the objective of COP.
We assume that the reward is a functionof a 1: T (solution sequences), ||
x
i  x j
|| i,j 2 { 1 ,...
N } (relative distances) and f (nodes features).
InTSP, capacitated VRP (CVRP) and prize collecting TSPs (PCTSP), the reward is the negative ofthe tour length.
In orienteering problem (OP), the reward is the sum of the prizes.
Having deﬁned CO-MDP, we deﬁne the solution mapping as follows:
⇡ ( P ) ⇠ F ✓ ( P ) = Q
Tt =1 p ✓ ( a t | s t ( P )) where p ✓ ( a t | s t ( P )) is the policy that produces a t at s t , and T is the maximum number of states in the solution construction process.
in
CO-MDP
Symmetricities are found in various COPs.
We conjecture that imposing those symmetricities on F ✓improves the generalization and sample efﬁciency of F ✓ .
We deﬁne the two identiﬁed symmetricitiesthat are commonly found in various COPs:
Deﬁnition 2.1 ( Problem Symmetricity ) .
Problem P i and P j are problem symmetric ( P i sym !
P j )if their optimal solution sets are identical.
Deﬁnition 2.2 ( Solution Symmetricity ) .
Two solutions of problem P ( ⇡ i ( P ) and ⇡ j ( P ) ) aresolution symmetric ( ⇡ i sym ! ⇡
j ) if R ( ⇡ i ) =
R ( ⇡ j ) .
An exemplary problem symmetricity found in various COPs is the rotational symmetricity:
Theorem 2.1 ( Rotational symmetricity ) .
For any orthogoanl matrix Q , the problem P and Q ( P ) ,{{ Qx i } Ni =1 , f } are problem symmetric: i.e., P sym !
Q ( P ) .
See Appendix A for the proof.
Rotational problem symmetricity is identiﬁed in every Euclidean COPs.
On the other hand, solutionsymmetricity cannot be identiﬁed easily as the properties of the solutions are distinct for every COP.
Symmetric Neural Combinatorial Optimization
This section presents Sym-NCO, an effective training scheme that leverages the symmetricities of
COPs.
Sym-NCO learns a solve F ✓ by minimizing the total loss function:
[Equation]
where L Sym-RL is REINFORCE loss term supported by symmetricity and L inv is the regularizationloss to induce that invariant representation. The L Sym-RL is composed with L ss , the REINFORCE loss term of Eq. (1) with solution symmetricity regularizing mechanism, and L ps , the REINFORCEloss term of Eq. (1) with both solution and problem symmetricity regularization.
↵ ,  2 [0 , 1] are theweight coefﬁcients.
In the following subsections, we explain each loss term in detail.
3.1 Regularizing REINFORCE with problem and solution symmetricities via L Sym-RL
As discussed in Section 2.2, COPs have problem and solution symmetricities.
We explain how tolearn solver F ✓ using REINFORCE with the specially designed baseline to approximately imposesymmetricities.
We provide the policy gradients to L ss ( ⇡ ( P )) and L ps ( ⇡ ( P )) in the context of the REINFORCE algorithm [29] with the proposed baseline schemes.
Leveraging solution symmetricity.
As deﬁned in Deﬁnition 2.2, the symmetric solutions must havethe same objective values.
We propose the REINFORCE loss L ss with the baseline specially designedto exploit the solution symmetricity of CO as follows:
[Equation]
where { ⇡ k } Kk =1 are the solutions of P sampled from F ✓ ( ⇡ | P ) , log F ✓ is the log-likelihood of F ✓ , Kis the number of sampled solutions, b ( P ) is a shared baseline which is the average reward from Ksolutions for the identical problem P .
The L ss approximately imposes solution symmetricity using REINFORCE algorithm with a novelbaseline b ( P ) .
The sum of advantage in the solution group { ⇡ k } Kk =1 is always zero:
[Equation]
The L ss induces competition among the rewards, R ( ⇡ 1 ) , ..., R ( ⇡ K ) which can be seen as a zero-sumgame. Therefore, L ss improves the overall reward quality of the solution group using the proposedcompetitive REINFORCE scheme, making the solver generate high rewarded solutions but a smallreward deviation between solutions.
The small reward-deviation indicates L ss approximately imposessolution symmetricity to solver F ✓ .
The POMO [21] employed a similar training technique with our L ss , that ﬁnds symmetric solutionsby forcing F ✓ to visit all possible initial cities when solving TSP and CVRP.
However, the reward of COPs, including CVRP, PCTSP, and OP, is usually sensitive to ﬁrst city selection.
Therefore, POMOcan be an excellent standalone method for TSP but can be further improved using our L ss loss term.
Leveraging problem symmetricity.
As discussed in Section 2.2, the rotational problem symmetricityis common in various COPs.
We propose the REINFORCE loss L ps which is equipped with problem symmetricity: [Equation] ⇤
[Equation]
log F [Equation] where Q is the distribution of random orthogonal matrices, Q l is the l th sampled rotation matrix,and ⇡ l,k is the k th sample solution of the l th rotated problem.
We construct L problem symmetricproblems, Q 1 ( P ) , ..., Q L ( P ) , by using the sampled rotation matrices, and smaple K symmetricsolutions from each of the L problems. Then, the shared baseline b ( P , Q ) is constructed by averaging L ⇥ K solutions.
Similar to the regularization scheme of L ss , the advantage term of L ps also induces competitionbetween solutions sampled from rotationally symmetric problems.
Since the rotational symmetricityis deﬁned such that x and Q l ( x ) have the same solution, the negative advantage value forces thesolver to ﬁnd a better solution.
As mentioned in Section 2.2, problem symmetricity in COPs isusually pre-identiﬁed (i.e. there is provable guaranteed symmetricity such as rotational symmetricity Theorem 2.1) ; L ps are applicable to general COPs.
Moreover, multiple solutions are sampled foreach symmetric problem so that L ps can also identify and exploit the solution symmetricity with asimilar approach taken for L ss . We provide detailed implementation and design guides regarding theintegration strategy of L ss and L ps in Appendix C.2. 3.2 Learning invariant representation with Pre-identiﬁed Symmetricity: L inv . By Theorem 2.1, the original problem x and its rotated problem Q ( x ) have identical solutions.
Therefore the encoder of F ✓ can be enforced to have invariant representation by leveraging thepre-identiﬁed symmetricity: rotation symmetricity.
We denote h ( x ) and h ( Q ( x )) as the hidden representations of x and P ( x ) , respectively.
To imposethe rotational invariant property on h ( x ) , we train solver F ✓ with an additional regularization lossterm L inv deﬁned as: [Equation] where S cos ( a, b ) is the cosine similarity between a and b . g ( · ) is the MLP-parameterized projectionhead.
For learning the invariant representation on rotational symmetricity, we penalize the differencebetween the projected representation g ( h ( x )) and g (( h ( Q ( x )))) , instead of directly penalizing thedifference between h ( x ) and ( h ( Q ( x )) .
This penalizing scheme allows the use of an arbitrary encodernetwork architecture while maintaining the diversity of h [30].
We empirically veriﬁed that thisapproach attains stronger solvers as described in Section 6.1.
Related Works
Deep construction heuristics Bello et al.
[18] propose one of the earliest DRL-NCO methods,based on PointerNet [8], and trained it with an actor-critic method.
Attention model (AM) [19]successfully extends [18] by swapping PointerNet with Transformer [31], and it is currently thede-facto standard method for NCO.
Notably, AM veriﬁes its problem agnosticism by solving severalclassical routing problems and their practical extensions [7, 17].
POMO [21] extends AM byexploiting the solution symmetricities in TSP and CVRP.
Even though POMO shows signiﬁcantimprovements from AM, it relies on problem-speciﬁc solution symmetricities (i.e., not problemagnostic).
MDAM [24] extends AM by employing an ensemble of decoders.
However, such an extension is inapplicable for stochastic routing problems (i.e., not problem agnostic).
Among thesepromising DRL-NCO methods, Sym-NCO achieved SOTA performances with problem-agnosticproperties.
Equivariant deep learning In deep learning, symmetricities are often enforced by employingspeciﬁc network architectures.
EDP-GNN [32] proposes a permutation equivariant graph neuralnetwork (GNN) that produces equivariant outputs to the input order permutations.
SE (3) -Transformer[33] restricts the Transformer so that it is equivariant to SE (3) group input transformation.
Similarly,
EGNN [28] proposes a GNN architecture that produces O ( n ) group equivariant output.
Thesenetwork architectures can dramatically reduce the search space of the model parameters.
Someresearch applies equivariant neural networks to RL tasks to improve sample efﬁciency [34].
Also,there are several works that exploited the symmetric nature of CO.
Ouyang et al.
[35] proposedequivariant encoding scheme by giving rule-based input transformation to input graph. Hudson etal. [36] suggested a line-graph embedding scheme, which is beneﬁcial to process CO graphs withrotational equivariant. Our Sym-NCO is a regularization method that is capable of learning existingpowerful CO models without giving rule-based hard constraints to the structure.
We empirically studythe beneﬁt of Sym-NCO over other symmetricity-based approaches in Section {+6.1 and Appendix D.
+}
Experiments
This section provides the experimental results of Sym-NCO for TSP, CVRP, PCTSP, and OP.
Focusingon the fact that Sym-NCO can be applied to any encoder-decoder-based NCO method, we implement
Sym-NCO on top of POMO [21] to solve TSP and CVRP, and AM [19] to solve PCTSP and OP,respectively.
We additionally validate the effectiveness of Sym-NCO on PointerNet [8].
5.1 Tasks and baseline selections
TSP aims to ﬁnd the Hamiltonian cycle with minimum tour length.
We employ Concorde [37] and
LKH-
[38] as the non-learnable baselines, and PointerNet [8], S2V-DQN [39], RL [20] AM [19],
POMO [21] and MDAM [24] as the neural constructive baselines.
CVRP is an extension of TSP that aims to ﬁnd a set of tours with minimal total tour lengths whilesatisfying the capacity limits of the vehicles.
We employ LKH-3 [38] as the non-learnable baselines,and RL[20], AM [19], POMO [21] and MDAM [24] as the constructive neural baselines.
PCTSP is a variant of TSP that aims to ﬁnd a tour with minimal tour length while satisfying the prizeconstraints.
We employ the iterative local search (ILS) [19] as the non-learnable baseline, and AM[19] and MDAM [24] as the constructive neural baselines.
OP is a variant of TSP that aims to ﬁnd the tour with maximal total prizes while satisfying the tourlength constraint.
We employ compass [40] as the non-learnable baseline, and AM [19] and MDAM[24] as the constructive neural baselines.
5.2 Experimental setting
Problem size.
We provide the results of problems with N = 100 for the four problem classes, andreal-world TSP problems with 50 < N < 250 from TSPLIB.
Hyperparameters We apply Sym-NCO to POMO, AM, and PointerNet.
To make fair comparisons,we use the same network architectures and training-related hyperparameters from their original papersto train their Sym-NCO-augmented models.
Please refer to Appendix Appendix C.1 for more details.
Dataset and Computing Resources We use the benchmark dataset [19] to evaluate the performanceof the solvers.
To train the neural solvers, we use Nvidia A100 GPU.
To evaluate the inference speed,we use an Intel Xeon E5-2630 CPU and Nvidia
[Table]
5.3 Performance metrics
This section provides detailed performance metrics:
Average cost.
We report an average cost of 10,000 benchmark instances which is proposed by [19].
Evaluation speed.
We report the evaluation speeds of solvers in a out-of-the-box manner as they areused in practice.
In that regard, the execution time of non-neural and neural methods are measured on
CPU and GPU, respectively.
Greedy/Multi-start performance.
For neural solvers, it is a common practice to measure multi-startperformance as its ﬁnal performance.
However, when those are employed in practice, such resourceconsuming multi-start may not be possible.
Hence, we discuss greedy and multi-start separately.
5.4 Experimental results
Results of TSP and CVRP As shown in Table 1, Sym-NCO outperforms the NCO baselinesin both the greedy rollout and multi-start settings with the fastest inference speed.
Remarkably, Sym-NCO achieves a 0 .
95% gap in TSP using the greedy rollout.
In the TSP greedy setting, it solves TSP 10,000 instances in a few seconds.
Results of PCTSP and OP As shown in Table 2, Sym-NCO outperforms the NCO baselines inboth the greedy rollout and multi-start settings.
In the multi-start setting, Sym-NCO outperforms theclassical PCTSP baseline (i.e., ILS) with the 43200180 ⇡ 240 ⇥ faster speed.
Gap
POMO 1.87% Sym-NCO 1.62% gap on TSPLIB
Results of the real-world TSP We evaluate POMO and Sym-NCO on TSPLib [41].
Table 3 shows that Sym-NCO outperforms POMO.
Please referto Appendix D.2 for the full benchmark results.
Application to various DRL-NCO methods As discussed in Section 3, Sym-NCO can be applied to various various DRL-NCO methods.
We validatethat Sym-NCO signiﬁcantly improves the existing DRL-NCO methods asshown in Fig. [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Time-performance analysis for multi-starts Multi-starts is a common method that improvesthe solution qualities while requiring a longer time budget.
We use the rotation augments [21] toproduce multiple inputs (i.e., starts).
As shown in Fig.
5, Sym-NCO achieves the Pareto frontierfor all benchmark datasets.
In other words, Sym-NCO exhibits the best solution quality among thebaselines within the given time consumption.
Discussion
6.1 Discussion of Regularization based Symmetricity Learning
In this section, we provide the discussion and ablations study for invariant representation learningusing problem symmetricity.
Ablation Study of L inv
As shown in Fig.
6b, L inv increases the cosine similarity of the projectedrepresentation (i.e., g ( h ) ).
We can conclude that L inv contributes to the performance improvements(see Fig.
6a).
We further verify that imposing similarity on h degrades the performance as demonstrated in Fig.
6c.
This proves the importance of maintaining the expression power of the encoder aswe mentioned in Section 3.2.
Comparison with EGNN EGNN [28] provably guarantees to process coordinate graph withrotational equivalency.
Also, EGNN empirically veriﬁed its’ high performance in point cloud tasks.
However, we observed that EGNN signiﬁcantly underperforms than Sym-NCO and fails to convergeas shown Fig.
7b.
This is because the euclidian CO has a fully connected input graph containinginformative coordinates, and we believe the equivariant neural network should be carefully craftedto consider such unique input graph structures of CO tasks.
On the other hand, Sym-NCO couldleverage the existing powerful NCO model without ﬁne modiﬁcation of the neural network.
Thesenumerical results conform well with our hypothesis that equivariance is necessary but not sufﬁcientfor obtaining the optimal parameters for the solver (see Fig. 1).
Therefore, we expect our approach [Figure] [Figure] can be simply extended to other domains requiring symmetricity and geometricity as positioned toleverage the existing legacy of powerful non-equivariant neural networks.
6.2 Limitations & future directions
Extended problem symmetricities.
In this work, we employ the rotational symmetricity (Theorem 2.1) as the problem symmetricity.
However, for some COPs, different problem symmetricities,such as scaling and translating P , can also be considered.
Employing these additional symmetricitiesmay further enhance the performance of Sym-NCO.
We leave this for future research.
Large scale adaptation.
Large scale applicability is essential to NCO (this work solves N < 250) .
Hence, we expect that the transfer- [42], curriculum- [43], and meta-learning approaches may improvethe generalizability of NCO to larger-sized problems.
Extension to the graph COP.
This work ﬁnds the problem symmetricity that is universallyapplicable for Euclidean COPs.
However, some COPs are deﬁned in non-Euclidean spaces such asasymmetric TSP.
Furthermore, there are also a bunch of existing neural combinatorial optimizationmodels that can solve graph COP [44, 45, 46, 47, 48, 49], where we can improve with a symmetricityregularization scheme.
We also leave ﬁnding the universal symmetricities of non-Euclidean COPsand applying them to existing graph COP models for future research. [Figure] and EGNN