Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown signiﬁcant merit over the conventional COsolvers as DRL-NCO is capable of learning CO solvers without supervised labelsattained from the veriﬁed solver.
This paper presents a novel training scheme, Sym-NCO, that achieves signiﬁcant performance increments to existing DRL-NCOmethods. Sym-NCO is a regularizer-based training scheme that leverages universalsymmetricities in various CO problems and solutions.
Imposing symmetricitiessuch as rotational and reﬂectional invariance can greatly improve the generalizationcapability of DRL-NCO as symmetricities are invariant features shared by various
CO tasks.
Our experimental results verify that Sym-NCO greatly improved theperformance of DRL-NCO methods in four CO tasks, including traveling salesmanproblem (TSP), capacitated vehicle routing problem (CVRP), prize collecting TSP(PCTSP), and orienteering problem (OP), without employing problem-speciﬁctechniques.
Remarkably, Sym-NCO outperformed not only the existing DRL-NCOmethods but also a competitive conventional solver, the iterative local search (ILS),in PCTSP at 240 ⇥ faster speed.
Source code will be available after the decision ismade.
Introduction
Combinatorial optimization problems (COPs) are mathematical optimization problems on discreteinput space that carry numerous valuable applications, including vehicle routing problems (VRPs)[1, 2], drug discovery [3, 4] and semi-conductor chip design [5, 6, 7].
However, ﬁnding an optimalsolution to COP is difﬁcult due to its NP-hardness.
Therefore, computing near-optimal solutions fastis essential from a practical point of view.
Conventionally, COPs were solved by integer program (IP) solvers or hand-crafted (meta) heuristics.
Recent advances in computing infrastructures and deep learning conceived the ﬁeld of neural combinatorial optimization (NCO), a deep learning-based COP solving strategy.
Depending on the trainingscheme, NCO methods are generally classiﬁed into supervised learning [8, 9, 10, 11, 12] and reinforcement learning (RL) [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Depending on the solutiongeneration scheme, NCO methods are also classiﬁed into improvement [15, 14, 13, 26, 16, 17, 23]and constructive heuristics [18, 19, 20, 21, 22, 24, 25].
Among the NCO approaches, deep RL(DRL)-based constructive heuristics (i.e., DRL-NCO) are favored over conventional approachesdue to the train-ability of RL that does not rely on existing COP solvers, and the tractability of the
[Figure]
constructive process that prevents rule-violation of speciﬁc task and guarantees qualiﬁed solutions[19].
Despite the strength of DRL-NCO, there exists a performance gap between the state-of-the-artconventional heuristics and DRL-NCO.
In an effort to close the gap, there have been attempts toemploy problem-speciﬁc heuristics to existing DRL-NCO methods [21, 27].
However, devising ageneral training scheme to improve the performance of DRL-NCO still remains challenging.
In this study, we propose the Symmetric Neural Combinatorial Optimization (Sym-NCO), a generaltraining scheme applicable to universal CO problems.
Sym-NCO is a regularization-based trainingscheme that leverages the symmetricities commonly found in COPs to increase the performance ofexisting DRL-NCO methods.
To this end, we ﬁrst identify the symmetricities present in various COPs.
Sym-NCO leverages two types of symmetricities innate in COP that are deﬁned on the Euclideangraph.
First, the problem symmetricity derived from rotational invariance of the solution; the rotatedgraph must exhibit the same optimal solution as the original graph as shown in Fig.
1a.
Second, thesolution symmetricity, which is the shared feature among solutions having identical optimal values.
For example, the solution symmetricity in the traveling salesmen problem (TSP) includes the ﬁrst-citypermutation invariance (See Fig. 1b). However, the solution symmetricity of general COPs mustbe automatically identiﬁed during the training process.
That is because the shared feature betweenmultiple optimal solutions is usually intractable without highly investigated domain knowledge. The Sym-NCO is composed of two novel regularization methods for leveraging symmetricities. First,we suggest a new advantage function on REINFORCE algorithm that automatically identiﬁes andexploits symmetricities without imposing misleading bias.
Second, we devise a novel representationlearning scheme to impose symmetricities by leveraging the pre-identiﬁed symmetricities.
We experimentally validated Sym-NCO on various existing DRL-NCO methods by solving theiroriginal target problems without employing any problem-speciﬁc techniques. By leveraging thesymmetricities of COPs, Sym-NCO achieved the following:
• High performances.
Sym-NCO achieved near-optimal performance in various COP tasks (lessthan 2%) with extremely high speed (few seconds to solve 10,000 instances).
Moreover, Sym- NCO surpassed the competitive PCTSP solver, ILS [19], at 240 ⇥ faster speed.
• Problem agnosticism. Sym-NCO does not employ problem-speciﬁc heuristics to solve various COPs. Sym-NCO is generally applicable to solve TSP, CVRP PCTSP, and OP.
• Architecture agnosticism.
Sym-NCO can easily be implemented to any encoder-decoder model and impose the symmetricities of COPs. Sym-NCO successfully improved the performance of existing encoder-decoder-based DRL-NCO methods, such as PointerNet [8, 18], AM [19] andPOMO [21]
Symmetricity in Combinatorial Optimization Markov Decision Process
This section presents several symmetric characteristics found in combinatorial optimization, which isformulated in the Markov decision process.
The objective of NCO is to train the ✓ -parameterizedsolver F ✓ by solving the following problem:
[Equation]
where P =
( x , f ) is a problem instance with the N node coordinates x = { x i } Ni =1 and corresponding
N features f = { f i } Ni =1 .
The ⇢ is a problem generating distribution, ⇡ ( P ) is a solution of P , and
R ( ⇡ ( P )) is the objective value of ⇡ ( P ) .
2.1 Combinatorial optimization Markov decision process
We deﬁne the combinatorial optimization Markov decision process (CO-MDP) as the sequentialconstruction of a solution of COP.
For a given P , the components of the corresponding CO-MDP aredeﬁned as follows:
• State.
The state s t =
( a 1: t , x , f ) is the t -th (partially complete) solution, where a 1: t representsthe previously selected nodes.
The initial and terminal states s 0 and s T are equivalent to theempty and completed solution, respectively.
In this paper, we denote the solution ⇡ ( P ) as thecompleted solution.• Action.
The action a t is the selection of a node from the un-visited nodes (i
e., a t 2 A t ={{ 1 , ..., N } \ { a 1: t  1 }} ).•
Reward.
The reward R ( ⇡ ( P )) is the objective of COP.
We assume that the reward is a functionof a 1: T (solution sequences), || x
i  x j || i,j 2 { 1 ,... N } (relative distances) and f (nodes features).
InTSP, capacitated VRP (CVRP), and prize collecting TSPs (PCTSP), the reward is the negative ofthe tour length.
In orienteering problem (OP), the reward is the sum of the prizes.
Having deﬁned CO-MDP, we deﬁne the solution mapping as follows:
[Equation]
where p ✓ ( a t | s t ( P )) is the policy that produces a t at s t , and T is the maximum number of states inthe solution construction process.
Symmetricities in CO -MDP
Symmetricities are found in various COPs.
We conjecture that imposing those symmetricities on F ✓improves the generalization and sample efﬁciency of F ✓ .
We deﬁne the two identiﬁed symmetricitiesthat are commonly found in various COPs:
Deﬁnition 2.1 ( Problem Symmetricity ) .
Problem P i and P j are problem symmetric ( P i sym !
P j )if their optimal solution sets are identical.
Deﬁnition 2.2 ( Solution Symmetricity ) .
Two solutions of problem P ( ⇡ i ( P ) and ⇡ j ( P ) ) aresolution symmetric ( ⇡ i sym ! ⇡ j ) if R ( ⇡ i ) =
R ( ⇡ j ) .
An exemplary problem symmetricity found in various COPs is the rotational symmetricity:
Theorem 2.1 ( Rotational symmetricity ) .
For any orthogoanl matrix Q , the problem P and Q ( P ) ,{{ Qx i } Ni =1 , f } are problem symmetric: i.e., P sym ! Q ( P ) .
See Appendix A for the proof.
Rotational problem symmetricity is identiﬁed in every Euclidean COPs.
On the other hand, solutionsymmetricity cannot be identiﬁed easily as the properties of the solutions are distinct for every COP.
Symmetric Neural Combinatorial Optimization
This section presents Sym-NCO, an effective training scheme that leverages the symmetricities of
COPs.
Sym-NCO imposes the symmetricities on F ✓ by minimizing the symmetric loss function thatis deﬁned as follows:
[Equation]
) where L inv , L ps , and L ss are the loss functions that impose invariant representation, problem-solutionjoint symmetricity, and solution symmetricity, respectively.
↵ ,  2 [0 , 1] are the weight coefﬁcients.
In the following subsections, we explain each loss component in detail.
By Theorem 2.1, the original problem x and its rotated problem Q ( x ) have identical solutions. Weimpose this solution symmetricity on the encoder of F ✓ by using the rotational invariant representation. We denote h ( x ) and h ( Q ( x )) as the hidden representations of x and P ( x ) , respectively. To imposethe rotational invariant property on h ( x ) , we deﬁne L inv as follows:
[Equation]
where S cos ( a, b ) is the cosine similarity between a and b .
g ( · ) is the MLP-parameterized projectionhead.
To impose the rotational invariance, we penalize the difference between the projected representationg ( h ( x )) and g (( h ( Q ( x )))) , instead of directly penalizing the difference between h ( x ) and ( h ( Q ( x )) .
This penalizing scheme allows the use of an arbitrary encoder network architecture while maintainingthe diversity of h [28].
We empirically veriﬁed that this approach attains stronger solvers as describedin Section 6.1.
Note that the rotational invariance of h can also be attained through the EGNN [29], an invariantencoder architecture.
However, this approach inevitably restricts the ﬂexibility of the encoderarchitecture, thus preventing the use of ﬂexible architectures, like Transformer [30].
We haveempirically validated that our approach is more effective than the EGNN.
3.2 Imposing problem and solution symmetricities via L ps and L ss
As discussed in Section 2.2, COPs have problem and solution symmetricities.
We explain how toimpose the symmetricities by minimizing L ps and L ss when training F ✓ .
We provide the policygradients to L ss ( ⇡ ( P )) and L ps ( ⇡ ( P )) in the context of the REINFORCE algorithm [31] with theproposed scheme.
Imposing solution symmetricity.
In general COPs, symmetric solutions are usually intractable.
As deﬁned in Deﬁnition 2.2, the symmetric solutions must have the same objective values.
Hence,we regularize F ✓ ( ⇡ | P ) with the gradient of L ss ( ⇡ ( P )) so that its realized solutions have the sameobjective value. The gradient of L ss ( ⇡ ( P )) is as follows:
[Equation]
Baseline
[Equation]
where { ⇡ k } Kk =1 are the solutions of P sampled from F ✓ ( ⇡ | P ) , and log F ✓ ( ⇡ ( P )) is the loglikelihood of F ✓ to generate ⇡ ( P ) . K is the number of sampled solutions.
The REINFORCE trains a solver by maximizing the expected reward, which is saturated in the end(i.e.
, sub-optimal solutions can be found) if properly trained. Saturation of reward value indicates asmall deviation of rewards between sampled solutions ⇡ 1 , ..., ⇡ K , which refers to the (near) solutionsymmetricity as deﬁned in Deﬁnition 2.2.
Although naive REINFORCE has the ability to imposesolution symmetricity by nature, it does not guarantee that the identiﬁed symmetric solutions areoptimal.
Our proposed baseline guides the solver to ﬁnd improved symmetric solutions by inducingcompetition within the symmetric solution groups. If any of a symmetric solution group has the higheroptimality over the rest of solutions in the group, the proposed advantage of the rest of solutionsbecomes a negative value.
Negative advantage then encourages the solver to ﬁnd the better solutionso that prevents to fall in a bad local optimum. Overall, the proposed L ss not only imposes solutionsymmetricity but also guides to ﬁnd near-optimal solutions.
POMO [21] employs a similar training technique that ﬁnds symmetric solutions by forcing F ✓to visit all possible initial cities when solving TSP and CVRP.
For TSP, the training technique isjustiﬁable as the permutation of the initial cities preserves the reward.
However, the reward of COPs,including CVRP, PCTSP, and OP, is usually sensitive to ﬁrst city selection.
To this end, we do notrestrict ﬁrst city selection (except TSP) but leave it to the training process of L ss . Therefore, thesolution symmetricity must be identiﬁed through the training process without employing potentiallymisleading bias. Though the solution symmetricity is a signiﬁcant feature to increase generalizationcapability, it is not always present nor found in all COPs. Such observations highlight the limitationsof leveraging only the solution symmetricities when deriving F ✓ .
This motivates us to devise a moregeneral method to leverage symmetricities in COPs.
Imposing problem and solution symmetricities.
As discussed in Section 2.2, the rotational problemsymmetricity is common in various COPs.
Thus, we regularize F ✓ in terms of the rotational problemsymmetricity with the gradient of L ps ( ⇡ ( P )) deﬁned as follows:
[Equation]
where Q is the distribution of random orthogonal matrices, Q l is the l th sampled rotation matrix, and⇡ l,k is the k th sample solution of the l th rotated problem.
L and K are the number of the sampledrotation matrix and solution symmetric solutions, respectively.
Similar to the regularization scheme of L ss , the advantage term of L ps also induces competitionbetween solutions sampled from rotationally symmetric problems.
Since the rotational symmetricityis deﬁned as x and Q l ( x ) having the same solution, the negative advantage value forces the solverto ﬁnd a better solution.
As mentioned in Section 2.2, problem symmetricity in COPs is usuallypre-identiﬁed; L ps are applicable to general COPs.
Moreover, multiple solutions are sampled for eachsymmetric problem so that L ps can impose solution symmetricity with a similar approach taken for
L ss .
Related Works
Deep construction heuristics Bello et al.
[18] propose one of the earliest DRL-NCO methods,based on PointerNet [8], and trained it with an actor-critic method.
Attention model (AM) [19]successfully extends [18] by swapping PointerNet with Transformer [30], and it is currently thede-facto standard method for NCO.
Notably, AM veriﬁes its problem agnosticism by solving severalclassical routing problems and their practical extensions [7, 17].
POMO [21] extends AM byexploiting the solution symmetricities in TSP and CVRP.
Even though POMO shows signiﬁcant improvements from AM, it relies on problem-speciﬁc solution symmetricities (i.e., not problemagnostic).
MDAM [24] extends AM by employing an ensemble of decoders.
However, such extensionis inapplicable for stochastic routing problems (i.e., not problem agnostic).
Among these promising DRL-NCO methods, Sym-NCO achieved SOTA performances with problem-agnostic properties.
Equivariant deep learning In deep learning, symmetricities are often enforced by employingspeciﬁc network architectures.
EDP-GNN [32] proposes a permutation equivariant graph neuralnetwork (GNN) that produces equivariant outputs to the input order permutations.
SE (3) -Transformer[33] restricts the Transformer so that it is equivariant to SE (3) group input transformation.
Similarly, EGNN [29] proposes a GNN architecture that produces O ( n ) group equivariant output.
Thesenetwork architectures can dramatically reduce the search space of the model parameters.
Someresearch applies equivariant neural networks to RL tasks to improve sample efﬁciency [34].
However,imposing the symmetricities via specialized network architecture (i.e., hardly constrained to satisfythe symmetries) can limit the representation capabilities of the models, which consequently limits thesolution quality of DRL-NCO.
We further discuss this issue in Section [-6.1 and Appendix D.
-]6.1.
Experiments
This section provides the experimental results of Sym-NCO for TSP, CVRP, PCTSP, and OP.
Focusingon the fact that Sym-NCO can be applied to any encoder-decoder-based NCO method, we implement
Sym-NCO on top of POMO [21] to solve TSP and CVRP, and AM [19] to solve PCTSP and OP,respectively.
We additionally validate the effectiveness of Sym-NCO on PointerNet [8].
5.1 Tasks and baseline selections
TSP aims to ﬁnd the Hamiltonian cycle with a minimum tour length.
We employ Concorde {+[35] and LKH-
 [36]+} as the non-learnable baselines, and PointerNet [8], S2V-DQN [37], RL [20] AM [19], POMO [21] and MDAM [24] as the neural constructive baselines.
CVRP is an extension of TSP that aims to ﬁnd a set of tours with minimal total tour lengths whilesatisfying the capacity limits of the vehicles.
We employ LKH-3 [36] as the non-learnable baselines,and RL[20], AM [19], POMO [21], and MDAM [24] as the constructive neural baselines.
PCTSP is a variant of TSP that aims to ﬁnd a tour with minimal tour length while satisfying the prizeconstraints.
We employ the iterative local search (ILS) [19] as the non-learnable baseline, and AM[19] and MDAM [24] as the constructive neural baselines.
OP is a variant of TSP that aims to ﬁnd the tour with maximal total prizes while satisfying the tourlength constraint.
We employ compass [38] as the non-learnable baseline, and AM [19] and MDAM[24] as the constructive neural baselines.
5.2 Experimental setting
Problem size.
We provide the results of problems with N = 100 for the four problem classes, andreal-world TSP problems with 50 < N < 250 from TSPLIB [39].
Hyperparameters We apply Sym-NCO to POMO, AM, and PointerNet.
To make fair comparisons,we use the same network architectures and training-related hyperparameters from their original papersto train their Sym-NCO-augmented models.
Please refer to Appendix Appendix C.1 for more details.
Dataset and Computing Resources We use the benchmark dataset [19] to evaluate the performanceof the solvers.
To train the neural solvers, we use Nvidia A100 GPU.
To evaluate the inference speed,we use an Intel Xeon E5-2630 CPU and Nvidia RTX2080Ti GPU to make fair comparisons with theexisting methods as proposed in [24].
[Table]
[Table]
5.3 Performance metrics
This section provides detailed performance metrics:
Average cost.
We report an average cost of 10,000 benchmark instances which is proposed by [19].
Evaluation speed.
We report the evaluation speeds of solvers in an out-of-the-box manner as they areused in practice.
In that regard, the execution time of non-neural and neural methods are measured on
CPU and GPU, respectively.
Greedy/Multi-start performance.
For neural solvers, it is a common practice to measure multistart performance as its ﬁnal performance.
However, when those are employed in practice, suchresource-consuming multi-start may not be possible.
Hence, we discuss greedy and multi-startseparately.
5.4 Experimental results
TSP and CVRP. As shown in Table 1, Sym-NCO outperforms the NCO baselines in both thegreedy rollout and multi-start settings with the fastest inference speed.
Remarkably, Sym-NCO [Figure] [Figure]
using the greedy rollout.
In the TSP greedy setting, it solves TSP10,000 instances in a few seconds.
PCTSP and OP. As shown in Table 2, Sym-NCO outperforms the NCO baselines in both thegreedy rollout and multi-start settings.
In the multi-start setting, Sym-NCO outperforms the classical PCTSP baseline (i.e., ILS) with the 43200180 ⇡ 240 ⇥ faster speed. Gap
[Table]
Real-world TSP. We evaluate POMO and Sym-NCO on TSPLIB[39].
Table 3 shows that Sym-NCO outperforms POMO.
Pleaserefer to Appendix D.2 for the full benchmark results.
Application to various DRL-NCO methods As discussed in
Section 3, Sym-NCO can be applied to various various DRL-NCOmethods.
We validate that Sym-NCO signiﬁcantly improves theexisting DRL-NCO methods as shown in Fig.
Time-performance analysis for multi-starts Multi-starts is acommon method that improves the solution qualities while requiring a longer time budget.
We usethe rotation augments [21] to produce multiple inputs (i.e., starts).
As shown in Fig.
4, Sym-NCOachieves the Pareto frontier for all benchmark datasets.
In other words, Sym-NCO exhibits the bestsolution quality among the baselines within the given time consumption.
Discussion
6.1 Soft invariant learning vs. hard constraint invariant learning
This paper presents a novel invariant learning scheme that imposes the problem symmetricity on NCO via regularization (i.e., soft invariant learning ). Furthermore, we devise L inv to impose thesymmetricity on the hidden representations.
In this section, we provide the discussion and ablationsof these design choices.
Ablation Study of L inv
As shown in Fig.
5b, L inv increases the cosine similarity of the projectedrepresentation (i.e., g ( h ) ).
We can conclude that L inv contributes to the performance improvements [Figure]
[Figure] (see Fig. 5a).
We further verify that imposing similarity on h degrades the performance as demonstrated in Fig.
5c.
This again proves the importance of maintaining the representation capability ofthe encoder.
Comparison with EGNN One distinct aspect of Sym-NCO is to impose the symmetricies throughthe soft invariant learning .
However, imposing the symmetriciticy through high constraint invariantlearning by modifying the network architecture is also a viable option. To further understand theeffects of the symmetricity-imposing mechanisms, we additionally train the high constraint invaraintlearning model, EGNN [29], as the encoder. As shown in Fig. 6a, the hard approach (i.e., EGNN)strictly enforces the symmetricities.
Nevertheless, we observed that EGNN signiﬁcantly underperforms than Sym-NCO, and fails to converge as shown Fig.
6b.
We suspect that the performancedifference originates mainly from the restricted network architecture of EGNN.
6.2 Limitations & future directions
Extended problem symmetricities.
In this work, we employ the rotational symmetricity (Theorem 2.1) as the problem symmetricity.
However, for some COPs, different problem symmetricities,such as scaling and translating P , can also be considered.
Employing these additional symmetricitiesmay further enhance the performance of Sym-NCO.
We leave this for future research.
Large scale adaptation.
Large scale applicability is essential to NCO (this work solves N < 250) .
Hence, we expect that the transfer- [40], curriculum- [41], and meta-learning approaches may improvethe generalizability of NCO to larger-sized problems.
Extension to the graph COP.
This work ﬁnds the problem symmetricity that is universallyapplicable for Euclidean COPs.
However, some COPs are deﬁned in the non-Euclidean spaces suchas asymmetric TSP.
We also leave ﬁnding the universal symmetricities of non-Euclidean COPs forfuture research.