Original: G ENERALIZATION IN D EEP RL FOR TSP P ROBLEMS VIA E QUIVARIANCE AND L OCAL S EARCH
Revised: G ENERALIZATION IN D EEP RL FOR TSP P ROBLEMS VIA E QUIVARIANCE AND L OCAL S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning (RL) has proved to be a competitive heuristic for solving small-sized instances of traveling salesman problems (TSP), but its performance on larger-sized instances is insufﬁcient.
Revised: Deep reinforcement learning (RL) has proved to be a competitive heuristic for solving small-sized instances of traveling salesman problems (TSP), but its performance on larger-sized instances is insufﬁcient.

Original: Since training on large instances is impractical, we design a novel deep RL approach with a focus on generalizability.
Revised: Since training on large instances is impractical, we design a novel deep RL approach with a focus on generalizability.

Original: Our proposition consisting of a simple deep learning architecture that learns with novel RL training techniques, exploits two main ideas.
Revised: Our proposition consisting of a simple deep learning architecture that learns with novel RL training techniques, exploits two main ideas.

Original: First, we exploit equivariance to facilitate training.
Revised: First, we exploit equivariance to facilitate training.

Original: Second, we interleave efﬁcient local search heuristics with the usual RL training to smooth the value landscape.
Revised: Second, we interleave efﬁcient local search heuristics with the usual RL training to smooth the value landscape.

Original: In order to validate the whole approach, we empirically evaluate our proposition on random and realistic
Revised: In order to validate the whole approach, we empirically evaluate our proposition on random and realistic

Original: TSP problems against relevant state-of-the-art deep RL methods.
Revised: TSP problems against relevant state-of-the-art deep RL methods.

Original: Moreover, we present an ablation study to understand the contribution of each of its components.
Revised: Moreover, we present an ablation study to understand the contribution of each of its components.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The traveling salesman problem (TSP) has numerous applications from supply chain management (Snyder & Shen, 2019) to electronic design automation (Gerez, 1999) or bioinformatics (Jones &
Revised: The traveling salesman problem (TSP) has numerous applications from supply chain management (Snyder & Shen, 2019) to electronic design automation (Gerez, 1999) or bioinformatics (Jones &

Original: Pevzner, 2004).
Revised: Pevzner, 2004).

Original: As an NP-hard problem, solving large-sized problem instances is generally intractable.
Revised: As an NP-hard problem, solving large-sized problem instances is generally intractable.

Original: Deep reinforcement learning (RL)-based heuristic solvers have been demonstrated (Bello et al., 2016; Dai et al., 2017; Kool et al., 2019; Ma et al., 2019) as being able to provide competitive solutions while being fast, which is crucial in many domains such as logistics for real-time operations.
Revised: Deep reinforcement learning (RL)-based heuristic solvers have been demonstrated (Bello et al., 2016; Dai et al., 2017; Kool et al., 2019; Ma et al., 2019) as being able to provide competitive solutions while being fast, which is crucial in many domains such as logistics for real-time operations.

Original: However, this ability has only been demonstrated on small-sized problems where RLbased solvers are usually trained and tested on small instances.
Revised: However, this ability has only been demonstrated on small-sized problems where RLbased solvers are usually trained and tested on small instances.

Original: When evaluated instead on larger instances, such solvers perform poorly.
Revised: When evaluated instead on larger instances, such solvers perform poorly.

Original: Since the computational cost of training on large instances is prohibitive, increasing the size of the training instances is not a practical option for obtaining efﬁcient solvers for larger instances.
Revised: Since the computational cost of training on large instances is prohibitive, increasing the size of the training instances is not a practical option for obtaining efﬁcient solvers for larger instances.

Original: To overcome this limitation, this paper investigates techniques to increase the generalization capability of deep RL solvers, which would allow to train on small instances and then solve larger ones.
Revised: To overcome this limitation, this paper investigates techniques to increase the generalization capability of deep RL solvers, which would allow to train on small instances and then solve larger ones.

Original: Generalization in deep RL (Franc¸ois-Lavet et al., 2018) can be improved by adjusting the following three components: input/representation space, objective function, and learning algorithm.
Revised: Generalization in deep RL (Franc¸ois-Lavet et al., 2018) can be improved by adjusting the following three components: input/representation space, objective function, and learning algorithm.

Original: We propose to achieve this by exploiting equivariance, local search, and  curriculum learning, which we explain next.
Revised: We propose to achieve this by exploiting equivariance, local search, and stochastic curriculum learning, which we explain next.

Original: Although our approach is demonstrated on TSP problems, we believe it can be naturally extended to other combinatorial optimization problems, but also to more classic RL domains.
Revised: Although our approach is demonstrated on TSP problems, we believe it can be naturally extended to other combinatorial optimization problems, but also to more classic RL domains.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Problems with spatial information, such as TSP problems, enjoy many spatial symmetries which can be exploited for RL training and generalization.
Revised: Problems with spatial information, such as TSP problems, enjoy many spatial symmetries which can be exploited for RL training and generalization.

Original: An RL solver is invariant with respect to some symmetry, if its outputs remain unchanged for symmetric inputs.
Revised: An RL solver is invariant with respect to some symmetry, if its outputs remain unchanged for symmetric inputs.

Original: For instance, translating all the city positions leave optimal solutions intact.
Revised: For instance, translating all the city positions leave optimal solutions intact.

Original: More generally, an RL solver is equivariant, if its outputs are also transformed with a corresponding symmetry for symmetric inputs.
Revised: More generally, an RL solver is equivariant, if its outputs are also transformed with a corresponding symmetry for symmetric inputs.

Original: For instance, if cities are permuted, a corresponding permutation is required on the outputs of the RL solver.
Revised: For instance, if cities are permuted, a corresponding permutation is required on the outputs of the RL solver.

Original: Invariance and equivariance of the solver with respect to some symmetries allows a smaller input space to be considered during training and more abstract representation insensitive to those symmetries to be learned while the trained solver still covers the whole input space.
Revised: Invariance and equivariance of the solver with respect to some symmetries allows a smaller input space to be considered during training and more abstract representation insensitive to those symmetries to be learned while the trained solver still covers the whole input space.

Original: Previous work has considered using local search as a simple additional step to improve the feasible solution output by a RL-based solver.
Revised: Previous work has considered using local search as a simple additional step to improve the feasible solution output by a RL-based solver.

Original: In contrast to  previous work, we further employ local search as a tool to smooth the objective function optimized during RL training.
Revised: In contrast to most previous work, we further employ local search as a tool to smooth the objective function optimized during RL training.

Original: To that aim, we interleave RL training with local search using the improved solution provided by the latter to train the RL solver via a modiﬁed policy gradient.
Revised: To that aim, we interleave RL training with local search using the improved solution provided by the latter to train the RL solver via a modiﬁed policy gradient.

Original: Moreover, we propose a novel simple baseline called the policy rollout baseline to reduce the variance of its estimation.
Revised: Moreover, we propose a novel simple baseline called the policy rollout baseline to reduce the variance of its estimation.

Original: Curriculum learning (Soviany et al., 2021) can accelerate training, but also improve generalization (Weinshall et al., 2018).
Revised: Curriculum learning (Soviany et al., 2021) can accelerate training, but also improve generalization (Weinshall et al., 2018).

Original: By modulating the difﬁculty of the training instances, as described by instance sizes, the RL-based solver can learn more easily and faster.
Revised: By stochastically adjusting the difﬁculty of the training instances, as described by instance sizes, the RL-based solver can learn more easily and faster.

Original: Since the solver sees various sizes of instances, this may prevent it to overﬁt to one particular instance size.
Revised: Since the solver sees various sizes of instances, this may prevent it to overﬁt to one particular instance size.

Original: The architecture of our model includes a graph neural network (GNN), a multi-layer perceptron (MLP), and an attention mechanism.
Revised: The architecture of our model includes a graph neural network (GNN), a multi-layer perceptron (MLP), and an attention mechanism.

Original: Due to all the used techniques, we name our model as eMAGIC (equivariance for e, MLP for  M, Attention for  A  ,  Graph neural network forG , Interleaved local search for I , andCurriculum learning for C ).
Revised: Due to all the used techniques, we name our model as eMAGIC ( e for equivariance, M for MLP, A for Attention , G for Graph neural network, I for Interleaved local search, and C for Stochastic Curriculum learning).

Original: We demonstrate that it can be trained on small random instances (up to 50 cities) and perform competitively on larger random or realistic instances (up to 1000 cities).
Revised: We demonstrate that it can be trained on small random instances (up to 50 cities) and perform competitively on larger random or realistic instances (up to 1000 cities).

Original: The only learning methods that can perform better than ours require learning on the instance to be solved (see Section 2 for a discussion).
Revised: The only learning methods that can perform better than ours require learning on the instance to be solved (see Section 2 for a discussion).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Research on exploiting deep learning (Vinyals et al., 2015;
Revised: Research on exploiting deep learning (Vinyals et al., 2015;

Original: Li et al., 2018; Joshi et al., 2019a; Prates et al., 2019; Xing & Tu, 2020) or RL (Bello et al., 2016; Dai et al., 2017; Kool et al., 2019; Deudon et al., 2018; Ma et al., 2019; Da Costa et al., 2020; Zheng et al., 2021;  Wu et al., 2021b) to design heuristics for solving TSP problems has become very active.
Revised: Li et al., 2018; Joshi et al., 2019a; Prates et al., 2019; Xing & Tu, 2020) or RL (Bello et al., 2016; Dai et al., 2017; Kool et al., 2019; Deudon et al., 2018; Ma et al., 2019; Da Costa et al., 2020; Zheng et al., 2021; Kwon et al., 2021; Wu et al., 2021b) to design heuristics for solving TSP problems has become very active.

Original: The current achievements demonstrate the potential of machine-learning-based approaches, but also reveal their usual limitation on solving larger-size instances, which has prompted much research on generalization (Joshi et al., 2019b; Fu et al., 2021; Wu et al., 2021b).
Revised: The current achievements demonstrate the potential of machine-learning-based approaches, but also reveal their usual limitation on solving larger-size instances, which has prompted much research on generalization (Joshi et al., 2019b; Fu et al., 2021; Wu et al., 2021b).

Original: Some experimental work (Joshi et al., 2019b) suggests that deep RL may provide more generalizable solvers than supervised learning, which is a further motivation for our own work.
Revised: Some experimental work (Joshi et al., 2019b) suggests that deep RL may provide more generalizable solvers than supervised learning, which is a further motivation for our own work.

Original: Another advantage of deep RL is that it does not require optimal solvers to train.
Revised: Another advantage of deep RL is that it does not require optimal solvers to train.

Original: We discuss next the related work that exploits equivariance or local search like our method.
Revised: We discuss next the related work that exploits equivariance or local search like our method.

Original: To the best of our knowledge, no other work uses  curriculum learning for designing an RL-based solver for TSP.
Revised: To the best of our knowledge, no other work uses stochastic curriculum learning for designing an RL-based solver for TSP, although Lisicki et al.

Revised: (2020) evaluated deterministic curriculum learning strategies on small TSP instances.

Original: For space reasons, our discussion below emphasizes the approaches using deep RL as a constructive heuristic (which builds a solution iteratively).
Revised: For space reasons, our discussion below emphasizes the approaches using deep RL as a constructive heuristic (which builds a solution iteratively).

Original: Invariance and equivariance have been important properties to exploit for designing powerful deep learning architectures (Bengio & Lecun, 1997; Gens & Domingos, 2014; Cohen & Welling, 2016).
Revised: Invariance and equivariance have been important properties to exploit for designing powerful deep learning architectures (Bengio & Lecun, 1997; Gens & Domingos, 2014; Cohen & Welling, 2016).

Original: They have also inspired some of the previous work on solving TSP problems, although they were often not explicitly discussed like in our work.
Revised: They have also inspired some of the previous work on solving TSP problems, although they were often not explicitly discussed like in our work.

Original: For instance, Deudon et al.
Revised: For instance, Deudon et al.

Original: (2018) use principal component analysis to exploit rotation invariance as a single preprocessing step.
Revised: (2018) use principal component analysis to exploit rotation invariance as a single preprocessing step.

Original: In contrast to their work, we compose various preprocessing transformations to be applied at every solving step.
Revised: In contrast to their work, we compose various preprocessing transformations to be applied at every solving step.

Original: To the best of our knowledge, we are the ﬁrst to propose such technique.
Revised: To the best of our knowledge, we are the ﬁrst to propose such technique.

Original: In addition, permutation invariance is the motivation for using attention models (Kool et al., 2019; Deudon et al., 2018) or GNNs (Ma et al., 2019) in RL solvers.
Revised: In addition, permutation invariance is the motivation for using attention models (Kool et al., 2019; Deudon et al., 2018) or GNNs (Ma et al., 2019) in RL solvers.

Original: Contrary to (Deudon et al., 2018), Kool et al.
Revised: Contrary to Deudon et al. (2018), Kool et al.

Original: (2019) select the next city to visit based on the ﬁrst and last visited cities while Ma et al.
Revised: (2019) select the next city to visit based on the ﬁrst and last visited cities while Ma et al.

Original: (2019) propose to use relative positions for translation invariance.
Revised: (2019) propose to use relative positions for translation invariance.

Original: Our proposition combines those two ideas, but compared to (Kool et al., 2019), our model is based on GNN, while compared to (Ma et al., 2019), it has a simpler architecture that does not require an LSTM.
Revised: Our proposition combines those two ideas, but compared to Kool et al. (2019), our model is based on GNN, while compared to Ma et al. (2019), it has a simpler architecture that does not require an LSTM.

Revised: For vehicle routing problems, Peng et al.

Revised: (2020) propose to remove visited cities from the RL agent’s input, a technique we also apply to TSP.

Revised: In contrast to their work, our GNN model is simpler and we investigate generalization from small instances to very large instances.

Original: To summarize, compared to all previous work, our paper investigates in a more systematic fashion the exploitation of equivariance to help training and improve generalization.
Revised: To summarize, compared to all previous work, our paper investigates in a more systematic fashion the exploitation of equivariance to help training and improve generalization.

Original: Several previous studies combine deep learning or RL with various local search techniques to ﬁnd better solutions: 2-opt (Deudon et al., 2018; Ma et al., 2019;
Revised: Several previous studies combine deep learning or RL with various local search techniques to ﬁnd better solutions: 2-opt (Deudon et al., 2018; Ma et al., 2019;

Original: Da Costa et al., 2020; Wu et al., 2021b), k -opt (Zheng et al., 2021), or Monte-Carlo tree search (Fu et al., 2021).
Revised: Da Costa et al., 2020; Wu et al., 2021b), k -opt (Zheng et al., 2021), or Monte-Carlo tree search (Fu et al., 2021).

Original: In contrast to these methods, we use a combined local search technique, which applies several heuristics in an efﬁcient way.
Revised: In contrast to these methods, we use a combined local search technique, which applies several heuristics in an efﬁcient way.

Original: Moreover, to the best of our knowledge, no previous work considers interleaving local search with policy gradient updates to obtain a more synergetic ﬁnal method.
Revised: Moreover, to the best of our knowledge, no previous work considers interleaving local search with policy gradient updates to obtain a more synergetic ﬁnal method for solving TSP.

Revised: For the bin packing problem, Cai et al.

Revised: (2019) use RL to learn a perturbative heuristics to provide a good initial solution to a heuristic optimizer.

Revised: Our interleaved training process is based on a similar idea, but our

Revised: RL agent learns a constructive heuristics and we provide an intuitive motivation why this approach may work via our smoothed policy gradient.

Original: All these machine-learning-based methods can be categorized as learning from a batch of instances or as directly learning on the instance to be solved.
Revised: All these machine-learning-based methods can be categorized as learning from a batch of instances or as directly learning on the instance to be solved.

Original: Like our work, mostpreviously-discussed methods are part of the ﬁrst category.
Revised: Like our work, most methods are part of the ﬁrst category.

Original: However, some recent work belongs to the second category (Zheng et al., 2021) or are hybrid (Fu et al., 2021).
Revised: However, some recent work belongs to the second category (Zheng et al., 2021) or are hybrid (Fu et al., 2021).

Original: For instance, Zheng et al.
Revised: For instance, Zheng et al.

Original: (2021) apply RL to make the LKH algorithm (Helsgaun, 2017), a classic efﬁcient heuristic for TSP, adaptive to the instance to be solved.
Revised: (2021) apply RL to make the LKH algorithm (Helsgaun, 2017), a classic efﬁcient heuristic for TSP, adaptive to the instance to be solved.

Original: In Fu et al.
Revised: In Fu et al.

Original: (2021)’s method, evaluations learned in a supervised way from  batch of instances are used in Monte-Carlo tree search so that solutions can  be foundin an adaptive for the current instance.
Revised: (2021)’s method, evaluations learned in a supervised way from a batch of instances are used in Monte-Carlo tree search so that solutions can adaptively be found for the current instance.

Original: Being adaptive to the current instance can undoubtedly boost the performance of the solver.
Revised: Being adaptive to the current instance can undoubtedly boost the performance of the solver.

Original: We leave for future work the improvement of our solver to become more adaptive to the instance to be solved.
Revised: We leave for future work the improvement of our solver to become more adaptive to the instance to be solved.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Following previous work, we focus on the symmetric 2D Euclidean TSP, which we recall below.
Revised: Following previous work, we focus on the symmetric 2D Euclidean TSP, which we recall below.

Original: We then explain how a TSP instance can be tackled with RL.
Revised: We then explain how a TSP instance can be tackled with RL.

Original: We ﬁrst provide some notations:
Revised: We ﬁrst provide some notations:

Original: For any positive integer N ∈ N , [ N ] denotes the set { 1 , 2 , . .
Revised: For any positive integer N ∈ N , [ N ] denotes the set { 1 , 2 , . .

Original: . , N } .
Revised: . , N } .

Original: Vectors and matrices are denoted in bold (e.g., x or X ).
Revised: Vectors and matrices are denoted in bold (e.g., x or X ).

Original: For a set of subscripts I , X I denotes the matrix formed by the rows of X whose indices are in I .
Revised: For a set of subscripts I , X I denotes the matrix formed by the rows of X whose indices are in I .

Original: Traveling Salesman Problem
Revised: Traveling Salesman Problem

Original: A symmetric 2D Euclidean TSP instance is described by a set of
Revised: A symmetric 2D Euclidean TSP instance is described by a set of

Original: N cities (identiﬁed to the set [ N ] ) and their coordinates in R 2 .
Revised: N cities (identiﬁed to the set [ N ] ) and their coordinates in R 2 .

Original: The goal in this problem is to ﬁnd the shortest tour that visits each city exactly once.
Revised: The goal in this problem is to ﬁnd the shortest tour that visits each city exactly once.

Original: The coordinates of city i are denoted x i ∈ R 2 .
Revised: The coordinates of city i are denoted x i ∈ R 2 .

Original: The matrix whose rows correspond to city coordinates is denoted X ∈ R N × 2 .
Revised: The matrix whose rows correspond to city coordinates is denoted X ∈ R N × 2 .

Original: A feasible solution (i.e., tour) of a TSP instance is a permutation σ over [ N ] with length equal to:
Revised: A feasible solution (i.e., tour) of a TSP instance is a permutation σ over [ N ] with length equal to:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107)·(cid:107) 2 denotes the (cid:96) 2 -norm, σ ( t ) ∈
Revised: where (cid:107)·(cid:107) 2 denotes the (cid:96) 2 -norm, σ ( t ) ∈

Original: [ N ] is the t -th visited city in tour σ , and by abuse of notation, σ ( N + 1) denotes σ (1) .
Revised: [ N ] is the t -th visited city in tour σ , and by abuse of notation, σ ( N + 1) denotes σ (1) .

Original: Therefore, solving a TSP instance consists in ﬁnding the permutation σ that minimizes the tour length
Revised: Therefore, solving a TSP instance consists in ﬁnding the permutation σ that minimizes the tour length

Original: L σ ( X ) deﬁned in Equation (1).
Revised: L σ ( X ) deﬁned in Equation (1).

Original: Since TSP solutions are invariant to scaling, we assume that the city coordinates are in the square [0 , 1] 2 .
Revised: Since TSP solutions are invariant to scaling, we assume that the city coordinates are in the square [0 , 1] 2 .

Original: In Appendix A, we recall some heuristics (e.g., insertion, k -opt) that have been proposed for solving TSP.
Revised: In Appendix A, we recall some heuristics (e.g., insertion, k -opt) that have been proposed for solving TSP.

Original: RL as a Constructive Heuristic RL can be used to construct a TSP tour σ sequentially.
Revised: RL as a Constructive Heuristic RL can be used to construct a TSP tour σ sequentially.

Original: Intuitively, at iteration t ∈ [ N ] , an RL solver (i.e., policy) selects the next unvisited city σ ( t ) to visit based on the current partial tour and the description of the TSP instance (i.e., coordinates of cities).
Revised: Intuitively, at iteration t ∈ [ N ] , an RL solver (i.e., policy) selects the next unvisited city σ ( t ) to visit based on the current partial tour and the description of the TSP instance (i.e., coordinates of cities).

Original: Therefore, this RL problem corresponds to a repeated N -horizon sequential decision-making problem.
Revised: Therefore, this RL problem corresponds to a repeated N -horizon sequential decision-making problem.

Original: As noticed by Kool et al.
Revised: As noticed by Kool et al.

Original: (2019), the decision for the next city to visit only depends on the description of the TSP instance and the ﬁrst and last visited cities.
Revised: (2019), the decision for the next city to visit only depends on the description of the TSP instance and the ﬁrst and last visited cities.

Original: In addition, we update the description of the TSP instance by removing the coordinates of visited cities.
Revised: In addition, we update the description of the TSP instance by removing the coordinates of visited cities.

Original: Surprisingly, to the best of our knowledge, no previous work exploits this simpliﬁed RL model.
Revised: Surprisingly, to the best of our knowledge, no previous work exploits this simpliﬁed RL model.

Original: Formally, in the RL language, at time step t ∈ [ N ] , an action a t represents the next city to visit, i.e., a t = σ ( t ) .
Revised: Formally, in the RL language, at time step t ∈ [ N ] , an action a t represents the next city to visit, i.e., a t = σ ( t ) .

Original: Let I 1 =
Revised: Let I 1 =

Original: [ N ] and I t +1 =
Revised: [ N ] and I t +1 =

Original: [ N ] \{ σ (1) , . . .
Revised: [ N ] \{ σ (1) , . . .

Original: , σ ( t ) } be the set of remaining cities after t cities have been already visited.
Revised: , σ ( t ) } be the set of remaining cities after t cities have been already visited.

Original: Moreover, let J 1 =
Revised: Moreover, let J 1 =

Original: [ N ] and J t
Revised: [ N ] and J t

Original: = I t ∪ { σ (1) , σ ( t ) } be the set of unvisited cities in addition of the ﬁrst and last visited cities ( σ (1) and σ ( t ) ).
Revised: = I t ∪ { σ (1) , σ ( t ) } be the set of unvisited cities in addition of the ﬁrst and last visited cities ( σ (1) and σ ( t ) ).

Original: Therefore, a t ∈ I t for all t ∈ [ N ] .
Revised: Therefore, a t ∈ I t for all t ∈ [ N ] .

Original: A state s t can be represented by a matrix X J t with ﬂags indicating the ﬁrst and last visited cities.
Revised: A state s t can be represented by a matrix X J t with ﬂags indicating the ﬁrst and last visited cities.

Original: This matrix includes the coordinates of unvisited cities ( X I t ) in addition to the coordinates of the ﬁrst and last visited cities ( x σ (1) and x σ ( t ) ).
Revised: This matrix includes the coordinates of unvisited cities ( X I t ) in addition to the coordinates of the ﬁrst and last visited cities ( x σ (1) and x σ ( t ) ).

Original: Note that at t = 1 , no city has been chosen yet, so the initial state s 1 only contains the list of city coordinates.
Revised: Note that at t = 1 , no city has been chosen yet, so the initial state s 1 only contains the list of city coordinates.

Original: A state s N contains the coordinates of thelast unvisited city and those of the ﬁrst and last visited cities.
Revised: A state s N contains the coordinates of the unvisited cities and those of the ﬁrst and last visited cities.

Original: After choosing the last city to visit in state s N , the tour σ is completely generated.
Revised: After choosing the last city to visit in state s N , the tour σ is completely generated.

Original: The immediate reward r ( s t , a t ) for an action a t in a state s t can be deﬁned as the negative length between the last visited city and the next chosen city, since we want the tour length to be small:
Revised: The immediate reward r ( s t , a t ) for an action a t in a state s t can be deﬁned as the negative length between the last visited city and the next chosen city, since we want the tour length to be small:

Original: [Equation]
Revised: [Equation]

Original: After choosing the ﬁrst city a 1 = σ (1) , the reward is zero since no length can be computed.
Revised: After choosing the ﬁrst city a 1 = σ (1) , the reward is zero since no length can be computed.

Original: After the ﬁnal action a N , an additional reward −|| x σ (1) − x σ ( N ) || 2 is added to complete the tour length.
Revised: After the ﬁnal action a N , an additional reward −|| x σ (1) − x σ ( N ) || 2 is added to complete the tour length.

Original: In deep RL, a policy π θ is represented as a neural network parametrized by θ .
Revised: In deep RL, a policy π θ is represented as a neural network parametrized by θ .

Original: The goal is then to ﬁnd θ ∗ that maximizes the objective function J ( θ ) :
Revised: The goal is then to ﬁnd θ ∗ that maximizes the objective function J ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: where for all t ∈ [ N ] , r t = r ( s t , a t ) , τ = ( s 1 , a 1 , s 2 , a 2 , . . .
Revised: where for all t ∈ [ N ] , r t = r ( s t , a t ) , τ = ( s 1 , a 1 , s 2 , a 2 , . . .

Original: , s N , a N ) is a complete trajectory, and p θ is the probability over trajectories induced by policy π θ .
Revised: , s N , a N ) is a complete trajectory, and p θ is the probability over trajectories induced by policy π θ .

Original: This objective function J ( θ ) can be optimized by a policy gradient (Williams, 1992) or actor-critic method (Sutton & Barto, 1998).
Revised: This objective function J ( θ ) can be optimized by a policy gradient (Williams, 1992) or actor-critic method (Sutton & Barto, 1998).

Original:  E QUIVARIANT M ODEL
Revised:  E QUIVARIANT M ODEL

Original: In this section, we present several techniques to exploit invariances and equivariances in the design of an RL solver.
Revised: In this section, we present several techniques to exploit invariances and equivariances in the design of an RL solver.

Original: Formally, a mapping f : A → B from a set A to a set B is invariant with respect to a symmetry
Revised: Formally, a mapping f : A → B from a set A to a set B is invariant with respect to a symmetry

Original: ρ A : A → A iff f ( x ) =
Revised: ρ A : A → A iff f ( x ) =

Original: f
Revised: f

Original: ( ρ A ( x )) for any x ∈ A .
Revised: ( ρ A ( x )) for any x ∈ A .

Original: More generally, given a symmetry that acts on A with ρ A : A → A and on B with ρ B : B → B , a mapping f : A → B is equivariant with respect to this symmetry iff ρ B ( f ( x ))
Revised: More generally, given a symmetry that acts on A with ρ A : A → A and on B with ρ B : B → B , a mapping f : A → B is equivariant with respect to this symmetry iff ρ B ( f ( x ))

Original: =
Revised: =

Original: f ( ρ A ( x )) for any x ∈ A .
Revised: f ( ρ A ( x )) for any x ∈ A .

Original: This deﬁnition shows that invariance is a special case of equivariance when ρ B is the identity function.
Revised: This deﬁnition shows that invariance is a special case of equivariance when ρ B is the identity function.

Original: Intuitively, equivariance for an RL solver (resp.
Revised: Intuitively, equivariance for an RL solver (resp.

Original: its value function) means that if a transformation is applied to its input, its output (resp.
Revised: its value function) means that if a transformation is applied to its input, its output (resp.

Original: its value) can be recovered by a corresponding transformation.
Revised: its value) can be recovered by a corresponding transformation.

Original: In the remaining of the paper, for simplicity, we often use equivariance to refer to both equivariance and invariance, since the latter is a special case of the former.
Revised: In the remaining of the paper, for simplicity, we often use equivariance to refer to both equivariance and invariance, since the latter is a special case of the former.

Original: Equivariance can be exploited in RL in various ways.
Revised: Equivariance can be exploited in RL in various ways.

Original: We consider some equivariant preprocessing methods on the description of TSP instances to standardize the kinds of instances the solver is trained on and evaluated on.
Revised: We consider some equivariant preprocessing methods on the description of TSP instances to standardize the kinds of instances the solver is trained on and evaluated on.

Original: In addition, we propose a simple deep learning model for which we apply some other equivariant preprocessing methods on its inputs to further reduce the input space.
Revised: In addition, we propose a simple deep learning model for which we apply some other equivariant preprocessing methods on its inputs to further reduce the input space.

Original: 4. E QUIVARIANT P REPROCESSING OF TSP I NSTANCE
Revised: 4. E QUIVARIANT P REPROCESSING OF TSP I NSTANCE

Original: An RL solver should be invariant with respect to any Euclidean symmetry (rotation, reﬂection, translations and their compositions) and to any positive scaling transformation applied on city positions.
Revised: An RL solver should be invariant with respect to any Euclidean symmetry (rotation, reﬂection, translations and their compositions) and to any positive scaling transformation applied on city positions.

Original: To enforce these invariances, we can apply these transformations to preprocess the inputs of the solver such that the transformed inputs are always in a standard form.
Revised: To enforce these invariances, we can apply these transformations to preprocess the inputs of the solver such that the transformed inputs are always in a standard form.

Original: Doing so allows the solver to be trained on more similar inputs.
Revised: Doing so allows the solver to be trained on more similar inputs.

Original: Concretely, for rotation invariance, we rotate and scale the city positions such that they are mostly distributed along the ﬁrst diagonal of the square [0 , 1] 2 .
Revised: Concretely, for rotation invariance, we rotate and scale the city positions such that they are mostly distributed along the ﬁrst diagonal of the square [0 , 1] 2 .

Original: This can be achieved by performing a principal component analysis, rotating the ﬁrst found axis by 45 ° anti-clockwise and scaling to ﬁt the cities in the [0 , 1] 2 square.
Revised: This can be achieved by performing a principal component analysis, rotating the ﬁrst found axis by 45 ° anti-clockwise and scaling to ﬁt the cities in the [0 , 1] 2 square.

Original: This transformation, which is slightly different from (Deudon et al., 2018), allows the cities to be as spread as possible in [0 , 1] 2 .
Revised: This transformation, which is slightly different from (Deudon et al., 2018), allows the cities to be as spread as possible in [0 , 1] 2 .

Original: For scaling and translation invariance, we apply a scaling and translation transformation to the city positions such that there are a maximum number of cities (i.e., 2 or 3 depending on conﬁguration) on the border of the square [0 , 1] 2 .
Revised: For scaling and translation invariance, we apply a scaling and translation transformation to the city positions such that there are a maximum number of cities (i.e., 2 or 3 depending on conﬁguration) on the border of the square [0 , 1] 2 .

Original: For reﬂections, we only consider horizontal, vertical, and diagonal ﬂips (with respect to the coordinate axes) for simplicity.
Revised: For reﬂections, we only consider horizontal, vertical, and diagonal ﬂips (with respect to the coordinate axes) for simplicity.

Original: Cities are reﬂected such that a majority of them are in a ﬁxed chosen region.
Revised: Cities are reﬂected such that a majority of them are in a ﬁxed chosen region.

Original: Since the symmetries can be composed, these preprocessing methods can be applied sequentially.
Revised: Since the symmetries can be composed, these preprocessing methods can be applied sequentially.

Original: Theoretically, it would be beneﬁcial to apply all of them in combination, however this has a computational cost.
Revised: Theoretically, it would be beneﬁcial to apply all of them in combination, however this has a computational cost.

Original: It is therefore more effective to only use a selection of the most effective ones.
Revised: It is therefore more effective to only use a selection of the most effective ones.

Original: Moreover, these preprocessing methods can be applied either once on the initial TSP instance ( X ), or at each solving step t on the remaining cities ( X I t ).
Revised: Moreover, these preprocessing methods can be applied either once on the initial TSP instance ( X ), or at each solving step t on the remaining cities ( X I t ).

Original: Theoretically, performing these preprocessings iteratively should help most, since the RL solver is only trained on standardized inputs.
Revised: Theoretically, performing these preprocessings iteratively should help most, since the RL solver is only trained on standardized inputs.

Original: This is conﬁrmed by our experimental analysis.
Revised: This is conﬁrmed by our experimental analysis.

Original: We ﬁnd out that the combination that provides the best results are rotation followed by scaling and translation, which will be used in the experimental evaluation of our method.
Revised: We ﬁnd out that the combination that provides the best results are rotation followed by scaling and translation, which will be used in the experimental evaluation of our method.

Original: However, our overall approach is generic and could include any other symmetries for which equivariance would hold for the RL solver.
Revised: However, our overall approach is generic and could include any other symmetries for which equivariance would hold for the RL solver.

Original: In Appendix E, we present the details of the evaluation of the different preprocessing methods.
Revised: In Appendix F, we present the details of the evaluation of the different preprocessing methods.

Original: In the remaining, we assume that the set of equivariant transformations is ﬁxed.
Revised: In the remaining, we assume that the set of equivariant transformations is ﬁxed.

Original: For any set of indices I ⊆
Revised: For any set of indices I ⊆

Original: [ N ] , we denote the matrix of positions
Revised: [ N ] , we denote the matrix of positions

Original: X I after preprocessing by (cid:102) X I and any coordinates x after preprocessing by ˜ x .
Revised: X I after preprocessing by (cid:102) X I and any coordinates x after preprocessing by ˜ x .

Original: Note that the preprocessing step (e.g., if it includes the rotation transformation described above) may depend on the initial matrix X I , but we prefer not to reﬂect it in the notations to keep them simple.
Revised: Note that the preprocessing step (e.g., if it includes the rotation transformation described above) may depend on the initial matrix X I , but we prefer not to reﬂect it in the notations to keep them simple.

Original: 4. E QUIVARIANT M ODEL
Revised: 4. E QUIVARIANT M ODEL

Original: Our proposed model, which represents the RL solver, has an encoder-decoder architecture (see Figure 1).
Revised: Our proposed model, which represents the RL solver, has an encoder-decoder architecture (see Figure 1).

Original: In our model, the ﬁrst city to visit is ﬁxed arbitrarily, since the construction of the tour should be invariant to the starting city.
Revised: In our model, the ﬁrst city to visit is ﬁxed arbitrarily, since the construction of the tour should be invariant to the starting city.

Original: Therefore, the decisions of the solver starts at time step t ≥ 2 .
Revised: Therefore, the decisions of the solver starts at time step t ≥ 2 .

Original: Our model is designed to take into account other equivariances that are known to hold in the problem.
Revised: Our model is designed to take into account other equivariances that are known to hold in the problem.

Original: Invariance with respect to translation can be further exploited by considering relative positions with respect to the last visited city instead of the original absolute positions, as suggested by Ma et al.
Revised: Invariance with respect to translation can be further exploited by considering relative positions with respect to the last visited city instead of the original absolute positions, as suggested by Ma et al.

Original: Therefore, we deﬁne the input of our RL solver to only include two pieces of information.
Revised: Therefore, we deﬁne the input of our RL solver to only include two pieces of information.

Original: First, the information about the current partial tour now only needs the relative preprocessed position of the ﬁrst visited city ( (cid:126) x σ (1) =
Revised: First, the information about the current partial tour now only needs the relative preprocessed position of the ﬁrst visited city ( (cid:126) x σ (1) =

Original: (cid:101) x
Revised: (cid:101) x

Original: σ (1) − (cid:101) x σ ( t ) ).
Revised: σ (1) − (cid:101) x σ ( t ) ).

Original: Since the last visited city is always represented by the origin, it can be dropped.
Revised: Since the last visited city is always represented by the origin, it can be dropped.

Original: Second, the information about the remaining TSP instance corresponds to the relative preprocessed positions of the unvisited cities ( (cid:126) x σ ( n ) =
Revised: Second, the information about the remaining TSP instance corresponds to the relative preprocessed positions of the unvisited cities ( (cid:126) x σ ( n ) =

Original: (cid:101) x
Revised: (cid:101) x

Original: σ ( n ) − (cid:101) x
Revised: σ ( n ) − (cid:101) x

Original: σ ( t − 1) for n ∈ I t ) and of the ﬁrst and last visited cities ( (cid:126) x σ (1) and the origin (cid:126) x σ ( t ) = 0 ).
Revised: σ ( t − 1) for n ∈ I t ) and of the ﬁrst and last visited cities ( (cid:126) x σ (1) and the origin (cid:126) x σ ( t ) = 0 ).

Original: We denote the matrix of those relative preprocessed positions by −→ X J t , i.e., the matrix whose rows are the rows of (cid:102)
Revised: We denote the matrix of those relative preprocessed positions by −→ X J t , i.e., the matrix whose rows are the rows of (cid:102)

Original: X J t (obtained from X J t ) minus (cid:101) x σ ( t ) .
Revised: X J t (obtained from X J t ) minus (cid:101) x σ ( t ) .

Original: The last visited city needs to be kept in this matrix, since it represents the labels of the nodes of the remaining graph on which the tour should be completed.
Revised: The last visited city needs to be kept in this matrix, since it represents the labels of the nodes of the remaining graph on which the tour should be completed.

Original: Encoder We denote the dimension of theembeddings by H . The encoder of our model is composed of a graph neural network (GNN) (Battaglia et al., 2018), which computes an embedding (cid:99) X J t ∈ R | J t |× H of the relative city positions −→ X J t ∈ R | J t |× 2 , and a multilayer perceptron (MLP), which computes an embedding ˆ x σ (1) ∈
Revised: Encoder With H denoting the embedding dimension, the encoder of our model is composed of a graph neural network (GNN) (Battaglia et al., 2018), which computes an embedding (cid:99) X J t ∈ R | J t |× H of the relative city positions −→ X J t ∈ R | J t |× 2 , and a multilayer perceptron (MLP), which computes an embedding ˆ x σ (1) ∈

Original: R H of the relative position of the ﬁrst visited city −→ x σ (1) ∈ R 2 .
Revised: R H of the relative position of the ﬁrst visited city −→ x σ (1) ∈ R 2 .

Original: The GNN encodes the information about the graph describing the remaining TSP problem.
Revised: The GNN encodes the information about the graph describing the remaining TSP problem.

Original: As a GNN, it is invariant with respect to the order of its inputs and its outputs depend on the graph structure and information (i.e., city positions).
Revised: As a GNN, it is equivariant with respect to the order of its inputs and its outputs depend on the graph structure and information (i.e., city positions).

Original: The MLP encodes the information about visited cities.
Revised: The MLP encodes the information about the ﬁrst city.

Original: Since we exploit the independence with respect to the visited cities between the ﬁrst and last cities (not included), our model does not require any complex architecture, like LSTM (Hochreiter & Schmidhuber, 1997) or GNN, for encoding the positions of the visited cities, in contrast to most previous work.
Revised: Since we exploit the independence with respect to the visited cities between the ﬁrst and last cities (not included), our model does not require any complex architecture, like LSTM (Hochreiter & Schmidhuber, 1997) or GNN, for encoding the positions of the visited cities, in contrast to most previous work.

Original: A simple MLP sufﬁces since only the relative position of the ﬁrst city is required.
Revised: A simple MLP sufﬁces since only the relative position of the ﬁrst city is required.

Original: Formally, the GNN computes its output (cid:99)
Revised: Formally, the GNN computes its output (cid:99)

Original: X J t ∈ R | J t |× H from inputs −→ X J t ∈ R | J t |× 2 through n GNN layers, which allows each city to get information from its neighbors up to n GNN edges away:
Revised: X J t ∈ R | J t |× H from inputs −→ X J t ∈ R | J t |× 2 through n GNN layers, which allows each city to get information from its neighbors up to n GNN edges away:

Original: [Equation]
Revised: [Equation]

Original: where Θ (0) ∈ R 2 × H and Θ ( (cid:96) )
Revised: where Θ (0) ∈ R 2 × H and Θ ( (cid:96) )

Original: ∈
Revised: ∈

Original: R H
Revised: R H

Original: × H are learnable weights, X
Revised: × H are learnable weights, X

Original: ( (cid:96) − 1) ∈ R | J t |× H is the input of the (cid:96) th layer of the GNN for (cid:96) ∈ [ n GNN ] , X ( n GNN ) =
Revised: ( (cid:96) − 1) ∈ R | J t |× H is the input of the (cid:96) th layer of the GNN for (cid:96) ∈ [ n GNN ] , X ( n GNN ) =

Original: (cid:99)
Revised: (cid:99)

Original: X J t , F ( (cid:96) ) :
Revised: X J t , F ( (cid:96) ) :

Original: R | J t |× H
Revised: R | J t |× H

Original: → R | J t |× H is the aggregation function, which is implemented as a neural network, and λ ∈ [0 , 1] is another trainable parameter.
Revised: → R | J t |× H is the aggregation function, which is implemented as a neural network, and λ ∈ [0 , 1] is another trainable parameter.

Original: Visual illustration of the GNN is deferred to Appendix G for space reasons.
Revised: Visual illustration of the GNN is deferred to Appendix H for space reasons.

Original: Decoder Once the embeddings (cid:99)
Revised: Decoder Once the embeddings (cid:99)

Original: X J t and (cid:98) x σ (0) are computed, the probability of selecting a next city to visit is obtained via a standard attention mechanism (Bello et al., 2016) using (cid:99) X J t and (cid:98) x σ (0) as the keys and query respectively.
Revised: X J t and (cid:98) x σ (0) are computed, the probability of selecting a next city to visit is obtained via a standard attention mechanism (Bello et al., 2016) using (cid:99) X J t and (cid:98) x σ (0) as the keys and query respectively.

Original: Formally, the decoder outputs a vector u ∈ R N expressed as: 
Revised: Formally, the decoder outputs a vector u ∈ R N expressed as: [Equation] otherwise

Original: [Equation]
Revised: [Equation]

Original: where u j is the j th entry of the vector u , (cid:99)
Revised: where u j is the j th entry of the vector u , (cid:99)

Original: X J t ,j is the j th row of the matrix (cid:99)
Revised: X J t ,j is the j th row of the matrix (cid:99)

Original: X J t , Θ g and Θ m are trainable matrices with shape H × H , w ∈ R N is another trainable weight vector.
Revised: X J t , Θ g and Θ m are trainable matrices with shape H × H , w ∈ R N is another trainable weight vector.

Original: Then, a softmax transformation turns u into a probability distribution p
Revised: Then, a softmax transformation turns u into a probability distribution p

Original: = (cid:0) p j (cid:1) j ∈
Revised: = (cid:0) p j (cid:1) j ∈

Original: [ N ] over unvisited cities:
Revised: [ N ] over unvisited cities:

Original: [Equation]
Revised: [Equation]

Original: where p j is the j th entry of the probability distribution p and u j is the j th entry of the vector u .
Revised: where p j is the j th entry of the probability distribution p and u j is the j th entry of the vector u .

Original: Note that the probability of any visited city j ∈ I t is zero since u j = −∞ in Equation (6).
Revised: Note that the probability of any visited city j ∈ I t is zero since u j = −∞ in Equation (6).

Original:  A LGORITHM & T RAINING
Revised:  A LGORITHM & T RAINING

Original: We introduce three innovative training techniques that we apply during our training process: combined local search, smoothed policy gradient, and stochastic curriculum learning.
Revised: We introduce three innovative training techniques that we apply during our training process: combined local search, smoothed policy gradient, and stochastic curriculum learning.

Original: Overall, these techniques can help train a fast and generalizable policy, which is able to generate tours that can easily be improved by local search.
Revised: Overall, these techniques can help train a fast and generalizable policy, which is able to generate tours that can easily be improved by local search.

Original: The overall algorithm 1 can be found in appendix C.
Revised: The overall algorithm 1 can be found in appendix C.

Original: Combined Local Search
Revised: Combined Local Search

Original: In contrast to previous work, which generally only considers 2-opt, we propose to use a combination of several (possibly random) local search methods (i.e., local insertion heuristic, random 2-opt, search 2 -opt and search random 3 -opt) to improve a tour generated by the
Revised: In contrast to previous work, which generally only considers 2-opt, we propose to use a combination of several (possibly random) local search methods (i.e., local insertion heuristic, random 2-opt, search 2 -opt and search random 3 -opt) to improve a tour generated by the

Original: RL solver.
Revised: RL solver.

Original: For space reasons, we defer their presentations in Appendix A. A local search method can heuristically improve a tour, but may get stuck in a local optimum.
Revised: For space reasons, we defer their presentations in Appendix A. A local search method can heuristically improve a tour, but may get stuck in a local optimum.

Original: Using a combination of them alleviates this issue, since different heuristics usually have different local optima.
Revised: Using a combination of them alleviates this issue, since different heuristics usually have different local optima.

Original: Another important distinction with previous work is that we also use local search during training, not only testing.
Revised: Another important distinction with previous work is that we also use local search during training, not only testing.

Original: Smoothed Policy Gradient For simplicity, we train our model with the REINFORCE algorithm (Williams, 1992), which leverages the policy gradient for optimization.
Revised: Smoothed Policy Gradient For simplicity, we train our model with the REINFORCE algorithm (Williams, 1992), which leverages the policy gradient for optimization.

Original: However, instead of using the standard policy gradient, which is based on the value of the tour generated by the policy, we compute the policy gradient with the value of the tour improved by our combined local search.
Revised: However, instead of using the standard policy gradient, which is based on the value of the tour generated by the policy, we compute the policy gradient with the value of the tour improved by our combined local search.

Original: While standard RL training may yield policies whose outputs may not easily be improved by local search, our new deﬁnition directly trains the RL solver to ﬁnd solutions that can be improved by local search, which allows RL and local search to have synergetic effects.
Revised: While standard RL training may yield policies whose outputs may not easily be improved by local search, our new deﬁnition directly trains the RL solver to ﬁnd solutions that can be improved by local search, which allows RL and local search to have synergetic effects.

Original: Intuitively, this new policy gradient amounts to smoothing the objective function J ( θ ) that is optimized.
Revised: Intuitively, this new policy gradient amounts to smoothing the objective function J ( θ ) that is optimized.

Original: Recall J ( θ ) = − E
Revised: Recall J ( θ ) = − E

Original: [ L σ ( X )] , where the expectation is taken with respect to σ , which is a random variable corresponding to the tour generated by policy π θ .
Revised: [ L σ ( X )] , where the expectation is taken with respect to σ , which is a random variable corresponding to the tour generated by policy π θ .

Original: In our approach, this usual objective function is replaced by J + ( θ ) = − E
Revised: In our approach, this usual objective function is replaced by J + ( θ ) = − E

Original: [ L σ + ( X )] , where σ + is a random variable corresponding to the improved tour obtained by our combined local search from σ .
Revised: [ L σ + ( X )] , where σ + is a random variable corresponding to the improved tour obtained by our combined local search from σ .

Original: Therefore, this last expectation is taken with respect to the probability distributions generated by policy π θ and our combined local search.
Revised: Therefore, this last expectation is taken with respect to the probability distributions generated by policy π θ and our combined local search.

Original: This objective function can be understood as J + ( θ ) =
Revised: This objective function can be understood as J + ( θ ) =

Original: − E [min σ (cid:48) ∈N ( σ ) L σ (cid:48)
Revised: − E [min σ (cid:48) ∈N ( σ ) L σ (cid:48)

Original: ( X )] where
Revised: ( X )] where

Original: N ( σ ) is a neighborhood of σ deﬁned by local search.
Revised: N ( σ ) is a neighborhood of σ deﬁned by local search.

Original: This stochastic min operation has a smoothing effect on J ( θ ) .
Revised: This stochastic min operation has a smoothing effect on J ( θ ) .

Original: That is why, we call the gradient of J + ( θ ) a smoothed policy gradient .
Revised: That is why, we call the gradient of J + ( θ ) a smoothed policy gradient .

Original: Formally, this novel policy gradient can be estimated on a batch of TSP instances X (1) , . . .
Revised: Formally, this novel policy gradient can be estimated on a batch of TSP instances X (1) , . . .

Original: , X ( B ) :
Revised: , X ( B ) :

Original: [Equation]
Revised: [Equation]

Original: where σ ( b ) is the tour generated by the current policy π θ on instance X ( b ) , σ ( b )+ is the improved tour obtained by our combined local search starting from σ ( b ) , and l ( b ) = − L σ ( b ) ( X ( b ) ) is a novel baseline, which we call policy rollout baseline, used to reduce the variance of the policy gradient estimation.
Revised: where σ ( b ) is the tour generated by the current policy π θ on instance X ( b ) , σ [Equation] b )+ is the improved tour obtained by our combined local search starting from σ ( b ) , and l ( b ) = − L σ ( b ) ( X ( b ) ) is a novel baseline, which we call policy rollout baseline, used to reduce the variance of the policy gradient estimation.

Original: It enjoys the nice property that it does not require additional calculations, since
Revised: It enjoys the nice property that it does not require additional calculations, since

Original: L σ ( b ) ( X ( b ) ) is computed when σ ( b ) is generated.
Revised: L σ ( b ) ( X ( b ) ) is computed when σ ( b ) is generated.

Original: In our experiments, the policy rollout baseline easily outperforms the previous greedy baselines (Ma et al., 2019; Kool et al., 2019).
Revised: In our experiments, the policy rollout baseline easily outperforms the previous greedy baselines (Ma et al., 2019; Kool et al., 2019).

Original: Note that by construction, L σ ( b )+ ( X ( b ) ) ≤
Revised: Note that by construction, L σ ( b )+ ( X ( b ) ) ≤

Original: L σ ( b ) ( X ( b ) )
Revised: L σ ( b ) ( X ( b ) )

Original: =
Revised: =

Original: − (cid:80)
Revised: − (cid:80)

Original: Nt =1 r ( b ) t .
Revised: Nt =1 r ( b ) t .

Original: Therefore, the smoothed policy gradient updates more if our combined local search can make more improvements upon this certain policy.
Revised: Therefore, the smoothed policy gradient updates more if our combined local search can make more improvements upon this certain policy.

Original: For completeness, we provide more details in Appendix B.
Revised: For completeness, we provide more details in Appendix B.

Original: Stochastic Curriculum Learning Curriculum Learning (CL) is widely used in machine learning (Soviany et al., 2021).
Revised: Stochastic Curriculum Learning Curriculum Learning (CL) is widely used in machine learning (Soviany et al., 2021).

Original: Its basic principle is to control the increase of the difﬁculty of training instances.
Revised: Its basic principle is to control the increase of the difﬁculty of training instances.

Original: CL can speed up learning and improve generalization (Weinshall et al., 2018).
Revised: CL can speed up learning and improve generalization (Weinshall et al., 2018).

Original: We apply stochastic CL to train our model.
Revised: We apply stochastic CL to train our model.

Original: Instead of a deterministic process, stochastic CL increases the difﬁculty according to a probability distribution.
Revised: Instead of a deterministic process, stochastic CL increases the difﬁculty according to a probability distribution.

Original: We choose the TSP size (i.e., number of cities) as a measure of difﬁculty for a TSP instance.
Revised: We choose the TSP size (i.e., number of cities) as a measure of difﬁculty for a TSP instance.

Original: We assume it to be in the range of R = { 10 , 11 , . . .
Revised: We assume it to be in the range of R = { 10 , 11 , . . .

Original: For each epoch e , we deﬁne the vector g ( e ) ∈
Revised: For each epoch e , we deﬁne the vector g ( e ) ∈

Original: R 41 (since |R| = 41 ) to be:
Revised: R 41 (since |R| = 41 ) to be:

Original: [Equation]
Revised: [Equation]

Original: where g ( e ) k is the k -th entry of g ( e ) and hyperparameter σ N is the standard deviation of this Gaussian density function.
Revised: where g ( e ) k is the k -th entry of g ( e ) and hyperparameter σ N is the standard deviation of this Gaussian density function.

Original: Then, g ( e ) is turned into a categorical distribution p ( e ) ∈ [0 , 1] 41 via a softmax:
Revised: Then, g ( e ) is turned into a categorical distribution p ( e ) ∈ [0 , 1] 41 via a softmax:

Original: [Equation]
Revised: [Equation]

Original: The k -th entry of p ( e ) gives the probability of choosing a TSP instance of size ( k + 10) at epoch e .
Revised: The k -th entry of p ( e ) gives the probability of choosing a TSP instance of size ( k + 10) at epoch e .

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We present three sets of experiments.
Revised: We present three sets of experiments.

Original: First, to validate the effectiveness of eMAGIC, we train our model on randomly-generated TSP instances (using  CL with sizes up to 50 ), and test the model on other randomly-generated TSP instances (TSP n where size n = 20 up to 1000 ).
Revised: First, to validate the effectiveness of eMAGIC, we train our model on randomly-generated TSP instances (using stochastic CL with sizes up to 50 ), and test the model on other randomly-generated TSP instances (TSP n where size n = 20 up to 1000 ).

Original: Second, to further prove of its generalization capability, we directly evaluate models trained on random instances on realistic symmetric 2D Euclidean TSP instances with sizes range from 51 to 1002 in TSPLIB (Reinelt, 1991).
Revised: Second, to further prove of its generalization capability, we directly evaluate models trained on random instances on realistic symmetric 2D Euclidean TSP instances with sizes range from 51 to 1002 in TSPLIB (Reinelt, 1991).

Original: Third, we conduct an ablation study to show the signiﬁcance of every component of eMAGIC (i.e., equivariance, stochastic CL, policy rollout baseline, combined local search, and RL).
Revised: Third, we conduct an ablation study to show the signiﬁcance of every component of eMAGIC (i.e., equivariance, stochastic CL, policy rollout baseline, combined local search, and RL).

Original: We evaluate three versions of our model: eMAGIC, eMAGIC(s), and eMAGIC(S). In the ﬁrst version, only one improved solution issampled (i.e., generated  from the RL policy and improved by our combined local search). In the second and third versions, we sample respectively 10 andimproved solutions and keep the best tours.
Revised: We evaluate four versions of our model: eMAGIC(G), eMAGIC(S), eMAGIC(s1) and eMAGIC(s10) where G means the tour is generated greedily from the RL policy, while the other ones are based on random sampling and differ with respect to the number of times eMAGIC is applied (once for s1, 10 times for s10 and 100 for S).

Original: The details about the experimental settings and the used hyperparameters, which are the same for all experiments, are provided in Appendix D.
Revised: The details about the experimental settings and the used hyperparameters, which are the same for all experiments, are provided in Appendix D.

Original: Performance on Randomly Generated TSP We compare the performance of our model withother methods in Tables 1 and 2, which covers various types of TSP solvers including exact solvers,  heuristics, and learning-based approaches.
Revised: Performance on Randomly Generated TSP We compare the performance of our model withother methods in Tables 1 and 2, which covers various types of TSP solvers including exact solvers, [Table] [Table] heuristics, and learning-based approaches.

Original: Column 1 and 2 of both Tables 1 and 2 corresponds to the method name and the method type, respectively.
Revised: Column 1 and 2 of both Tables 1 and 2 correspond to the method name and the method type, respectively.

Original: Columns 3, 4, and 5 provide the average tour length, gap to the optimal (provided by Concorde (Applegate et al., 2004)), and computational time, respectively.
Revised: Columns 3, 4, and 5 provide the average tour length, gap to the optimal (provided by Concorde (Applegate et al., 2004)) [Table] and computational time, respectively.

Revised: For space reasons, we only include results of eMAGIC(G) and eMAGIC(S) in Tables

Revised: [Table]

Revised: and 2, and leave results of other versions in Table 4 in Appendix E.1.

Original:  Tables 1 and 2 show that the computational times of exact solvers become prohibitive as the TSP size increases, with Gurobi  not being to solve TSP instances larger than 200 under a reasonable time budget.
Revised: Moreover, we provide the result variances for our methods and evaluate them on TSP10,000 in resp. Appendices E.2 and E.3. Tables 1 and 2 show that the computational times of exact solvers become prohibitive as the TSP size increases. Note that Gurobi is not able to solve TSP instances larger than 200 under a reasonable time budget.

Original: The classic heuristic methods are relatively fast, but their performances are not satisfactory.
Revised: The classic heuristic methods are relatively fast, but their performances are not satisfactory.

Original: Among all learning methods, Fu et al.
Revised: Among all learning methods, Fu et al.

Original: (2021) provide excellent results, but it is based on MonteCarlo tree search, which adapts to the instance to be solved.
Revised: (2021) provide excellent results, but it is based on MonteCarlo tree search, which adapts to the instance to be solved.

Original: Without this adaptivity, our model with or without sampling provides competitive results.
Revised: Without this adaptivity, our model with or without sampling provides competitive results.

Original: It can be better than Fu et al.
Revised: It can be better than Fu et al.

Original: up to
Revised: up to

Original: TSP200.
Revised: TSP200.

Original: This somewhat suggests the limit of our approach, which trains on small instances and directly generalizes to large ones, without learning on the instance to be solved.
Revised: This somewhat suggests the limit of our approach, which trains on small instances and directly generalizes to large ones, without learning on the instance to be solved.

Original: We expect that our approach could be improved by training on slightly bigger instances (e.g., up to 100) or adding an adaptive component like in Fu et al.[Table] [Table] See footnotes of Table 1.
Revised: We expect that our approach could be improved by training on slightly bigger instances (e.g., up to 100) or adding an adaptive component like in Fu et al.

Original: Performance on Realistic TSP Table 3 compares the performances of a variety of learning-based
Revised: Performance on Realistic TSP Table 3 compares the performances of a variety of learning-based

Original: TSP solvers on instances from TSPLIB.
Revised: TSP solvers on instances from TSPLIB.

Original: Each column of Table 3 represents the average gap to the optimal solution over the instances indicated by the corresponding rows.
Revised: Each column of Table 3 represents the average gap to the optimal solution over the instances indicated by the corresponding rows.

Original: We treat the tested instances as small cases if their sizes are under 200 and big cases otherwise.
Revised: We treat the tested instances as small instances if their sizes are under 200 and large instances otherwise.

Original: And we extend the[Table] [Table] testing to larger instances with size up to 1002 and leave the performances of other models as empty for size ranging from 400 to 1002 since inthe their papers, the testings stop at instances with sizes around 400.
Revised: And we extend the testing to larger instances with size up to 1002 and leave the performances of other models as empty for size ranging from 400 to 1002 since in their papers, the testings stop at instances with sizes around 400.

Original: We can observe from Table 3 that our model can perform much better than the other learning-based solvers not only for small problems but also large ones, which demonstrates the strong ability and the practical signiﬁcance of eMAGIC to tackle realistic TSP problems.
Revised: We can observe from Table 3 that our model can perform much better than the other learning-based solvers not only for small problems but also large ones, which demonstrates the strong ability and the practical signiﬁcance of eMAGIC to tackle realistic TSP problems.

Original: When the testing size increases, most models suffer from a relatively big increasing of the average gap while ours only increases less than 1% and remains in a good absolute value ( 3 . 40% ) for even larger instances, which indicates a strong generalization ability of our model.
Revised: When the testing size increases, most models suffer from a relatively big increasing of the average gap while ours only increases less than 1% and remains in a good absolute value ( 3 . 40% ) for even larger instances, which indicates a strong generalization ability of our model.

Original: Further details and experimental results are provided in Appendix H.
Revised: Further details and experimental results are provided in Appendix I.

Original: Ablation Study We demonstrate the strength of all the techniques we applied (including equivariance, stochastic CL, policy rollout baseline, combined local search, and RL) using an ablation study.
Revised: Ablation Study We demonstrate the strength of all the techniques we applied (including equivariance, stochastic CL, policy rollout baseline, combined local search, and RL) using an ablation study.

Original: We turn off each feature one at a time to see if the performance drops compared to the full version of eMAGIC.
Revised: We turn off each feature one at a time to see if the performance drops compared to the full version of eMAGIC(s1).

Original: For equivariance, we remove all the equivariant procedures during training/testing (e.g., deleting visited cities, preprocessing, and using relative positions).

Original: For the policy rollout baseline, we replace it with the self-critic baseline, which is a greedy baseline implemented in (Ma et al., 2019).

Original: For combined local search, we only apply it during testing to check if it can help improve the training process.

Original: For RL, we directly apply combined local search without performing any learning.

Original:  Table 4 demonstrates that each technique plays a role in our model.For space reasons, the ablation study for stochastic CL and individual equivariant components is deferred to Appendix F.
Revised: For space reasons, details of the ablation study are deferred to Appendix G. Table 10 and 11 demonstrate that each technique plays a role in our model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a combination of novel techniques (notably, equivariance, combined local search, smoothed policy gradient) for designing an RL solver for TSP, which shows a good generalization capability.
Revised: We presented a combination of novel techniques (notably, equivariance, combined local search, smoothed policy gradient) for designing an RL solver for TSP, which shows a good generalization capability.

Original: We demonstrated its effectiveness both on random and realistic instances, which shows that our model can reach state-of-the-art performance.
Revised: We demonstrated its effectiveness both on random and realistic instances, which shows that our model can reach state-of-the-art performance.

Original: For future work, the approach can be further improved in various ways, e.g., extending it to the actor-critic scheme and exploiting invariances with the critic; or making it adaptive and learn on the instance to be solved.
Revised: For future work, the approach can be further improved in various ways, e.g., extending it to the actor-critic scheme and exploiting invariances with the critic; or making it adaptive and learn on the instance to be solved.

Original: The approach could also be applied on other combinatorial optimization problems and other RL problems.
Revised: The approach could also be applied on other combinatorial optimization problems and other RL problems.

Original: A C OMBINED L OCAL S EARCH
Revised: A C OMBINED L OCAL S EARCH

Original: We ﬁrst provide a brief introduction to TSP heuristics and then give the detailed presentation of each local search method we used, together with our whole combined local search algorithm.
Revised: We ﬁrst provide a brief introduction to TSP heuristics and then give the detailed presentation of each local search method we used, together with our whole combined local search algorithm.

Original: Since TSP is an NP-hard problem (Papadimitriou, 1977), solving exactly large TSP instances is generally impractical.
Revised: Since TSP is an NP-hard problem (Papadimitriou, 1977), solving exactly large TSP instances is generally impractical.

Original: Hence, many heuristics have been proposed to solve TSP.
Revised: Hence, many heuristics have been proposed to solve TSP.

Original: Two important categories of heuristics are constructive heuristics and local search heuristics.
Revised: Two important categories of heuristics are constructive heuristics and local search heuristics.

Original: On the one hand, constructive heuristics iteratively build a tour from scratch and therefore, do not rely on existing tours.
Revised: On the one hand, constructive heuristics iteratively build a tour from scratch and therefore, do not rely on existing tours.

Original: As an example, insertion heuristics are constructive: they ﬁrst choose a starting city randomly, then repeatedly insert an unvisited city into the partial tour that minimizes the increase of tour length until all the cities are included in the tour.
Revised: As an example, insertion heuristics are constructive: they ﬁrst choose a starting city randomly, then repeatedly insert an unvisited city into the partial tour that minimizes the increase of tour length until all the cities are included in the tour.

Original: The unvisited city can be selected in different ways leading to various versions of insertion heuristics: random insertion, nearest insertion, or farthest insertion (see Kool et al.
Revised: The unvisited city can be selected in different ways leading to various versions of insertion heuristics: random insertion, nearest insertion, or farthest insertion (see Kool et al.

Original: for implementation details).
Revised: for implementation details).

Original: Among them, farthest insertion usually yields the best results.
Revised: Among them, farthest insertion usually yields the best results.

Original: On the other hand, local search heuristics try to improve a given complete tour by perturbing it.
Revised: On the other hand, local search heuristics try to improve a given complete tour by perturbing it.

Original: They are widely used as post optimization for TSP solvers, such as Deudon et al.
Revised: They are widely used as post optimization for TSP solvers, such as Deudon et al.

Original: ; Ma et al.
Revised: ; Ma et al.

Original: One important class of local search heuristic is k -opt , which improves an existing complete tour σ by repeatedly performing the following operation: remove k edges of the current tour and reconnect the obtained subtours in order to decrease the tour length.
Revised: One important class of local search heuristic is k -opt , which improves an existing complete tour σ by repeatedly performing the following operation: remove k edges of the current tour and reconnect the obtained subtours in order to decrease the tour length.

Original: For instance, one 2-opt operation would replace:
Revised: For instance, one 2-opt operation would replace:

Original: [Equation]
Revised: [Equation]

Original: by
Revised: by

Original: [Equation]
Revised: [Equation]

Original: , ..., σ ( N ) (cid:1) where i < j < N if L σ (cid:48) ( X ) <
Revised: , ..., σ ( N ) (cid:1) where i < j < N if L σ (cid:48) ( X ) <

Original: L σ ( X ) .
Revised: L σ ( X ) .

Original: Based on k -opt, the LKH algorithm (Helsgaun, 2017) can often achieve nearly optimal solutions, but requires a long runtime.
Revised: Based on k -opt, the LKH algorithm (Helsgaun, 2017) can often achieve nearly optimal solutions, but requires a long runtime.

Original: Based on how many edges are removed and how the subtours are reconnected during one step, every local search heuristic can be seen as a special case of k -opt.
Revised: Based on how many edges are removed and how the subtours are reconnected during one step, every local search heuristic can be seen as a special case of k -opt.

Original: Since the number of possible k -opt operations is O ( N k ) , 2-opt and 3-opt are usually preferred to efﬁciently search for fast improvements of existing tours, although they can get stuck in local optima.
Revised: Since the number of possible k -opt operations is O ( N k ) , 2-opt and 3-opt are usually preferred to efﬁciently search for fast improvements of existing tours, although they can get stuck in local optima.

Original: Hence, in our work, we use a combination of several local search methods, including local insertion heuristics, random 2 -opt, search 2 -opt, and search random 3 -opt (see below for their deﬁnitions), to efﬁciently ﬁnd potential 2-opt and 3-opt operations to improve the solution proposed by the RL solver.
Revised: Hence, in our work, we use a combination of several local search methods, including local insertion heuristics, random 2 -opt, search 2 -opt, and search random 3 -opt (see below for their deﬁnitions), to efﬁciently ﬁnd potential 2-opt and 3-opt operations to improve the solution proposed by the RL solver.

Original: Instead of running each algorithm once for long enough, we run every local search shortly one by one and repeat this process for multiple times so that the combined local search are able to avoid more local optima.
Revised: Instead of running each algorithm once for long enough, we run every local search shortly one by one and repeat this process for multiple times so that the combined local search are able to avoid more local optima.

Original: Intuitively, the rationale is that once a certain heuristic gets stuck in a local optimum, another can help get it out by trying a different operation.
Revised: Intuitively, the rationale is that once a certain heuristic gets stuck in a local optimum, another can help get it out by trying a different operation.

Original: Next, we explain the above four heuristics, and then illustrate how we leverage them to form our combined local search.
Revised: Next, we explain the above four heuristics, and then illustrate how we leverage them to form our combined local search.

Original: A. R ANDOM 2 - OPT
Revised: A. R ANDOM 2 - OPT

Original: For random 2 -opt, we randomly try two edges for performing a 2-opt operation and repeat this process for α × N β times, where α, β > 0 are two hyperparameters controlling the strength of this heuristic as well as its runtime.
Revised: For random 2 -opt, we randomly try two edges for performing a 2-opt operation and repeat this process for α × N β times, where α, β > 0 are two hyperparameters controlling the strength of this heuristic as well as its runtime.

Original: Theoretically, random 2 -opt can potentially cover all 2-opt operations and such a procedure makes random 2 -opt much more ﬂexible than trying all N ( N − 1) / 2 possible pairs of edges.
Revised: Theoretically, random 2 -opt can potentially cover all 2-opt operations and such a procedure makes random 2 -opt much more ﬂexible than trying all N ( N − 1) / 2 possible pairs of edges.

Original: In this paper, we set α = 0 .
Revised: In this paper, we set α = 0 .

Original: 5 and β = 1 . 5 for all experiments.
Revised: 5 and β = 1 . 5 for all experiments.

Original: A. L OCAL I NSERTION H EURISTIC
Revised: A. L OCAL I NSERTION H EURISTIC

Original: For local insertion heuristic, inspired by the insertion heuristic, we iterate through all cities and ﬁnd the best positions to insert them from their original locations.
Revised: For local insertion heuristic, inspired by the insertion heuristic, we iterate through all cities and ﬁnd the best positions to insert them from their original locations.

Original: Let σ be the current tour and σ t,t (cid:48) be the tour where we exchange the positions of σ ( t ) with σ ( t (cid:48) ) .
Revised: Let σ be the current tour and σ t,t (cid:48) be the tour where we exchange the positions of σ ( t ) with σ ( t (cid:48) ) .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: for t (cid:48) (cid:54) =
Revised: for t (cid:48) (cid:54) =

Original: t − 1 and σ t,t − 1 = σ .
Revised: t − 1 and σ t,t − 1 = σ .

Original: The local insertion heuristic (see Algorithm 1) iterates over every t ∈ [ N ] .
Revised: The local insertion heuristic (see Algorithm 1) iterates over every t ∈ [ N ] .

Original: For each t , we need to ﬁnd t ∗ such that:
Revised: For each t , we need to ﬁnd t ∗ such that:

Original: [Equation]
Revised: [Equation]

Original: where the deﬁnition of L is given in Equation (1).
Revised: where the deﬁnition of L is given in Equation (1).

Original: Then we replace σ by σ t,t ∗ .
Revised: Then we replace σ by σ t,t ∗ .

Original: Theoretically, the local insertion heuristic is a special case of 3-opt where two of the removed edges cover a same node.
Revised: Theoretically, the local insertion heuristic is a special case of 3-opt where two of the removed edges cover a same node.

Original: Algorithm
Revised: Algorithm

Original: 1: Input : A matrix of city coordinates X
Revised: 1: Input : A matrix of city coordinates X

Original: =
Revised: =

Original: ( x i )
Revised: ( x i )

Original: i ∈
Revised: i ∈

Original: [ N ] , current tour σ 2: Output : An improved tour σ 3: for t = 1 to N do 4: t ∗ =
Revised: [ N ] , current tour σ 2: Output : An improved tour σ 3: for t = 1 to N do 4: t ∗ =

Original: arg min t (cid:48)
Revised: arg min t (cid:48)

Original: L σ t,t (cid:48)
Revised: L σ t,t (cid:48)

Original: ( X ) 5: σ ← σ t,t ∗ 6: end for 7: return σ
Revised: ( X ) 5: σ ← σ t,t ∗ 6: end for 7: return σ

Original: A. S EARCH 2 - OPT
Revised: A. S EARCH 2 - OPT

Original: For search 2 -opt, we ﬁrst iterate through all edges and for each ﬁrst edge, we search for the best second edge to do the 2 -opt.
Revised: For search 2 -opt, we ﬁrst iterate through all edges and for each ﬁrst edge, we search for the best second edge to do the 2 -opt.

Original: Let σ be the current tour and σ ( t,t (cid:48) ) be the tour where we reverse the cities between σ ( t ) and σ ( t (cid:48) ) .
Revised: Let σ be the current tour and σ ( t,t (cid:48) ) be the tour where we reverse the cities between σ ( t ) and σ ( t (cid:48) ) .

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: (13) where t < t (cid:48) , and σ ( t,t ) = σ .
Revised: (13) where t < t (cid:48) , and σ ( t,t ) = σ .

Original: Search 2 -opt (see Algorithm 3) iterates over every t ∈ [ N ] .
Revised: Search 2 -opt (see Algorithm 2) iterates over every t ∈ [ N ] .

Original: For each t , we ﬁnd t ∗ such that:
Revised: For each t , we ﬁnd t ∗ such that:

Original: [Equation]
Revised: [Equation]

Original: Then we replace σ by σ ( t,t ∗ ) .
Revised: Then we replace σ by σ ( t,t ∗ ) .

Original: Search 2 -opt potentially covers all possible 2-opt operations.
Revised: Search 2 -opt potentially covers all possible 2-opt operations.

Original: Algorithm 2 Search 2 -opt
Revised: Algorithm 2 Search 2 -opt

Original: current tour σ 2: Output : An improved tour σ 3: for t = 1 to N do 4: t ∗ =
Revised: current tour σ 2: Output : An improved tour σ 3: for t = 1 to N do 4: t ∗ =

Original: arg min t (cid:48) ≥ t L σ ( t,t (cid:48) )
Revised: arg min t (cid:48) ≥ t L σ ( t,t (cid:48) )

Original: ( X ) 5: σ ← σ ( t,t ∗ ) 6: end for 7: return σ
Revised: ( X ) 5: σ ← σ ( t,t ∗ ) 6: end for 7: return σ

Original: A. S EARCH R ANDOM 3 - OPT
Revised: A. S EARCH R ANDOM 3 - OPT

Original: For search random 3 -opt, the algorithm ﬁrst randomly picks two edges.
Revised: For search random 3 -opt, the algorithm ﬁrst randomly picks two edges.

Original: Then for these two randomly-picked edges, similar to search 2 -opt, we search to ﬁnd the best third edge to apply 3opt.
Revised: Then for these two randomly-picked edges, similar to search 2 -opt, we search to ﬁnd the best third edge to apply 3opt.

Original: Let σ be the current tour, edges from σ ( t 1 ) to σ ( t 1 + 1) and σ ( t 2 ) to σ ( t 2 + 1) be the two randomly picked edges and σ ∗ ( t 1 ,t 2 ,t 3 ) be the optimal tour that differs from σ only in edges from σ ( t 1 ) to σ ( t 1 + 1) , σ ( t 2 ) to σ ( t 2 + 1) and σ ( t 3 ) to σ ( t 3 + 1) .
Revised: Let σ be the current tour, edges from σ ( t 1 ) to σ ( t 1 + 1) and σ ( t 2 ) to σ ( t 2 + 1) be the two randomly picked edges and σ ∗ ( t 1 ,t 2 ,t 3 ) be the optimal tour that differs from σ only in edges from σ ( t 1 ) to σ ( t 1 + 1) , σ ( t 2 ) to σ ( t 2 + 1) and σ ( t 3 ) to σ ( t 3 + 1) .

Original: For randomly picked t 1 and t 2 , search random 3 -opt (see Algorithm A.4) iterates over every t 3 ∈ [ N ] and ﬁnd t ∗ 3 such that:
Revised: For randomly picked t 1 and t 2 , search random 3 -opt (see Algorithm 3) iterates over every t 3 ∈ [ N ] and ﬁnd t ∗ 3 such that:

Original: [Equation]
Revised: [Equation]

Original: We repeat this process for α × N β times, where α, β > 0 are the same two hyperparameters as in random 2 -opt.
Revised: We repeat this process for α × N β times, where α, β > 0 are the same two hyperparameters as in random 2 -opt.

Original: We set α = 0 .
Revised: We set α = 0 .

Original: 5 , β = 1 . 5 for all experiments.
Revised: 5 , β = 1 . 5 for all experiments.

Original: Search random 3 -opt potentially covers all possible 3-opt operation.
Revised: Search random 3 -opt potentially covers all possible 3-opt operation.

Original: A. C OMBINED L OCAL S EARCH
Revised: A. C OMBINED L OCAL S EARCH

Original: With these 4 different local search heuristics, our combined local search applies them one by one sequentially and repeat this for I times (see Algorithm 4), where I is a hyperparameter, which we set to I = 10 for all experiments in this paper.
Revised: With these 4 different local search heuristics, our combined local search applies them one by one sequentially and repeat this for I times (see Algorithm 4), where I is a hyperparameter, which we set to I = 10 for all experiments in this paper.

Original: B P OLICY G RADIENT OF E MAGIC
Revised: B P OLICY G RADIENT OF E MAGIC

Original: As promised in Section 5 - Smoothed Policy Gradient, we elaborate on the detailed mathematical derivation for Equation (8).
Revised: As promised in Section 5 - Smoothed Policy Gradient, we elaborate on the detailed mathematical derivation for Equation (8).

Original: The objective function J ( θ ) in Equation (3) can be approximated with the empirical mean of the total rewards using B trajectories sampled with policy π θ :
Revised: The objective function J ( θ ) in Equation (3) can be approximated with the empirical mean of the total rewards using B trajectories sampled with policy π θ :

Original: [Equation]
Revised: [Equation]

Original: (cid:35)
Revised: (cid:34)

Original: [Equation]
Revised: [Equation]

Original: |
Revised: (cid:88) t = r ( b ) t (cid:35) =

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | 
Revised: | |

Original: [Equation]
Revised: [Equation]

Original:  b [Equation] t
Revised: | (cid:88) b =

Original: [Equation] r [Equation] (16) where ˆ E B represents the empirical mean operator and r ( b ) t is the t -th reward of the b -th trajectory.
Revised: N (cid:88) t = r ( b ) t , (16) where ˆ E B represents the empirical mean operator, L σ ( X ) is the tour length of σ output by the RL policy and r ( b ) t is the t -th reward of the b -th trajectory.

Original: The policy gradient to optimize J ( θ ) can be estimated by:
Revised: The policy gradient to optimize J ( θ ) can be estimated by:

Original: [Equation]
Revised: [Equation]

Original: However, instead of directly using Equation (17) to update our policy, we interleave the policy gradient with our combined local search, as well as the policy rollout baseline l ( b ) to update our policy:
Revised: However, Recall J ( θ ) is the standard objective used in most deep RL methods applied to TSP. Instead, we optimize J + ( θ ) = − E [ L σ + ( X )] where L σ + ( X ) is the tour length of σ after applying local search. This helps integrate better RL and local search by smoothing the value landscape and training an RL agent to output a tour that can be improved by local search.

Revised: This new objective function can be rewritten:

Original: [Equation]
Revised: [Equation]

Original: which is the same as Equation (8).
Revised: [Equation] where ρ ( σ ) denotes the distribution over tours induced by the application of the stochastic local search on σ . Taking the gradient of this new objective: [Equation]

Original: Please refer to Section 5 and Appendix A for details regarding our combined local search and the policy rollout baseline l ( b ) . 
Revised: where τ = ( s 1 , a 1 , . . . ) and σ is its associated tour. We simply approximate the conditional expectation over ρ ( σ ) by a sample. Therefore, our gradient estimate is an unbiased estimator of the gradient of our new objective J + ( θ ) . Using our policy rollout baseline introduces some bias to the estimation of the smoothed policy gradient.

Revised: However, the variance reduction helps with achieving greater performance and smaller variance, as we observed in our experiments.

Original: C O VERALL A LGORITHM
Revised: C O VERALL A LGORITHM

Original: We provide the pseudo-code of our overall training algorithm in Algorithm 5.
Revised: We provide the pseudo-code of our overall training algorithm in Algorithm 5.

Revised: N ← Sample from p ( e ) according to Stochastic CL 5: for t = 1 to T do 6: ∀ b ∈ { 1 , ..., B } X ( b ) ← Random TSP instance with N cities 7: ∀ b ∈ { 1 , ..., B } σ ( b ) ← Apply π θ on X ( b ) after all the equivariant preprocessing steps 8: ∀ b ∈ { 1 , ..., B } σ ( b )+ ← Apply the combined local search on σ ( b ) 9:

Revised: Use σ ( b ) and σ ( b )+ to calculate ∇ θ J + ( θ ) according to Equation (8) 10: θ ← Update in the direction of ∇ θ J + ( θ ) 11: end for 12: end for

Original: D S ETTINGS AND H YPERPARAMETERS
Revised: D S ETTINGS AND H YPERPARAMETERS

Original: All our experiments are run on a computer with an Intel(R) Xeon(R) E5-2678 v3 CPU and a NVIDIA 1080Ti GPU.
Revised: All our experiments are run on a computer with an Intel(R) Xeon(R) E5-2678 v3 CPU and a NVIDIA 1080Ti GPU.

Original: In consistency with previous work, all our randomly generated TSP instances are sampled in [0 , 1] 2 uniformly.
Revised: In consistency with previous work, all our randomly generated TSP instances are sampled in [0 , 1] 2 uniformly.

Original: During training, stochastic CL chooses a TSP size in R = { 10 , 11 , . .
Revised: During training, stochastic CL chooses a TSP size in R = { 10 , 11 , . .

Original: , 50 } for each epoch e according to Equations (9) and (10), with σ N set to be 3 We trained for 200 epochs in each experiment, with 1000 batches of 128 random TSP instances in each epoch.
Revised: , 50 } for each epoch e according to Equations (9) and (10), with σ N set to be 3. We trained for 200 epochs in each experiment, with 1000 batches of 128 random TSP instances in each epoch.

Original: We set the learning rate to be 10 − 3 and the learning rate decay to be 0 .
Revised: We set the learning rate to be 10 − 3 and the learning rate decay to be 0 .

Original: 96 in each experiment.
Revised: 96 in each experiment.

Original: In each experiment, we set α = 0 .
Revised: In each experiment, we set α = 0 .

Original: 5 , β = 1 . 5 , γ = 0 .
Revised: 5 , β = 1 . 5 , γ = 0 .

Original: 25 and I = 10 for the parameters of our combined local search; As for random TSP testing and the ablation study, we test on TSP instances with size 20, 50, 100, 200, 500 and 1000 to evaluate the generalization capability of our model; as for realistic TSP, we test on TSP instances with sizes up to 1002 from the TSPLIB library.
Revised: 25 and I = 10 for the parameters of our combined local search; As for random TSP testing and the ablation study, we test on TSP instances with size 20, 50, 100, 200,and 1000 to evaluate the generalization capability of our model; as for realistic TSP, we test on TSP instances with sizes up to 1002 from the TSPLIB library.

Original: With respect to our model architecture, our MLP encoder has an input layer with dimension 2, two hidden layers with dimension 128 and 256, respectively, and an output layers with dimension 128; for the GNN encoder, we set H = 128 and n GNN = 3 . 
Revised: With respect to our model architecture, our MLP encoder has an input layer with dimension 2, two hidden layers with dimension 128 and 256, respectively, and an output layers with dimension 128; for the GNN encoder, we set H =and n GNN = 3 . E M ORE E XPERIMENTS ON E MAGIC E. M ORE VERSIONS OF E MAGIC As promised in Section [Table]

Revised: we illustrate all versions of eMAGIC in this section:

Revised: [Table]

Original: E E VALUATION OF E QUIVARIANT P REPROCESSING M ETHODS E. C OMPARISONS BETWEEN DIFFERENT SYMMETRIES USED DURING PREPROCESSING
Revised: [Table] See footnotes of Table 4. E. V ARIANCE A NALYSIS OF E MAGIC As promised in Section

Revised: [Table]

Revised: we provide the variance analysis of eMAGIC in this section.

Revised: In our experiments

Revised: [Table]

Revised: we repeated all our experiments (training + testing) with 3 random seeds.

Revised: The variances are shown below:

Revised: [Table]

Revised: See footnotes of Table 4.

Revised: The variances are quite low, showing that our method gives relatively good and stable results.

Revised: Note the variances increase with the number of sampling (s1, s10 and S) for larger TSP instances since there is more room for improvement.

Revised: E. E XPERIMENTS ON E XTREMELY L ARGE TSP I NSTANCES

Revised: In Table 7, we show the performances of eMAGIC on extremely large TSP instances (i.e., TSP 10,000) and the comparisons with a few methods that are able to generalize to TSP 10,000.

Revised: For the hyperparameters of the combined local search, we use I = 2 and β = 1 . 4 .

Revised: We modify the hyperparameters in this way because we ﬁnd that for large TSP instances, doing too many iterations of local search is not efﬁcient.

Revised: The other experimental settings in this test are the same as Section 6.

Revised: Also

Revised: [Table]

Revised: we test Fu et al.

Revised: (2021)’s method with a limited the time budget, denoted by AGCRN+M lim : since we only reproduce their method successfully with CPU, we decrease their hyperparameter T (the MCTS will end no longer than T seconds) from 0 .

Revised: 04 n to 0 .

Revised: 04 n/ 1 .

Revised: 8 ∗ (28 / 60) = 0 .

Revised: 010 n (1.8h and 28m are respectively the runtimes of their model and eMAGIC(G)).

Revised: By this, we expect that their model and eMAGIC(G) will have a similar running time

Revised: [Table]

Revised: [Table]

Revised: LS - Combined Local Search

Revised: BS - Beam Search

Revised: We can observe that our models outperform AM (Kool et al., 2019) to a large extent.

Revised: Comparing to AGCRN+M (Fu et al., 2021) and LKH3 (Helsgaun, 2017), our methods run much faster without a big gap of performance.

Revised: Plus, if we give AGCRN+M (Fu et al., 2021) a time budget comparable to our method (i.e., run AGCRN+M and eMAGIC(G) for the same amount of time), we can see that our algorithm outperforms AGCRN+M. Note that MCTS in AGCRN+M is written in C++.

Revised: We could reduce our runtime if we had also written our code in C++ instead of Python.

Revised: Therefore, our method offers a better trade-off in terms of performance vs runtime: it can generate relatively good results in much less time.

Revised: F E VALUATION OF E QUIVARIANT P REPROCESSING M ETHODS F. C OMPARISONS BETWEEN DIFFERENT SYMMETRIES USED DURING PREPROCESSING

Original: We present the comparison results of applying rotation, translation and reﬂection.
Revised: We present the comparison results of applying rotation, translation and reﬂection.

Original: The comparisons are done by testing on random TSP cases followed the same setting in the experiment section.
Revised: The comparisons are done by testing on random TSP instances followed the same setting in the experiment section.

Original: As in table E.1, the rotation has the best performance on small and large TSP instances.
Revised: As in table 8, the rotation has the best performance on small and large TSP instances.

Original: By this, we choose rotation for our ﬁnal algorithm.
Revised: By this, we choose rotation for our ﬁnal algorithm.

Original: [Table]
Revised: [Table]

Original: E. C OMPARISONS BETWEEN ONE PREPROCESSING APPLICATION AND ITERATION PREPROCESSING APPLICATIONS
Revised: TSP24 .7.05% 24 .7 . F. C OMPARISONS BETWEEN ONE PREPROCESSING APPLICATION AND ITERATION PREPROCESSING APPLICATIONS

Original: We present the comparison results of one preprocessing application and iteration preprocessing applications.
Revised: We present the comparison results of one preprocessing application and iteration preprocessing applications.

Original: The comparisons are done by testing on random TSP cases followed the same setting in the experiment section.
Revised: The comparisons are done by testing on random TSP instances followed the same setting in the experiment section.

Original: As in table E.2, the iteration preprocessing applications has better performance on small and large TSP instances.
Revised: As in table 9, the iteration preprocessing applications has better performance on small and large TSP instances.

Original: By this, we choose iteration preprocessing applications for our ﬁnal algorithm
Revised: By this, we choose iteration preprocessing applications for our ﬁnal algorithm.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: F M ORE A BLATION S TUDIES
Revised: G A BLATION S TUDIES

Original: This section is the continuation of Section 6 - Ablation Study.
Revised: This section illustrates the details of Section 6 - Ablation Study.

Revised: G. A BLATION S TUDY ON K EY T RAINING T ECHNIQUES

Original: In this section,  we ﬁrst perform an ablation study  of stochastic CL, meaning we ﬁxed our TSP size to be 50 during training.
Revised: In this section, as promised in Section 6 - Ablation Study [Table] we will illustrate the details of the ablation study regarding some key training techniques of eMAGIC. For equivariance, we remove all the equivariant procedures during training/testing (e.g., deleting visited cities, preprocessing, and using relative positions).

Revised: For the policy rollout baseline, we replace it with the self-critic baseline, which is a greedy baseline implemented in Ma et al.

Original: Moreover, we perform an ablation study on each component of our equivalent model, which includes deleting the visited cities during training, equivariant preprocessing operation
Revised: For combined local search, we only apply it during testing to check if it can help improve the training process. For RL, we directly apply combined local search without performing any learning. Table 10 demonstrates that each technique plays a role in our model. [Table]

Revised: G. A BLATION S TUDY ON O THER T RAINING T ECHNIQUES

Revised: In this section

Original: [Table]
Revised: [Table]

Original:  and using relative positions during training, as mentioned in Section 6 - Ablation Study.
Revised: we ﬁrst perform an ablation study of stochastic CL, meaning we ﬁxed our TSP size to be 50 during training. Moreover [Table] we perform an ablation study on each component of our equivalent model, which includes deleting the visited cities during training, equivariant preprocessing operation, and using relative positions during training, as mentioned in Section 6 - Ablation Study.

Original: For the ablation study of these three components, we remove them during training and testing.
Revised: For the ablation study of these three components, we remove them during training and testing.

Original: Table 7 illustrates the results of our ablation study (including the full version for comparison)
Revised: Table 11 illustrates the results of our ablation study (including the full version for comparison), which demonstrate that each component is effective in our model.

Original: [Table]
Revised: [Table]

Original: which demonstrate that each component is effective in our model
Revised: In Section 6, when doing the ablation study for the combined local search, we do not apply it during training, but we still apply it during testing to check if it can help improve the training process. Here, we compare the pure RL algorithm (full algorithm without local search both in training and testing) and the full algorithm to demonstrate the strength of Combined Local Search:

Original: [Table] [Table]
Revised: [Table] [Equation]

Original: G GNN E NCODER D ETAILS

Original: Figure 2 is a detailed illustration of the GNN architecture introduced in Section 4.2.
Revised: Figure 2 is a detailed illustration of the GNN architecture introduced in Section 4.2.

Original: Please notice that Figure 2 could be regarded as a zoom-in version of the GNN part in Figure 1.
Revised: Please notice that Figure 2 could be regarded as a zoom-in version of the GNN part in Figure 1.

Original: The aggregation function used in GNN is represented by a neutral network followed by a ReLU function on each entry of the output.
Revised: The aggregation function used in GNN is represented by a neutral network followed by a ReLU function on each entry of the output.

Original: H R EALIST TSP I NSTANCES IN TSPLIB
Revised: I R EALIST TSP I NSTANCES IN TSPLIB

Original: As promised in Section 6, Tables 9 and 10 list the performances of various learning-based TSP solvers and heuristic approaches upon realistic TSP instances in TSPLIB.
Revised: As promised in Section 6, Tables 14 to 16 list the performances of various learning-based TSP solvers and heuristic approaches upon realistic TSP instances in TSPLIB.

Original: The bold numbers show the best performance among all the approaches.
Revised: The bold numbers show the best performance among all the approaches.

Original: We can observe that most bold numbers are provided by eMAGIC, meaning our approach provides excellent results for TSPLIB.
Revised: We can observe that most bold numbers are provided by eMAGIC, meaning our approach provides excellent results for TSPLIB.

Original: In addition, we also provide our results for the larger instances (with size up to 1002) from TSPLIB in Table 8.
Revised: In addition, we also provide our results for the larger instances (with size up to 1002) from TSPLIB in Table 13.

Original: Tableonly contains the experiments of our model since no other model can generalize to large size TSP instances with adequate performance. 
Revised: Table 13 only contains the experiments of our model since no other model can generalize to large size TSP instances with adequate performance. [Table] [Table] kroC

Original: [Table]
Revised: [Table]

Revised: kroD

Original: [Table] 
Revised: [Table] [Table]

Original: 03% kroB 26, 26,282 0 .
Revised: kroE [Table] [Table] % 1in 14, 14,379 0.00% 18,194 26 .

Revised: 48% pr 44, 44,346 0.10% 53,056 19 .

Revised: 76% 45, 1 . 83% pr 59, 59,075 0.08% 66,010 11 .

Original: 52% pr 73, 73,682 0 . 
Revised: 39% bier 118,282 119,151 0.73% 142,707 20 . 78% ch 6, 6,162 0.85% 7,120 16 . 62% pr 96, 97,264 0.51% 105, 9 . [Table]

Original: 92% u 42, 42,080 0 .[-
-]
Revised: 79% pr 58, 58,537 0.00% 71,006 21 . [Table] 54% ch 6, 6,592 0.98% 7,916 21 . kroA 26, 26,727 0.77% 31,244 17 .[-
-]

Original:  rat 2, 2,377 2 .
Revised: 14% kroB 26, 26,282 0.58% 31,407 20 . 13% pr 73, 73,682 0.00% 85,616 16 . 17% u 42, 42,080 0.00% 51,327 21 . 97% 45, 7 . 97% rat 2, 2,377 2.32% 2,913 25 .

Original: 84% d 15, 15,874 0 .
Revised: 11% d 15, 15,874 0.60% 17,962 13 .

Original:  kroA 29, 29,840 1 . 27% kroB 29, 29,743 1 . 04% 36,412 23 .
Revised: 26% kroA 29, 29,840 1.61% 35,958 22 . 44% 30, 5 . 44% kroB 29, 29,743 1.04% 36,412 23 .

Original: 67% ts 126,643 126,939 0 .
Revised: 69% 31, 7 . 66% ts 126,643 126,939 0.23% 158,748 25 .

Original: 52% tsp 3, 3,981 1 . 66% 4,701 20 .
Revised: 35% 136, 7 . 63% tsp 3, 3,981 1.66% 4,701 20 .

Original: 32% pr 80, 80,436 0 .
Revised: 08% pr 80, 80,436 0.08% 97,348 21 .

Original: 23% gi 2, 2,417 1 .
Revised: 87% gi 2, 2,417 1.64% 2,963 24 .

Original: 93% pr 49, 
Revised: 69% pr 49, 49,908 1.57% 65,946 34 . 21% 52, 6 . 57% a 2, 2,635 2.17% 2,989 15 . 17% pr 48, 48,905 1.48% 59,786 24 . 06% 51, 7 . 69% lin 42, 42,948 2.19% − − 45, 7 . 96%

Original: [Table] [Table]
Revised: [Table] [Table]

Original: kroC
Revised: ei

Original: [Table]
Revised: [Table]

Revised: ber1in

Original: [Table]
Revised: [Table]

Original: kroD

Original: [Table]
Revised: [Table]

Original: kroE
Revised: st

Original: [Table]
Revised: [Table]

Original: bier
Revised: ei

Original: [Table]
Revised: [Table]

Revised: pr

Revised: [Table]

Revised: rat

Revised: [Table]

Revised: [Table]

Revised: kroA 21,

Revised: [Table]

Revised: [Table]

Revised: kroB 22,

Revised: [Table]

Revised: [Table]

Revised: kroC 20,

Revised: [Table]

Revised: [Table]

Revised: kroD 21,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: kroE 22,

Revised: [Table]

Revised: [Table]

Revised: rd

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ei

Revised: [Table]

Revised: % lin 14, 16,156 12 .

Revised: [Table]

Original: {+% pr 44, 54,
+}
Revised: {+% pr 44, 54,
+}

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: % pr 59, 59, 0 .

Revised: [Table]

Revised: [Table]

Revised: % bier 118,282 121,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ch 6,

Revised: [Table]

Revised: [Table]

Original: kroA
Revised: [Table] [Table] [Table] pr 96, 98,

Original: [Table] 
Revised: [Table] [Table]

Revised: pr 58,

Revised: [Table]

Revised: [Table]

Revised: ch 6,

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: kroA 26,

Revised: [Table]

Revised: [Table]

Original: 85% kroB 26, 28, 7 . 80% 35, 60 .
Revised: [Table] [Table] kroB 26, 28, [Table]

Original: 96% pr 73, 75, 2 . 20% 103, 39 .
Revised: [Table] [Table] [Table] pr 73, 75, 2 . 20% 103, 39 .

Original: 78% u 42, 42, 1 . 51% 115,372 174 .
Revised: [Table] % u 42, 42, 1 . [Table] [Table]

Original: 27% rat 2, 2,955 27 . 
Revised: [Table] % rat 2, 2,955 27 . [Table]

Original: 14% d 15, − − 68,104 331 .
Revised: [Table] [Table] [Table] % d 15, − − 68,104 331 .

Original: kroA 29, 32,522 10 .
Revised: kroA 29, 32,522 10 .

Original: 55% kroB 29, − − 50, 72 .
Revised: 79% kroB 29, − − 50, 72 .

Original: 47% ts 126,643 127, 0 .
Revised: 79% 31, 7 . 08% ts 126,643 127, 0 .

Original: 05% tsp 3, 4,354 11 .
Revised: 04% tsp 3, 4,354 11 .

Original: 24% pr 80, 91,560 13 .
Revised: 70% 4, 9 . 30% pr 80, 91,560 13 .

Original: 07% gi 2, 2, 4 .
Revised: 68% gi 2, 2, 4 .

Revised: 30% pr 49, 59,109 20 .

Revised: 84% a 2, 2,898 12 .

Revised: 36% pr 48, 59,422 23 .

Revised: 70% lin 42, − − − − 45, 9 .

Revised: 25% ∗ as

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: kroA

Revised: [Table]

Revised: kroB

Revised: [Table]